Table of Contents

• Introduction
• What is a Perceptron?
• Example: Spam Detector Implementation
• Training a Perceptron
• Limitations of Perceptron
• Conclusion

Introduction

To me Perceptron sounds like a characater from the Transformer franchaise, where there is indeed a Cybertron. In Machine Learning, Pertreptron is an intelligent decision-making unit created by Frank Rosenblatt back in 1958. It is essentially a binary classifier. One of the earlier applications of Perceptrion is spam email filtering. We take as input the frequency of alert words such as ‘free’ or ‘winning’, the reputation of the email sender, the length of the email, and some other features, and output 1 (Spam) or 0 (Not spam).

(A visual representation of how a perceptron processes inputs and produces an output)

What is a Perceptron?

A Perceptron is a decision-maker that takes inputs (like spam words in emails) and outputs a binary decision (yes or no). It does so by first calculating a weighted sum:

\[z = (w_1 \times x_1) + (w_2 \times x_2) + \ldots + (w_n \times x_n) + \text{bias}\]

Or, in summation notation:

\[z = \sum_{i=1}^{n} w_i x_i + b\]

Then it applies what’s called a step (activation) function to ‘force’ the sums into either 0 or 1, and we get our binary classifier.

\[\text{output} = \begin{cases} 1, & \text{if } z \geq 0 \\ 0, & \text{if } z < 0 \end{cases}\]

Where:

• If the weighted sum (z) is zero or more, the perceptron outputs 1.
• If the weighted sum is less than zero, the perceptron outputs 0.

Example: Spam Detector

Suppose we have two clues (inputs) about an email:

1. Does it have spammy words (like “free money!”)? (1 if yes, 0 if no)
2. Is the sender a bit shady? (1 if yes, 0 if no)

Let’s say an email is ONLY spam if BOTH are true (this is like a logical “AND”).

import numpy as np
import matplotlib.pyplot as plt # For a cool graph later!

class Perceptron:
    def __init__(self, learning_rate=0.1, n_epochs=10): 
        # How fast it learns and for how long
        self.lr = learning_rate
        self.n_epochs = n_epochs
        self.weights = None # We don't know the best weights yet!
        self.bias = 0 # Or the best bias!

    # This is where the learning happens
    # The function name fit implies that we are fitting the coefficients 
    # (weights) to a correct result.
    def fit(self, X, y): # X is our input data, y is the right answers
        n_features = X.shape[1] # How many clues (features) we have
        self.weights = np.zeros(n_features) # Start with guessing weights are zero

        for _ in range(self.n_epochs): # Go through the data a few times
            for x_i, target_label in zip(X, y): # Look at each email example
                # Make a guess
                weighted_sum = np.dot(x_i, self.weights) + self.bias
                prediction = 1 if weighted_sum >= 0 else 0

                # How wrong were we?
                error = target_label - prediction

                # Nudge the weights and bias to be better next time
                update_amount = self.lr * error
                self.weights += update_amount * x_i
                self.bias += update_amount

    # After learning, how does it make a prediction?
    def predict(self, X):
        weighted_sum = np.dot(X, self.weights) + self.bias
        return np.where(weighted_sum >= 0, 1, 0) # 1 if sum >=0, else 0

# Our toy email data: [spammy_words, suspicious_sender]
X_emails = np.array([
    [0, 0],  # Not spammy, not suspicious -> Not Spam (0)
    [0, 1],  # Not spammy, but suspicious -> Not Spam (0)
    [1, 0],  # Spammy, but not suspicious -> Not Spam (0)
    [1, 1]   # Spammy AND suspicious -> SPAM! (1)
])
y_labels = np.array([0, 0, 0, 1]) # The "correct" answers

# Let's train our perceptron!
spam_detector = Perceptron(learning_rate=0.1, n_epochs=10)
spam_detector.fit(X_emails, y_labels)

predictions = spam_detector.predict(X_emails)
print("Our detector's predictions:", predictions) # Hopefully [0 0 0 1]
print("The actual labels:", y_labels)
print("Learned weights (how important each clue is):", spam_detector.weights)
print("Learned bias:", spam_detector.bias)

def plot_decision_boundary(X, y, model):
    plt.figure(figsize=(7,5))
    plt.scatter(X[y == 0][:, 0], X[y == 0][:, 1], 
                label="Not Spam", c="skyblue", marker="o", s=100)
    plt.scatter(X[y == 1][:, 0], X[y == 1][:, 1], 
                label="Spam!", c="salmon", marker="x", s=100)

    # The line our perceptron "draws" to separate spam from not-spam
    # Equation: w1*x1 + w2*x2 + b = 0  => x2 = -(w1*x1 + b) / w2
    # We need to make sure weights[1] isn't zero to avoid division by zero!
    if model.weights[1] != 0:
        x1_vals = np.array([min(X[:,0])-0.5, max(X[:,0])+0.5]) # A range for our line
        x2_vals = -(model.weights[0] * x1_vals + model.bias) / model.weights[1]
        plt.plot(x1_vals, x2_vals, "k--", label="Decision Boundary")
    else: # If weight[1] is zero, the boundary is a vertical line x1 = -bias/weight[0]
        if model.weights[0] != 0:
            x1_val = -model.bias / model.weights[0]
            plt.axvline(x=x1_val, color='k', linestyle='--', 
                       label="Decision Boundary")
        else: # If both weights are zero, it's a bit weird, maybe no boundary or all one class
            print("Can't draw boundary if both weights are zero (or close to it)!")

    plt.xlabel("Clue 1: Spammy Words? (0=No, 1=Yes)")
    plt.ylabel("Clue 2: Suspicious Sender? (0=No, 1=Yes)")
    plt.xticks([0,1])
    plt.yticks([0,1])
    plt.legend()
    plt.grid(True, linestyle=':', alpha=0.7)
    plt.title("Perceptron Deciding: Spam or Not Spam?")
    plt.xlim(-0.5, 1.5)
    plt.ylim(-0.5, 1.5)
    plt.show()

plot_decision_boundary(X_emails, y_labels, spam_detector)

What’s Happening in the Code?

1. Perceptron Class:
   • __init__: Sets up how fast it’ll learn (learning_rate) and how many times it’ll look at the data (n_epochs).
   • fit: This is the training part. It guesses, checks if it’s right, and if not, it tweaks its weights and bias to do better next time. It keeps doing this for all the examples, over and over.
   • predict: Once trained, this is how it makes a call on new (or old) data.
2. Toy Data: We made up a tiny dataset. It’s “linearly separable,” meaning you could draw a straight line to separate the spam from the not-spam examples (you’ll see this in the plot!).
3. Training: We create a Perceptron and tell it to fit our data.
4. Output: It should predict [0 0 0 1], meaning it learned our “AND” rule! The weights and bias tell us how it learned that rule.
5. Plot: The graph shows our data points and the “decision boundary” – the line the perceptron uses to make its decision. Everything on one side is “not spam,” and on the other is “spam.”

This is a super simple example, but it shows the core idea! Real spam detectors use way more clues (features) and much more data. In the real world, we have moved to using deep learning and more sophisticated models for combating spam than perceptrons.

Training a Perceptron (How Does It Actually Learn?)

‘Training’ a model in this context means that we have many emails and we know whether these emails are spam or not. This means that we already know the answers and we are training the model to find the best weights that can arrive at the correct conclusion.

1. First Guess (Initialization): The perceptron starts out not knowing anything. So, it makes a wild guess for its weights (how important each clue is) and its bias. Often, it just sets all weights to zero or some small random numbers.

2. Then the perceptron starts with one email and makes a prediction. It compares this prediction with the known result.

3. The difference between the prediction and the known result is called error. Based on the error, the perceptron adjusts the weights by applying the learning rate to the difference, if there is an error. This adjustment will hopefully lead to a better result in the next round. The rate of this learning process is controlled by the value of learning_rate. A larger value means faster learning rate but we may also miss the correct value since we are jumping too quickly. A smaller value, on the other hand, means we may need to learn many more times and the process will be slower.

4. Linearly Separable: It means that we can draw a straight line to divide the spam from non-spam emails.

Limitations of Perceptron

The Dreaded XOR Problem (and Linear Separability)

Remember how we said the perceptron works great if you can draw a straight line to separate your groups? That’s called being “linearly separable.”

Consider the XOR (eXclusive OR) problem. XOR means “one or the other, but not both”:

• (0,0) -> 0
• (0,1) -> 1
• (1,0) -> 1
• (1,1) -> 0

It is linearly non-separable! AND and OR gates, on the other hand, are linearly separable.

AND Gate (like our spam example):

• (0,0) -> 0
• (0,1) -> 0
• (1,0) -> 0
• (1,1) -> 1

We can draw a line to separate the (1,1) point from the others.

Conclusion

The perceptron, despite its simplicity, holds a special place in the history of machine learning. It was one of the first attempts to create a machine that could learn from examples and make decisions, much like how our brains work. While it has its limitations - particularly its inability to solve non-linearly separable problems like XOR - it laid the groundwork for the sophisticated neural networks we use today.

Think of the perceptron as the “Hello World” of neural networks. It teaches us fundamental concepts that are still relevant:

• How to combine multiple inputs with weights
• How to make binary decisions
• How to learn from mistakes
• The importance of linear separability

In modern machine learning, we rarely use single perceptrons anymore. Instead, we use multi-layer perceptrons (MLPs) and deep neural networks that can solve much more complex problems. These advanced networks are essentially stacks of perceptron-like units with more sophisticated activation functions and learning algorithms.

So the next time you use a spam filter or interact with any AI system, remember that it all started with this simple but revolutionary idea from Frank Rosenblatt in 1958. The perceptron may be basic, but it’s the foundation upon which modern AI is built.