Perceptrons

↑ Top

Introduction

This is the first in a series exploring the foundations of modern AI.

As I near the end of my six‑year Open University degree, I have been reflecting on what I have learned, how I’ve applied it to my work, and how I can share that knowledge with others.

Two years ago, I enrolled in TM358: Machine Learning and Artificial Intelligence, where I studied the mathematical foundations of AI and applied them to a range of practical problems. This experience aligned perfectly with my transition into professional AI/ML work — and I haven’t looked back since.

My final‑year project has been centred on creating my own chatbot, a journey that led me deep into the mechanics of transformers, natural language processing, and more.

But at its core, everything I’ve studied — from chatbots to large‑scale transformers — ultimately traces back to one foundational idea: Frank Rosenblatt’s 1958 Perceptron, a simple but revolutionary model for how information can be stored and organized in the brain.

In preparing for an upcoming community presentation on quantization in AI, I decided to start this series by revisiting Rosenblatt’s work — the starting point for much of what we now call modern artificial intelligence and machine learning.

The Goal

By the end of this post, you will understand:

• What it means for data to be linearly separable and why that matters for perceptrons (with clear 2D and 3D examples).
• How a perceptron takes inputs, applies weights and biases, and uses summation to make decisions.
• How a simple step activation function enables binary classification.
• Why perceptrons struggle with non‑linearly separable problems — and what that means for their limitations.

What Does It Mean for Data to Be Linearly Separable?

Before diving into how a perceptron works, it’s important to understand the type of problems it was designed to solve. The key concept here is linear separability.

In simple terms, data is linearly separable if you can draw a straight line (in 2D), a plane (in 3D), or a hyperplane (in higher dimensions) that cleanly separates two classes of points. Every point on one side belongs to one class, and every point on the other side belongs to another class.

Visualising Decision Boundaries

Imagine you’re plotting two types of points on a 2D graph:

• Red circles represent Class A.
• Blue circles represent Class B.

If these two groups can be divided by drawing a single straight line, they’re linearly separable.

In 3D, the concept is the same, except instead of a line, you’d use a plane. And in higher dimensions (which is where most machine learning happens), we generalize this idea to hyperplanes.

Why Linear Separability Matters

A single perceptron can only classify data that’s linearly separable.

If the classes are mixed together in a way that no single line (or plane) can separate them, the perceptron fails. This limitation famously shows up in problems like XOR — one of the simplest examples of non‑linear separability.

Understanding this boundary is crucial: it’s the dividing line (literally) between what the earliest neural networks could solve and why more complex, multi-layer networks became necessary.

This is exactly the kind of boundary Rosenblatt envisioned perceptrons learning — but it also shows why they hit a wall when faced with more complex problems.

When Data Isn’t Linearly Separable: The XOR Problem

But not all datasets are this simple.

A classic example is the XOR problem, where the classes are arranged so that no single straight line can separate them — making it impossible for a simple perceptron to classify them correctly

In 3D, the situation is similar — the classes can be positioned so that no single plane can cleanly separate them.

This simple example illustrates the core limitation of single-layer perceptrons: if no straight line (or plane) can separate the classes, they fail. Later in this series, we’ll see how multi-layer perceptrons overcome this challenge.

Next, let’s open the hood and see how a perceptron actually works — exploring inputs, weights, and how these simple units make decisions.

Inside the Perceptron: How It Processes Information

So far, we’ve looked at what kinds of problems a perceptron can solve. Now let’s open the hood and see how it actually processes information.

At its core, a perceptron is a very simple model of a neuron. It takes several inputs, applies a bit of math to them, and then decides whether to “fire” (activate) or not.

Inputs, Weights and Biases

Think of a perceptron like a hiring manager scoring job applications.

• Inputs: These are the candidate’s attributes — years of experience, education level, portfolio quality, certifications. In data terms, these are the features of your dataset.
• Weights: Not all attributes are equally important. Weights represent how much the manager values each one. Maybe experience is weighted heavily, while a minor certification carries little influence.
• Bias: Even before looking at the applications, the manager has a baseline tendency — perhaps they’re under pressure to fill the role quickly (lowering the threshold for acceptance), or perhaps they’re being very selective (raising the bar). This “bias” shifts the decision point up or down, independent of the actual attributes.

Put together: The perceptron multiplies each attribute by its importance, adds them up, adjusts for the baseline bias, and then decides whether the candidate gets shortlisted.

A Quick Primer: What’s a Dot Product?

Before we jump into the math, let’s unpack two terms you’ll see: matrix multiplication and dot product.

• A vector is just an ordered list of numbers. For example:

$$\mathbf{x} = \begin{bmatrix} 0.5 & 0.8 & 0.2 \end{bmatrix}$$

could represent three features of an email (like spam score, length, and exclamation count).

• A dot product is how we combine two vectors: multiply each pair of numbers and add them together.

For example:

$$\begin{aligned} \left[ \begin{matrix} 0.5 & 0.8 & 0.2 \end{matrix} \right] \left[ \begin{matrix} 1.0 \\ 0.5 \\ 0.2 \end{matrix} \right] &= (0.5 \cdot 1.0) + (0.8 \cdot 0.5) + (0.2 \cdot 0.2) \\ &= 0.94 \end{aligned}$$

This is matrix multiplication in its simplest form: a row vector multiplied by a column vector.

That’s all the perceptron is doing at this stage: combining features (inputs) with their importance (weights) to produce a single number.

Summation: Combining It All

Once we have our inputs, weights, and bias, the perceptron combines them into a single score $z$:

$$\begin{aligned} z &= \mathbf{w}^\top \mathbf{x} + b \\ &= (w_1 \cdot x_1) + (w_2 \cdot x_2) + \dots + (w_n \cdot x_n) + b \end{aligned}$$

This is just the dot product of the inputs and weights, plus the bias. In plain English: multiply each input by its importance, add them all up, then adjust with the baseline bias.

Worked Example

Suppose we have three features:

• $x_1 = 0.50$
• $x_2 = 0.80$
• $x_3 = 0.20$

And three weights:

• $w_1 = 1.00$
• $w_2 = 0.50$
• $w_3 = 0.20$

With a bias $b = 0.00$.

The perceptron’s weighted sum becomes:

$$\begin{aligned} z &= (0.50 \cdot 1.00) + (0.80 \cdot 0.50) + (0.20 \cdot 0.20) + (1 \cdot 0.00) \\ &= 0.94 \end{aligned}$$

This single value $z$ is what the perceptron uses in the next step — passing it through an activation function to make a binary decision.

The Step Activation Function: Turning Math Into Decisions

So far, our perceptron has taken inputs, multiplied them by their weights, added a bias, and produced a single score $z$. But that’s not the final output.

To actually make a decision, we need one more step: an activation function.

What is an Activation Function?

An activation function takes the combined score $z$ and turns it into an output — often a simple decision like “yes” or “no.”

In neural networks, activation functions can be complex (like ReLU or sigmoid), but for perceptrons, we use the simplest one of all: the step function.

The Step Function

The step function does exactly what its name implies — it “steps” the output from one value to another once $z$ crosses a threshold.

$$f(z) = \begin{cases} 1 & \text{if } z \geq \theta \\ 0 & \text{if } z < \theta \end{cases}$$

Where $\theta$ is the threshold.

• If $z$ is above or equal to the threshold → output 1 (e.g., yes, Class A).
• If $z$ is below the threshold → output 0 (e.g., no, Class B).

In other words: if the evidence is strong enough, classify as 1. Otherwise, classify as 0.

How This Enables Classification

With the step function, the perceptron turns its weighted sum into a binary decision.

• Spam or not spam
• Dog or cat
• Approve or reject

This is exactly how Rosenblatt’s perceptron performed classification: by drawing a boundary (line, plane, or hyperplane) in the data and assigning everything on one side to one class, and everything on the other to another.

How Does a Perceptron Learn?

So far, we’ve treated the perceptron’s weights and bias as if they magically exist. But how does it actually learn them?

The short answer is trial and error.

During training, the perceptron is shown inputs along with their correct outputs (labels). It makes a prediction, compares it to the correct answer, and adjusts its weights slightly if it was wrong. Over time, these small adjustments shift the perceptron’s decision boundary to correctly separate the training data.

This is the simplest form of learning for a neural network — and the ancestor of the more powerful backpropagation algorithm used in deep learning today.

The Perceptron Learning Rule

The perceptron adjusts its weights using a simple rule:

$$\begin{aligned} w_i &\leftarrow w_i + \Delta w_i \\ \Delta w_i &= \eta (y - \hat{y}) x_i \end{aligned}$$

Where:

• $\eta$ is the learning rate — how big a step to take each time.
• $y$ is the true label — the correct answer.
• $\hat{y}$ is the predicted output — what the perceptron guessed.
• $x_i$ is the input for that weight.

In plain English:

• If the perceptron gets it right, do nothing.
• If it gets it wrong, nudge each weight in the direction that would have produced the correct output.

Over many iterations, these small adjustments shift the perceptron’s decision boundary until it correctly separates the classes in the training data.

From Training to Inference

Once training is complete, the perceptron stops adjusting its weights. These learned weights and bias are now fixed — this phase is called inference.

During inference, the perceptron no longer learns; it simply applies its trained weights to new, unseen inputs to make predictions.

Looking Ahead: From Single Neurons to Deep Learning

In this post, we’ve unpacked one of the simplest — yet most important — ideas in machine learning: the perceptron.

You learned:

• What linear separability means and why it defines the kinds of problems a perceptron can solve.
• How a perceptron processes inputs using weights, a bias, and summation to produce a single score.
• How the step activation function turns that score into a binary decision.
• Why single-layer perceptrons are limited — they fail on non-linearly separable problems like XOR.
• How perceptrons learn through trial and error, using a simple weight update rule — a primitive ancestor of the backpropagation algorithm used in deep learning today.

This foundation sets the stage for everything that came after Rosenblatt’s original idea — ultimately leading to the multi-layer perceptrons and deep networks that power modern AI.

In my next post, we’ll explore different activation functions and how they change what a perceptron can do — paving the way for more flexible and powerful models.

Later in the series, we’ll dive into multi‑layer perceptrons and backpropagation — the key innovations that transformed simple neurons into the deep networks powering today’s AI systems.

Thank you for reading. In the next post, we will explore activation functions and how they expand what perceptrons can do.