A Hands-On Introduction to Quantisation

Watch the talk

The problem it answers

Not every model can run in the cloud. Some must run on a device, on site or offline, and inference costs grow with every call. Quantisation is how a model fits on smaller hardware: it stores the model's weights in fewer bits, trading accuracy for size and speed. Knowing where that trade bites is what separates a model that runs well on constrained hardware from one that runs badly.

What you'll take away

By the end, you'll be able to:

• Describe how a neural network turns inputs into decisions, from a single perceptron's weights and bias to a multi-layer network's forward pass.
• Compare activation functions, from step and sigmoid to ReLU and Mish, by how each shapes a neuron's output.
• Explain why a single perceptron can't solve XOR, and why layers with non-linear activations can.
• Explain quantisation through audio first, where each extra bit of depth improves the signal-to-noise ratio by about 6 dB.
• Calculate how quantising weights from 32-bit floats to 16, 8, 4 or 2 bits shrinks a model and the data moved at inference.
• Evaluate the trade between size and accuracy for a given use, and identify which models suit edge devices: smaller, non-LLM models tend to suit them better than large language models.

See it working

Each concept is shown live in an interactive app, where you change weights, activation functions and bit depths and see the effect straight away. It ends with a quantised model running on a Raspberry Pi: the Homeric chatbot from my undergraduate research. It's a virtual Homer, the ancient Greek poet credited with the Iliad and the Odyssey, built as an interactive museum exhibit that visitors can question to learn more about both poems.

Previous and upcoming events

Events where I've run this, or will, with links to each event and its recording where there is one.