MODULE 1 — FOUNDATION

How a Small Language Machine works

Here we open the black box. First the intuition, then the mechanics: text becomes numbers, the machine predicts the next token, context changes everything and probabilities guide the choice. In the end, a prototype you run in the browser.

Module principle

Building small is the best way to understand big models.

Lesson complete15 minFundação

What is a language machine?

The central idea before any code

▸ Lesson objective

Understand that a language machine tries to continue a piece of text based on learned patterns.

The most important idea

A language machine receives a piece of text and tries to predict which continuation makes the most sense.

For example: if you write ‘The cat climbed the…’, a likely continuation would be ‘roof’. The model does not need to imagine a cat in a human way. It calculates patterns it has seen before.

Input, processing and output

We can think of three parts: you hand it a sentence; the machine analyses the patterns; it picks a likely continuation.

If the input is ‘I like coffee with’, the output might be ‘milk’, ‘sugar’ or ‘cinnamon’, depending on what it has learned.

The machine starts knowing nothing — and that is fine

At stage zero, our machine knows nothing. It needs to receive texts, observe repetitions and build some rule for deciding what comes next.

The first version can be extremely simple: count which words tend to follow other words.

A useful analogy

Imagine someone who does not speak English but has read thousands of sentences. They notice that after ‘good’ the word ‘morning’ often follows. Without understanding the meaning, they can still complete the pattern.

A Small Language Machine starts exactly there: by observing patterns.

[ practice ]

Quick exercise

Complete these mentally: ‘I brush my teeth with…’, ‘For breakfast I drink…’, ‘The sky is full of…’. Then ask yourself: why did those answers come so quickly?

See expected answer

Because your brain has already seen many similar patterns. The machine tries to imitate that ability using statistics.

✓ What you take away

A Small Language Machine receives text, observes context, estimates possible continuations, picks a continuation and repeats the process.

Living glossary

Terms we will encounter in this module.

Language model

A system that learns text patterns in order to predict or generate new sequences of text.

Corpus

A set of texts used as observation or training material for the machine.

Vocabulary

The list of text pieces the machine knows at a given moment. In real models, usually a token vocabulary.

ID

A number used as a label to represent a word, token or piece of text.

Sequence

The order of numbers, words or tokens. In language, order almost always changes meaning.

Prediction

The choice of the next word or token based on context and observed patterns.

Probability

A simple measure of how likely a continuation seems compared to other possibilities.

Distribution

A list of possibilities with weights or chances associated with each.

Sampling

A choice made by controlled random draw, respecting the probabilities of the available options.

Temperature

A concept used to control whether the model's choice will be more conservative or more varied.

Token

A piece of text used by the machine. It can be a word, part of a word, a space, punctuation or a combination.

Tokenization

The process of breaking human text into tokens before transforming those tokens into numbers.

Context

The sequence of tokens available to the machine at the moment it needs to choose the next continuation.

Context window

The amount of text the machine can consider at once to guide its answer.

Bigram

A pair of sequential elements, like one word followed by another: "I" → "like". A simple way to observe patterns.

Trigram

A sequence of three elements, or a rule where two previous words help predict the next.

N-gram

A group of n sequential elements. A bigram has 2 elements; a trigram has 3; and so on.

Continuation table

A simple memory that records which words or tokens appeared after each observed context.

Fallback

An emergency rule used when the machine encounters a context it does not recognize.

Generation cycle

The repeated process of looking at the context, choosing the next token, appending it to the text and continuing.

Embedding

A rich numeric representation of a token, word or text snippet in a space of meaning.

Vector space

A mathematical map where items can be positioned and compared by proximity.

Semantic neighbourhood

A group of words or tokens that are close together because they appear in similar contexts or have related uses.

Attention

A mechanism that assigns different weights to parts of the context to decide which information is most relevant.

Self-attention

A mechanism in which each token observes the other tokens in the same text to update its own representation.

Attention weight

A value indicating how much importance a part of the context receives at a given processing step.

Attention head

One of the parallel attention focuses in a Transformer. Each head can learn to observe different relations in the text.

Prototype

A small, transparent and experimental version created to learn how the system works before making it more complex.

How a Small Language Machine works

Module principle

Building small is the best way to understand big models.