Part One could finish cat, car, and can. Now we extend its training examples into three sentences. When the prompt ends in “is ”, the last character is a space in every case. Our tiny model looks up the same row and makes the same prediction, even though the earlier subject differs.

↓ First, try giving the count model more memory.

Instead of a row for “ ”, use a row for the last three characters, “is ”. That is still the same for all three prompts. Eight characters retain the subject and can distinguish “ cat is ”, “ car is ”, and “ can is ”. Move the context control below and watch which earlier characters remain visible.

↓ A longer lookup has a catch: what if that exact context never appeared?

Choose “the cab is ” in the experiment. Its eight-character context was never observed, so there is no row to look up. A shorter context can provide a fallback, and adding small counts can avoid zero probabilities. Those are useful count-model improvements—but neither creates an understanding of the new subject.

↓ Can we reuse pieces of what we learned, rather than demand a separate row for every full context?

Longer contexts create many possible combinations. A counting table stores the observed combinations separately. A neural model instead uses a shared calculation whose adjustable numbers are reused for many inputs. This can help generalization; it is a different approach, not a requirement for our original tiny task.

↓ A shared calculation needs an input representation with adjustable values, rather than just an arbitrary token label.

Give every character an ID to locate it in a table. Its row contains several adjustable numbers: a vector. When that row represents a token, it is called an embedding. The entries tell later calculations how strongly to use different shared patterns. The ID finds the row; the embedding supplies values to calculate with.

↓ Let’s make the recipe do something visible, using the same t/r/n choices from Part One.

A weight is an adjustable multiplier: 2 doubles a number, 0 removes its contribution, and a negative value reverses its sign. Use the embedding [0.75, 0.25] to multiply two shared patterns, then add. This gives a score for each possible next character. Higher scores express stronger preference, not percentages.

↓ These predictions only become useful when the recipe and patterns are adjusted using real examples.

Softmax converts scores into probabilities. Loss measures how little probability the model assigned to the observed next token. Gradients tell us how a small change to each adjustable number would affect that loss; backpropagation computes them through the calculation. An optimizer then takes a small update step. Repeat this across examples to train the embeddings and weights together.

↓ Token embeddings can now be useful inputs. For longer text, we still need to preserve order and combine context.

The words cat and act contain the same characters. Looking up their embeddings gives the same three rows in different orders. If we simply add the rows, the result is identical. Position information lets later calculations distinguish where each token occurred.

↓ Now the model can combine information from different positions without treating every position identically.

Attention compares the current representation with available positions and turns comparison scores into mixing shares. It combines their information using those shares. Unlike a fixed lookup row, this mixture depends on the input. To complete “the cat is ”, the model can use information from the earlier subject as well as the final space.

↓ The context-dependent result feeds later transformations and the output-score calculation.

A small decoder-only transformer combines token representations and position information with causal attention and a feedforward network. The feedforward network applies learned transformations and a nonlinear rule, allowing more than one linear remix. Residual connections carry an earlier representation forward; normalization helps control numerical scales. These blocks produce a vector used to score next tokens.

↓ Once trained, generation is still the same loop you already understand.

Part One looked up a count row. The neural version calculates probabilities from a learned representation of the context. Both choose a token, append it, and repeat until END or a length limit. During generation, the learned weights stay fixed. Evaluate on held-out text to check whether the extra machinery actually helped.

Return to the tiny model whenever you want to see the complete generation loop without the extra machinery.