tensoratlas

A visual atlas of machine learning

Understand the ideas.
Build a language model.

See the journey first. Explore the concepts when you need them.

The learning atlas 22 introductions

Part one · a model you can see working

Ten examples. One tiny language model.

Count what follows each character, turn counts into chances, and generate a word. No vectors or embeddings needed yet.

  1. 01The goal
  2. 02Pieces & boundaries
  3. 03Count pairs
  4. 04Counts to chances
  5. 05Choose a token
  6. 06Write, then stop
  7. 07What it cannot do
01

I want to generate “cat”, “car”, or “can”.

I will build a model that grows a word one character at a time. It will learn only from a tiny collection of examples, and all its decisions will be visible.

Our training collection
cat × 6 car × 3 can × 1
Repeating a word makes it more common in the training collection. This first model needs counts, not neural networks.

First, I need to show where each example begins and ends.

02

Break each word into characters, with a start and a stop.

Each character is a token: one piece the model handles. I also add two special tokens, START and END. They mark boundaries; they are not letters in the word. START lets the model choose a first character. END tells it to stop.

One example
START → c → a → t → END
Another example
START → c → a → r → END
The third example
START → c → a → n → END
Each repeated word contributes its whole sequence. Keeping the boundaries prevents the end of one word from becoming the beginning of the next.

Now I can count every pair of neighboring tokens.

03

Count what follows each token.

For every example, I record each neighboring pair. All ten words start with c, and every c is followed by a. After a, the collection splits: six t’s, three r’s, and one n. Each final letter is followed by END.

Always the same beginning
START → c: 10 c → a: 10
The branching point
a → t: 6 a → r: 3 a → n: 1
How words end
t → END: 6 r → END: 3 n → END: 1
This table is the learned model. A model that predicts using only the previous token is called a bigram model.

Counts tell me how often something happened. I need chances to make a choice.

04

Divide by the total for the current token.

After a there are ten observations. Six out of ten gives t a probability of 60%; three gives r 30%; one gives n 10%. For START and c there is only one observed next token, so that next token has probability 100%.

After a
t: 6 ÷ 10 = 60% r: 3 ÷ 10 = 30% n: 1 ÷ 10 = 10%
After c
a: 10 ÷ 10 = 100%
Divide by the total in that token’s row—not by all pairs in the collection. Each row’s probabilities add up to 100%.

To write a word, I need to choose from these chances.

05

Use a random draw to follow the probabilities.

Imagine ten tickets: six say t, three say r, and one says n. Drawing a ticket gives more common choices a better chance without always choosing the most common one. This is sampling.

Tickets after a
t t t t t t r r r n
Equivalent draw from 0 up to 1
below 0.6 → t 0.6 to below 0.9 → r 0.9 to below 1 → n
A few generated words need not match 60/30/10 exactly. Randomness is part of sampling, not a change to the learned counts.

I can repeat this choice after every token, starting at START.

06

Choose, append, and repeat until END.

Start at START. Its row gives c, so write c. The c row gives a, so write a. The a row chooses t, r, or n. That final letter’s row gives END, so stop without writing the marker. The counts stay fixed while the word grows.

One possible run
START → c → a → r → END
The text we return
car
That is a complete working language model. In this collection it can generate only cat, car, and can. Try changing their counts below.

The experiment follows exactly this table. Nothing is hidden in a trained neural network.

A working model · all calculations happen here

Make it write.

Change the training collection. Inspect the counts. Follow each choice from START to END.

10 examples. The table below is learned by counting neighboring tokens.

Training table · probabilities are rounded for display
Previous tokenNext tokenCountChance
STARTc10100%
ca10100%
at660%
ar330%
an110%
tEND6100%
rEND3100%
nEND1100%
Generated text—

Current token: START. Ready to choose the first character.

    The highlighted rows are the choices available now. A new word starts at START. Changing the collection rebuilds the table and clears the current word.

    07

    A working model can still be very limited.

    This model remembers only the last token. It cannot distinguish different longer prefixes ending in the same character. It also gives unobserved transitions zero probability. Our three-word collection is too small to teach general language; generating a familiar word does not prove understanding.

    It knows
    Which token followed the previous one.
    It does not know
    What a cat, car, or can means.
    A useful next experiment
    Use two previous tokens instead of one.
    Longer contexts can help, but require more observations and create more unseen combinations. Learned representations become relevant when we tackle those problems.

    First understand this complete loop. Then explore why a more capable model needs different tools.

    You now have a complete model: examples → counts → probabilities → generated text.

    Part two: why go beyond counting? →

    Linear algebra

    Vectors

    Describe an observation with an ordered list of numbers.

    Start here · No earlier lesson needed

    Dot product

    Combine inputs and weights into a single score.

    Builds on Vectors

    Matrices

    Apply several weighted sums at once.

    Builds on Dot product

    Calculus

    Functions

    Connect an input to an output through a rule.

    Start here · No earlier lesson needed

    Derivatives

    Measure how an output changes near an input.

    Builds on Functions

    Probability & statistics

    Probability

    Express uncertainty with numbers between zero and one.

    Start here · No earlier lesson needed

    Distributions

    Describe how probability is spread across outcomes.

    Builds on Probability

    Optimization

    Loss functions

    Turn prediction errors into a quantity to minimize.

    Builds on Functions

    Gradient descent

    Find a minimum by taking small steps downhill.

    Builds on Derivatives + Loss functions

    Supervised learning

    Linear regression

    Predict a numerical target with a weighted sum.

    Builds on Dot product + Loss functions

    Neural networks

    Artificial neurons

    Follow a weighted sum through an activation function.

    Builds on Dot product + Functions

    Vision

    Convolution

    Reuse a small filter across a larger input.

    Builds on Matrices

    Sequence models

    Sequences

    Represent data whose order carries information.

    Builds on Vectors

    Attention

    Build an output by weighting information from other positions.

    Builds on Dot product + Sequences

    Unsupervised learning

    Clustering

    Group observations using a notion of similarity.

    Builds on Vectors

    Graph learning

    Graphs

    Describe objects through their connections.

    Start here · No earlier lesson needed

    Reinforcement learning

    Agents & rewards

    Learn actions through interaction and feedback.

    Builds on Probability

    Evaluation

    Train & test

    Measure performance on data excluded from fitting.

    Builds on Loss functions + Probability

    Generalization

    Regularization

    Encourage models to fit patterns that carry beyond training data.

    Builds on Linear regression + Train & test

    Deployment

    Inference & deployment

    Use a trained model to make predictions in a real setting.

    Builds on Train & test

    Architectural patterns

    Autoencoders

    Learn a representation by reconstructing an input.

    Builds on Artificial neurons + Loss functions

    Interpretability

    Feature effects

    Inspect how a prediction responds to a changing input.

    Builds on Functions + Train & test