Simple Machine Learning
A field journal · v.01
← Back
FiledSeptember 16, 2026·5 min read·By The Editor

Tokens and Embeddings

A token is a chunk of text; an embedding turns it into a point in space where nearness means similarity. Together they explain why models miscount letters, fumble arithmetic, and still find the right paragraph.

llmbeginnerintuition

Two words you cannot avoid once you start using language models, and they describe consecutive steps of the same process.

A token is a chunk of text. An embedding is that chunk turned into a list of numbers that carries its meaning. Text goes in, tokens come out, embeddings go into the model.

Tokens: the model does not see words

Models do not read characters, and they do not quite read words either. Text is carved into tokens — chunks chosen because they appear often.

Figure 1 · what the model actually reads
How text is split into tokensThree examples: a common sentence splits into whole words, an unusual word splits into pieces, and a number splits digit by digit.Common wordsThe⎵cat⎵satone token each — cheapUnusual wordunbelievabilityone word, three tokens — rarer spellings cost moreNumbers473.82digits split apart — which is why arithmetic is hard for itTOKENS, NOT WORDSrule of thumb: 1 token ≈ ¾ of a word ≈ 4 characters
A model never sees letters or words — it sees tokens, chunks carved out by frequency. Common words are single tokens; rare ones shatter into pieces. This is why models miscount letters and fumble arithmetic: they are not looking at the characters you are.

A rough rule: one token is about ¾ of a word, or four characters. A thousand words is roughly 1,300 tokens.

In plain English

Common words get their own token. Rare words get chopped into pieces the model has seen before. This keeps the vocabulary manageable — about 100,000 entries instead of every word in every language — while still letting the model spell out anything at all.

This explains several things people find baffling:

  • "How many r's in strawberry?" The model may see st|raw|berry. It is being asked about letters it cannot individually see.
  • Arithmetic is shaky. 473.82 can split digit by digit. The model is manipulating text-shaped fragments, not numbers.
  • Some languages cost more. Tokenizers are trained mostly on English. The same sentence in another script can take two or three times the tokens — and you are billed per token.
  • Context limits are in tokens. A "128k context window" is roughly 96,000 words, not 128,000.

Embeddings: meaning becomes geometry

A token is still just an ID — number 4,912 in the vocabulary. That ID means nothing by itself. The next step gives it meaning.

Each token is mapped to a vector: a long list of numbers, typically a few hundred to a few thousand of them. That vector is its embedding, and it is a position in space.

Figure 2 · embeddings put meaning on a map
Embedding spaceWords plotted as points. Animals cluster together, royalty clusters elsewhere, and the arrow from king to queen matches the arrow from man to woman.animalscatdogkittenpuppykingqueenmanwomansame directionsame directionking − man + woman ≈ queenthe direction itself encodes “gender”MEANING BECOMES GEOMETRYsimilar meaning → nearby points → large dot product
Each token becomes a long list of numbers — a point in space. Training arranges that space so related things sit near each other, and so that directions carry meaning. Nobody designed these positions; they are what falls out of predicting text well.

The arrangement is not arbitrary. Training pushes tokens that appear in similar contexts toward similar positions, because that is what makes the next word easier to predict. The result is that distance means similarity — and, famously, that directions mean something too. The step from king to queen is close to the step from man to woman.

In plain English

Nobody sat down and decided where "cat" goes. These positions are ordinary weights, set by gradient descent like every other number in the model. The structure appeared because it was useful for guessing text.

How similarity is actually measured

Comparing two embeddings is a dot product — the operation that asks how much two vectors agree. Usually it is normalised first, giving cosine similarity:

similarity=a⋅b∥a∥ ∥b∥\text{similarity} = \frac{\mathbf{a} \cdot \mathbf{b}}{\lVert \mathbf{a} \rVert \, \lVert \mathbf{b} \rVert}
In plain English

Divide out both lengths and you are left with the angle alone: 1 means pointing the same way, 0 means unrelated, −1 means opposite. Dividing matters because a long vector otherwise scores highly against everything.

That single formula powers semantic search, recommendations, deduplication, and the retrieval step in RAG: embed the question, embed every document, return the closest.

Where they sit in the model

  1. Tokenize. Your text becomes a list of token IDs.
  2. Embed. Each ID becomes a vector.
  3. Add position, because attention is order-blind.
  4. Run the stack. Every layer rewrites those vectors using context.
  5. Unembed. The final vector becomes a probability for every token.

Worth knowing: the embedding that goes in is context-free — "bank" starts identical in every sentence. By the time it has passed through the stack, it is not: the river and the money sense have been pulled apart by their neighbours. That is a large part of what the layers are for.

Practical notes

  • Count tokens, not words, for cost and context limits. Roughly words × 1.33.
  • Embeddings are model-specific. Vectors from different models are not comparable. Re-embed everything if you switch.
  • Normalise before comparing, unless your library already does.
  • Chunk sensibly for retrieval. One embedding per 200–500 words; a whole document averaged into one vector means nothing in particular.
  • Bias is baked in. If the training text associated two concepts, the geometry will too. Embeddings are a mirror of the corpus.

The short version

  • A token is a chunk of text, about ¾ of a word. Models read tokens, not letters — hence the spelling and arithmetic failures.
  • An embedding is a token as a vector: a point in space where nearness means similarity.
  • Similarity is a dot product, normalised into cosine similarity.
  • Input embeddings are context-free; the layers make them context-aware.
  • Embeddings only compare within one model.
← Archive— end of essay —Front page →