What an LLM really is: tokens, embeddings and attention

The most useful thing you can understand about a language model is also the most uncomfortable one:

It does not know anything. It predicts the next piece of text.

It does not query a database. It does not reason in the sense we use the word. It has no opinion and no intent. It produces a probability distribution over the next token, picks one, appends it to the text, and repeats, fast enough to look like conversation.

That sentence sounds reductive. It is the key to everything that follows.

Token: the unit that sets price and limit#

The model does not see letters or words. It sees tokens, which are pieces of words.

Rule of thumb: in English, one token is about four characters; a hundred tokens are roughly seventy five words.

In Portuguese and Spanish it is worse. Tokenisers were trained mostly on English, so the same text in another language burns twenty to forty per cent more tokens. Literally: you pay more to say the same thing in your own language.

Three practical consequences:

Price. Billing is per input and output token, at different rates: output is usually several times more expensive. That changes the design: asking for a short structured answer is cheaper than asking for a long piece of prose.

Limit. The context window is measured in tokens, and everything counts: the instruction, the history, the retrieved document and the answer still to be generated.

Behaviour. Counting letters in a word is hard because the model does not see letters. Maths with large numbers fails because the number is split into arbitrary pieces. It is not stupidity; it is representation.

Embedding: meaning turned into geometry#

An embedding turns text into a vector of numbers (hundreds or thousands of dimensions) such that texts with similar meaning land close together in that space.

Close as measured by cosine similarity, which compares the angle between vectors.

What that unlocks: search by meaning. "How do I cancel my subscription" finds a document that talks about "ending your plan", with not a single word in common. Traditional keyword search would not find it.

What embeddings do not do well, and where people get disappointed:

→ Negation. "With interest" and "without interest" land close together, because the texts are nearly identical.

→ Identifiers. Product codes, invoice numbers, SKUs: embeddings are bad at these.

→ Domain jargon. A generic model does not know your company's internal vocabulary.

That is why, in practice, hybrid search (semantic plus keyword) almost always beats either one on its own.

Attention: the mechanism that made it work#

When producing each token, the model looks at all the previous ones and weighs which of them matter.

In the sentence "the dog that was in the yard barked", to decide who barked, the model has to connect "barked" to "dog" and not to "yard". Attention does that.

That mechanism, introduced in 2017 in the paper "Attention Is All You Need", is the reason for the jump in quality that led to today's models.

The cost: attention compares every token against every other one, which grows with the square of the context size. That is why long context is expensive and slow, and why there is so much research into efficient attention.

How a model becomes an assistant#

Three stages, and they explain a lot.

Pre-training. It reads an enormous amount of text and learns only to predict the next token. This is where it acquires knowledge and language. It is the stage that costs tens of millions of dollars.

Supervised fine-tuning (SFT). It is shown thousands of examples of good conversation: a question and the ideal answer. This is where it learns the assistant format.

Preference alignment (RLHF and variants). Humans compare pairs of answers and mark which is better. The model is tuned to produce the preferred kind. This is where it learns tone, refusal and helpfulness.

That explains why a base model knows a lot and is terrible at conversation, and why products built on similar models behave so differently. The last two stages are much of the difference.

1. 1Pre-trainingReads an enormous amount of text and learns only to predict the next token. This is where knowledge comes from. Costs tens of millions.
2. 2Supervised fine-tuningThousands of examples of good conversation. This is where the assistant format comes from.
3. 3Preference alignmentHumans compare pairs of answers. This is where tone, refusal and helpfulness come from.

Hallucination is not a bug#

This is the most important part.

The model invents confidently because it was trained to produce the most likely text. A plausible piece of text is more likely than a refusal. And there is no internal mechanism in it that separates "I know this" from "I am completing this".

Hallucination is a direct consequence of the training objective.

What reduces it, in order of effectiveness:

1. Give it the right context. That is what RAG is for.

2. Require a source citation. It makes invention verifiable.

3. Teach it to refuse. An explicit instruction plus refusal examples in the prompt. Without an example, the model does not learn the shape of a refusal.

4. Low temperature for factual tasks.

5. Check the output with deterministic code. If the model returned an id, verify that it exists.

What does not reduce it: politely asking it not to hallucinate.

The rule that sums it up: the model proposes, the code disposes.

Temperature and context window#

Temperature controls how far the choice of the next token strays from the most likely one. Near zero is as deterministic as it gets, still not fully reproducible, because of parallel execution details.

Extraction, classification and code want low temperature. Creative writing wants high.

The context window has grown a lot: today we have hundreds of thousands up to a million tokens. But a large context does not mean uniform attention: there is a documented effect of losing what sits in the middle. The model pays more attention to the beginning and the end.

Design consequence: instruction at the start, material in the middle, question at the end.

And cost grows with context. Stuffing a giant document into every call is expensive and makes quality worse. Retrieving the right passage beats sending everything, which is, once again, the argument for RAG.

What to take away#

Tokens set price and limit. Embeddings enable search by meaning, with known blind spots. Attention is the mechanism, and it is what makes context expensive. Hallucination is a consequence of the objective, not a defect.

And the architecture decision that follows from all of it: your job is not to write a pretty prompt. It is to decide what goes into the window, in what order, and what stays out.