AI / ML topic What Happens When You Hit Enter?

You type a question into a chatbot and press Enter. A beat later, words start streaming out, a few at a time, like someone typing fast. It feels like one action, but it’s a small pipeline, and each stage explains something you’ve probably noticed: why the first word takes a moment, why the rest flow quickly, and why the same question gets a different answer twice.

Text becomes numbers

A model can’t read letters. Your message is first chopped into tokens, common chunks of text learned from data with byte-pair encoding. As a rule of thumb for English, one token is about four characters, or three-quarters of a word. Each token becomes an ID number, and each ID is swapped for an embedding, a long list of numbers standing for its meaning.

Every word looks back at the words before it

Now the transformer does its work. Its main step is attention, which the 2017 paper Attention Is All You Need made the whole design: each token looks back at all the tokens before it and weighs which ones matter. In “we sat on the river bank”, bank pays attention to river and settles on the right meaning. A model repeats this through dozens of layers, each refining every token’s meaning a little further.

The output of all that is a single thing: a score for every token in the vocabulary, saying how likely each one is to come next. That’s all a language model ever produces: one next token.

The next word is a dice roll

The model doesn’t simply take the top-scoring token. It rolls weighted dice, which is sampling. Temperature reshapes the dice first: low values make the favourite almost certain, high values give long shots a real chance. OpenAI’s API documents it as a number from 0 to 2, where “higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic.”

Many systems also let you trim the unlikely tail before rolling, a trick called nucleus sampling. And even at temperature 0, answers can differ run to run. In 2025, researchers at Thinking Machines traced that to the server: how many other users’ requests are batched with yours changes the arithmetic very slightly, and one flipped token sends the rest of the answer somewhere new.

Then it does it again, and again

The chosen token is added to the text, and the whole thing runs again for the next one. That loop is why answers stream in word by word, and it splits serving into two very different phases.

First, prefill: the model reads your entire prompt in one parallel pass. That’s heavy math on many tokens at once. It’s most of the pause before anything appears, measured as time to first token. It’s also why a long pasted document slows the start of a reply, even when the reply itself is short.

Then decode: one token per step, each step reading the entire model from memory to produce a single word. Researchers describe it as “a compute-intensive prompt computation” phase followed by “a memory-intensive token generation” phase, which is why GPUs spend so much of decoding waiting.

The cache that makes it cheap

What makes decode affordable is the KV cache. It keeps each token’s attention data, so the model doesn’t reprocess the whole conversation for every new word. It isn’t small, either: for one 13-billion-weight model, the vLLM paper works out 800 KB per token, and up to 1.6 GB for a single long request. Hit Enter and watch it fill:

That cache is one reason a model’s context window has a limit, and one reason long conversations cost more. As of September 2026, Anthropic’s larger Claude models take 1 million tokens, and every one of them needs a slot.

Making it faster

Almost every serving trick targets one of these phases. Batching lets many users share each read of the model during decode. Quantization makes the model smaller to read.

Speculative decoding lets a small model guess several tokens ahead and has the big one check them in one pass, for 2 to 3 times the speed on the paper’s test model, with identical output. Some systems now run prefill and decode on different GPUs so the two stop getting in each other’s way.

So what happens when you hit Enter?

Your text becomes tokens, tokens become numbers, and the transformer turns them into odds for one next token. A weighted dice roll picks it, and the whole loop runs again, token after token, until the answer is done. The pause before the first word is prefill; the steady stream after it is decode, one memory-bound step per token. Every stage is something someone has to serve, scale and pay for.

References & further reading

In the order a request meets them.