Next-token prediction — how an answer is actually produced
In one paragraph
A model does not compose an answer and then write it down. At each position it produces a probability for every possible next piece of text, something draws one, and the whole thing repeats with that piece added. Fluency, variability and confident invention are all consequences of that loop — not separate features.
one prompt, two equally likely answers
recorded for gpt-4o-mini in OpenAI’s cookbook — the same distribution as the demo below
The recorded top three candidates after one prompt, with a temperature slider applied to them. At temperature 1 they read 49.6%, 49.6% and 0.8%; at temperature 2 the same three read 47.0%, 47.0% and 6.0%. A second recorded distribution, three words further into the same sentence, reads 97.1%, 2.3% and 0.7%. The model produces the ranking; something outside the model does the picking.
The loop
Text is produced one piece at a time. At each position the model assigns a score to every candidate in its vocabulary, and OpenAI states what those scores are:
A logprob is log(p), where p = probability of a token occurring at a specific position based on the previous tokens in the context.
Higher values, the same page notes, “suggest a higher likelihood of the token in that context”. Then a candidate is drawn, appended, and the model is run again on the longer text. That is the entire loop. There is no separate stage where an answer is planned, checked or looked up.
The demo above is one snapshot of that ranking, recorded by OpenAI rather than simulated by us. Two of its three candidates sit at 49.6% each — two continuations of the same sentence that the model considers equally good.
What the loop explains
Why the same prompt gives different answers. A near-tie is decided by a draw. OpenAI states that its APIs are “non-deterministic by default (which means model outputs may differ from request to request)”, and warns that even a fixed seed does not guarantee determinism.
Why answers diverge more than they should. Once a different piece has been drawn, every later position is conditioned on a different history. A fork at word four is not a variation, it is a different text.
Why invented facts sound so convincing. The ranking measures fit, not existence. A citation is a sequence of individually ordinary choices — a surname, a year, a journal — with nothing in the loop asking whether the result is real.
Why the model guesses instead of hedging. That part is not architecture but incentives. Research published by OpenAI in 2025 argues that models “sometimes guess when uncertain, producing plausible yet incorrect statements instead of admitting uncertainty”, and names the reason:
the training and evaluation procedures reward guessing over acknowledging uncertainty
Technical detail: what the recorded distribution does and does not show
The fixture behind the demo is an output cell from OpenAI's own cookbook, kept with its model, prompt, capture date and request parameters. It lists the top three candidates out of the full vocabulary, so the recorded probabilities sum to 99.3% rather than 100%. Rescaling over those three shows the shape of the choice, not the whole distribution, and the demo says so on screen.
The temperature slider is honest arithmetic over recorded numbers, not a live call: the softmax is applied here, in your browser, to logprobs that were captured once. Nothing on this site simulates a model.
Common questions
- 1 Does the model know what it is going to say before it says it?
- Not in the sense of having a draft. Each position is decided in turn, conditioned on everything already written. That is why an answer can start down one path and commit to it — the earlier words are input to the later ones.
- 2 Are the probabilities a measure of truth?
- No. They score how well a piece of text fits the context, which is a statement about language, not about the world. A false sentence that reads naturally can be ranked far above a true one that reads oddly.
- 3 What does temperature actually change?
- It rescales the ranking before the draw. Low temperature concentrates the probability on the leader; high temperature flattens the distribution and gives outsiders a real chance. It does not make the model more or less correct, only more or less predictable.