Why the same prompt gives different answers

What’s actually happening

At every position, the model produces a score for each possible next token. OpenAI describes what those scores are: “A logprob is log(p), where p = probability of a token occurring at a specific position based on the previous tokens in the context”, and higher values “suggest a higher likelihood of the token in that context”.

Note what that is and is not. It is a ranking. It is not a decision. Something has to turn the ranking into one actual word, and that something draws at random — weighted by the scores, but at random.

The demo above uses a distribution OpenAI published in its own cookbook, so nothing here is simulated. Rescaled over its top three candidates at temperature 1, two of them sit at 49.6% each. Two different, perfectly sensible continuations of the same sentence, separated by nothing. Run that prompt twice and you get two answers, not because anything changed, but because a coin was flipped.

Then it compounds. Once a different word has been drawn, everything after it is conditioned on a different history — the third sentence of the second run is not a variation of the third sentence of the first, it is a continuation of a different text. Small forks near the start produce answers that read as entirely different approaches.

The temperature slider shows the other half. Temperature rescales the ranking before the draw: lower makes the leader dominate, higher flattens everything out. In the recorded distribution, the third candidate is a 0.8% outsider at temperature 1 and a 6.0% real possibility at temperature 2. The model did not change its mind. The dice did.

Vendors say this openly. OpenAI’s guide to the seed parameter opens with it:

Even fixing the seed is a best effort, not a promise — the same page warns that “Determinism is not guaranteed, and you should refer to the system_fingerprint response parameter to monitor changes in the backend”, and that responses can still differ when everything visible matches.

So the honest summary is that a prompt is not a function. Sending it twice is running an experiment twice, and experiments have spread.

What to do about it

Save the output, not the prompt. If a particular answer was good, keep the answer. Re-running the prompt that produced it is not a way of getting it back.

Ask for several versions in one go. “Give me three subject lines” makes the variability work for you and keeps them all in front of you at once, rather than discovering variant two after you have lost variant one.

Pin what must not vary. Specify the structure, the length, the field names, the order. What you constrain in the prompt stops being subject to the draw; what you leave open is redrawn every time.

Use disagreement as a signal. Ask the same factual question three times in fresh chats. Stable answers were well supported by the training data; answers that wobble were not. This is one of the cheapest checks available, and it costs nothing but a minute.

Do not read the variation as a change of mind. A second, contradictory answer is not the model correcting itself or being evasive. It is a second draw from a distribution that was never narrow.