Why AI invents sources and citations

What’s actually happening

Text is produced one token at a time. At each step the model ranks what could come next and something picks from that ranking. OpenAI’s own description of the mechanism is arithmetic rather than metaphor:

A logprob is log(p), where p = probability of a token occurring at a specific position based on the previous tokens in the context.

Look at the demo above. One prompt, one position, three candidates — and two of them are almost exactly as likely as each other. That is the whole machine: a ranking over what fits, and a choice made from it.

Now apply that to a reference. “According to” is a very ordinary continuation of a factual claim. A surname is an ordinary continuation of “according to”. A year between 2015 and 2023 is ordinary after a surname. A journal name that goes with that topic is ordinary after the year. Each step is individually reasonable, and the assembled result is a citation that no library holds.

At no point in that sequence is there a step that asks whether the reference exists. There is no lookup, no index, no list of real papers being consulted — unless the product you are using has been wired to go and fetch one, which is a separate mechanism bolted on top.

The obvious question is why the model does not simply say it is unsure. Research published by OpenAI in 2025 answers it in terms of incentives rather than architecture — language models:

sometimes guess when uncertain, producing plausible yet incorrect statements instead of admitting uncertainty

And they do so because “the training and evaluation procedures reward guessing over acknowledging uncertainty”. The authors compare it to a student on an exam. A blank answer scores zero; a guess sometimes scores.

That framing is worth holding onto, because it predicts the pattern you actually see. The fabrications cluster exactly where a confident-looking answer is cheap and verification is expensive: page numbers, court case names, standards numbers, quotations, API parameters, statutes. The rarer and more specific the fact, the more likely you are looking at a fluent guess.

What to do about it

Make it fetch, then open the link. If your app has web search or document retrieval, use it and click through. A reference the model retrieved has a URL you can check; a reference it produced from memory has a shape you recognise and nothing behind it.

Ask for the claim and the source separately. Get the argument first, decide whether it is worth sourcing, then ask for the source. Requesting “with citations” in the same breath invites the model to furnish decoration for text it has already committed to.

Verify by the identifier, not by the title. Search the DOI, the ISBN, the case number. Titles of non-existent papers are often near-misses of real ones and survive a casual search; identifiers either resolve or do not.

Treat a refusal as a good outcome. “I am not certain there is a published source for this” is the answer the grading incentives punish, which is precisely why it is worth more than a tidy reference list.

Never pass an unchecked citation onward. A fabricated source in your own document becomes your error the moment you send it. The check takes half a minute; the correction does not.