Why AI misses things in the middle of a long document

What’s actually happening

A long input is not read the way you read: there is no cursor moving from line to line, and no point at which the model has “got to page 40”. The whole input is available at once, and the model weighs all of it while producing each word. Availability, though, is not the same as use.

The clearest measurement of the gap is Lost in the Middle, published in TACL in 2023. The setup is deliberately simple: give the model a set of documents where exactly one contains the answer, ask the question, and then move the answer document to a different position and ask again. Nothing else changes — not the question, not the number of documents, not the amount of text.

The result, in the authors’ words:

performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models

The demo above is those published numbers redrawn, not a picture of the paper. With twenty documents, the same model answered 75.8% correctly when the answer sat first and 53.8% when it sat tenth — a gap of twenty-two points produced entirely by moving one paragraph.

Two things follow that are worth separating. First, the effect is about position, and it is not a bug that a vendor patch removes. Independent testing by Chroma across eighteen newer models found that:

models do not use their context uniformly; instead, their performance grows increasingly unreliable as input length grows

Second, the vendors now name the phenomenon themselves. Anthropic’s own documentation states that “as token count grows, accuracy and recall degrade, a phenomenon known as context rot”, and that this “makes curating what’s in context just as important as how much space is available”.

That is the practical shape of it. A long document does fit. Fitting is the easy part.

What to do about it

Ask about one section at a time. Paste chapter four and ask about chapter four. A short input where everything is relevant beats a long one where the relevant part is buried at position seventeen.

Put the question next to the material. If the answer must come from a particular passage, quote that passage in the message. You are choosing the position instead of hoping.

Do not trust a single sweep. “Find every clause about termination” over eighty pages is exactly the task the measurements above say is unreliable. Run it section by section, or run it twice and compare the two lists — the differences are where to look.

Ask it to quote before it concludes. Requiring the exact sentence it is relying on turns an unverifiable summary into something you can check in ten seconds, and it fails loudly when there is nothing to quote.

Know the size you are working with. Context fit measures a document, including a PDF, in tokens and against each model’s window — useful less for the limit than for seeing how much material the question is competing with.