Why AI answers get worse the longer the chat gets

What’s actually happening

The obvious explanation is that the chat filled up and the beginning fell out. Sometimes that is what happened — and if so, the fix is a different one. But the effect you are seeing usually starts long before the window is anywhere near full, and the measurements say why.

The clearest result comes from Context Length Alone Hurts LLM Performance Despite Perfect Retrieval, published in Findings of EMNLP 2025. The authors removed the usual excuse — that the model failed to find the relevant part — by handing it the relevant part. Their finding:

even when models can perfectly retrieve all relevant information, their performance still degrades substantially (13.9%–85%) as input length increases but remains well within their claimed context lengths

They then pushed harder. The drop survived replacing the irrelevant text with blank space, and survived masking it so the model could attend only to what mattered. Length itself was the cost.

Independent testing points the same way. Chroma ran eighteen models — including current frontier ones — and concluded:

model performance degrades as input length increases, often in surprising and non-uniform ways

The same report finds that “models do not use their context uniformly; instead, their performance grows increasingly unreliable as input length grows”. Non-uniform is the word to keep: this is not a smooth slope you can plan around.

Vendors no longer argue with it. Anthropic’s own documentation now says that “more context isn’t automatically better”, that “as token count grows, accuracy and recall degrade, a phenomenon known as context rot”, and draws the practical conclusion that this “makes curating what’s in context just as important as how much space is available”.

The demo above is the length axis of an older, carefully controlled study. Hold the position of the answer fixed at the very first document and grow the pile around it: 72.1% correct out of ten documents, 68.6% out of twenty, 66.9% out of thirty. Same question, same position, more text. The numbers are from 2023 models and are not a measurement of anything you can use today; the direction is what has kept being reproduced.

Put that next to what a long working session actually looks like — an hour of half-abandoned attempts, pasted fragments, corrections and dead ends, all of it still in the request — and the slow decline stops being mysterious. You have been growing the input all afternoon.

What to do about it

Start a new chat at each change of task. Not when things break — when the task changes. The transcript of how you got here is rarely input to where you are going.

Carry a summary, not the transcript. Ten lines of decisions and constraints, written by you, beat forty exchanges of context. Ask the model to draft that summary before you close the old chat, then read it before pasting it.

Make it recite before it reasons. The EMNLP authors propose exactly this as a mitigation: prompting the model to recite the retrieved evidence before attempting to solve the problem, which turns a long-context task into a short-context one. On their benchmark it improved GPT-4o by up to 4% over an already strong baseline. In a chat, it is one sentence: “First quote the parts you are relying on, then answer.”

Prune what you paste. Three relevant pages beat eighty pages of which three are relevant — and the eighty are actively costing you, not merely wasting space.

Watch for the early sign. Instructions being followed loosely, earlier constraints reappearing as questions, summaries drifting towards the generic. That is the moment to restart, not an hour later.

Know the size you are carrying. Context fit counts a document or a transcript in tokens and shows it against each model’s window, which turns “this is getting long” into a number.