Why the AI answer stops in the middle

What’s actually happening

Two limits are involved and they are constantly confused. Getting them apart is most of the answer.

The context window is how much can be in one request: your document, the conversation, the system prompt and the reply, all together. The output limit is how much the model may generate in that one reply. They are different numbers, and the second is far smaller.

GPT-4 Turbo reads 128,000 tokens and writes at most 4,096. GPT-4o reads the same 128,000 and writes 16,384. Claude Haiku 4.5 reads 200,000 and writes 64,000.

Four thousand tokens sounds like a lot until you convert it. Ordinary English prose runs at roughly three quarters of a word per token — measured on this site’s own text with a published encoding — so GPT-4 Turbo’s entire reply budget is about three thousand words. A long article. Ask it for a full contract review and it stops halfway through, mid-sentence, without comment.

That is the ordinary case, and the vendors name it plainly. Anthropic’s stop-reason table lists max_tokens as “The response reached your max_tokens limit”, with the advice to “raise max_tokens or continue the response”. OpenAI reports the same event as a status of incomplete with incomplete_details giving the reason max_output_tokens.

There is a second, rarer case, and it is where the intuition most people have is simply wrong.

It is tempting to say the answer stopped because the request did not fit in the window. On current models that is not what happens. Anthropic documents the behaviour exactly:

On Claude 4.5 models and newer, if input tokens plus max_tokens exceeds the context window size, the API accepts the request. If generation then reaches the context window limit, it stops with stop_reason: "model_context_window_exceeded"

The oversized request is not rejected — it is accepted, and the generation runs until it hits the wall. Anthropic’s stop-reason table describes that outcome as “The response filled the model’s context window”, and tells you to “treat the response as truncated”.

A refusal on the way in is a third thing again, and it needs the input alone to be oversized: “If the input alone already exceeds the model’s context window, the API returns a 400 invalid_request_error (‘prompt is too long’) on every model.” That one never produces half an answer, because it never produces any. It is a different symptom entirely.

So the ranking of causes, from most to least likely behind a mid-sentence stop: the reply budget ran out; the reasoning budget ran out before visible text appeared; the whole window filled up during generation. Only the third has anything to do with your document being large.

What to do about it

Say continue. It is not a workaround, it is the documented remedy. The stopped text is still in the conversation, so the model resumes from it. Repeat as needed.

Ask for the work in parts, up front. “Sections one to three now, four to six next” produces complete sections instead of a truncated whole, and each part gets its own budget.

Beware of the tasks that inflate output. Full rewrites, translations of long text, line-by-line tables and “repeat the original alongside your version” all multiply the length of the answer while the budget stays fixed. Ask for the changes, not the full text plus the changes.

Read a blank reply as a budget signal, not a refusal. A long pause followed by nothing usually means the reasoning consumed the output budget. Ask a narrower question rather than repeating the same one.

Count what you are asking for. Token counter will tell you how long the thing you want back actually is — paste a comparable document and compare it with the limits above. For the input side, Context fit does the same against each model’s window.