Harvard-linked study finds reasoning models get stuck on their first few words

AI reasoning models are supposed to think step by step. A new paper suggests they mostly start those steps the same way, and that habit makes them harder to steer.

Researchers found that two cheap tricks, banning the usual opening words and adding an example, improved how well reasoning models followed instructions by 34 to 41 percentage points. For a field that spends heavily on bigger models, that is a large gain from tweaking a few words at the start.

What the researchers found

The paper is titled “The First Few Tokens Matter: Understanding Chain-of-Thought Controllability.” It was posted to arXiv on October 29, 2026, as arXiv:2610.40939, and runs 20 pages with 15 figures.

The authors include Harvard student Hugh Van Deventer and colleagues. They are affiliated with Harvard and the Cambridge Boston Alignment Initiative.

Their focus is chain-of-thought, or CoT. That is the visible “scratchpad” reasoning a model writes before giving a final answer. The study pays particular attention to large language models such as the OpenAI o1 series.

The core finding is something the authors call entropy collapse. It shows up in the first tokens a model produces right after its opening thinking tag.

A quick translation helps. A token is a chunk of text, roughly a word or part of one. Entropy, in this context, measures how varied the model’s choices are. High entropy means many plausible next words. Low entropy means the model is effectively locked onto a few.

So entropy collapse means that at the very start of its reasoning, the model keeps reaching for the same small set of familiar opening tokens. That narrowness matters because the opening shapes everything after it. If a user asks the model to reason in a certain style or include certain information, the model’s default opener can pull it back toward its usual script.

Two small fixes, one big jump

The researchers tested interventions aimed squarely at those first few tokens. Two stood out.

The first is prefilling. Here, the user supplies the opening of the model’s reasoning, even as little as a single token, instead of letting the model choose.

The second combines two moves. The researchers banned the common default opening tokens, then added in-context examples showing what the desired reasoning should look like.

Together, these strategies raised compliance by 34 to 41 percentage points across the models tested. Compliance here means how often the model’s reasoning actually followed the instructions it was given.

The gains were not uniform across every type of control, though. The study separates controllability into different capabilities.

Addition and style controllability covers cases where a model is asked to add information or change how its reasoning is expressed. These saw the clear improvement.

Suppression is different. It covers asking a model to avoid specific information in its reasoning. Here, the effects of the interventions were variable.

That split is telling. The authors suggest these distinct capabilities may need different evaluation approaches rather than being lumped into one score.

What this means

The most practical takeaway is cost. The interventions described are low-cost adjustments made at inference time, meaning when the model is being used, not when it is being trained.

The uneven suppression results are the part worth watching. Many oversight scenarios depend on a model avoiding certain content, not just adding it. If suppression does not respond as consistently to these tricks, then measuring it separately, as the authors suggest, becomes important. Treating all controllability as one number could make a model look more steerable than it really is in the cases that matter most.