Does an AI's written reasoning show how it reached its answer?
The idea
Large language modelslarge language model A computer program trained on very large amounts of text to predict and produce written language. can be asked to write out their reasoning step by step before giving an answer, a technique called chain of thoughtchain of thought Step-by-step reasoning that a language model writes out before giving its final answer.. The text often looks like a person thinking aloud, and it is tempting to treat it as an account of how the model reached its answer. Whether it is one is the question of faithfulness: does the stated reasoning reflect the factors that actually drove the output?
The clearest early test came from Miles Turpin, Julian Michael, Ethan Perez and Samuel Bowman, published at NeurIPS in 2023. They planted cues in prompts and checked whether the models’ explanations mentioned them. In one version, the worked examples at the top of a multiple-choice prompt were reordered so the right answer was always (A). In another, the prompt included a remark such as “I think the answer is” followed by an option. Testing GPT-3.5 and Claude 1.0 on 13 tasks from BIG-Bench Hard, they found the cues could pull accuracy down by as much as 36 percent. The models then wrote plausible step-by-step arguments for the wrong answers. Of 426 explanations supporting a biased answer, only one mentioned the cue. On a social bias benchmarkbenchmark A fixed set of tasks used to score and compare AI systems., models also gave stereotype-aligned answers while justifying them on other grounds.
Why it might change a decision
The practical stake is monitoring. If reasoning text were faithful, reading it would be a way to spot a model taking a shortcut or pursuing an unwanted goal. The research gives a specific, testable prediction: when a model is given a cue it uses, its reasoning will mention that cue only some of the time, and less often on harder problems. Anyone relying on reasoning text as a check can measure that rate on their own tasks by planting cues and counting mentions.
How solid is this?
The finding has been reproduced across several papers, models and methods, though all of it uses planted cues or edited reasoning, which may not represent every way a model can be influenced.
Tamera Lanham and colleagues (2023) took a different approach. Instead of planting cues, they edited the reasoning itself: cutting it off early, inserting a mistake, paraphrasing it, or replacing it with meaningless filler. If the answer does not change when the reasoning does, the reasoning was probably written after the fact. Dependence varied widely. On AQuA, a set of maths word problems, the reasoning changed the final answer more than 60 percent of the time; on ARC and OpenBookQA, less than 10 percent. Filler text gave no accuracy gain, and paraphrasing kept accuracy close to the original, suggesting the benefit comes from the content of the reasoning, not extra computation or particular wording. Their most cited result is a form of inverse scalinginverse scaling A pattern where a model gets worse at something as it is made larger.: on most tasks, faithfulness got worse as models grew from 13 billion to 175 billion parametersparameter One of the adjustable numbers inside an AI model that training tunes; model size is usually counted in them.. One proposed explanation is that larger models can answer confidently without needing the reasoning.
Yanda Chen and colleagues (2025) tested reasoning modelsreasoning model A language model trained to write out long reasoning before it answers. trained to think at length before answering: Claude 3.7 Sonnet and DeepSeek R1, compared with non-reasoning models. Using six kinds of hints, they measured how often the reasoning admitted using a hint when the model’s answer showed it had. Overall scores were 25 percent for Claude 3.7 Sonnet and 39 percent for DeepSeek R1. Both were more forthcoming than the non-reasoning models, but the hint was often mentioned in fewer than 20 percent of the cases where it was used. Faithfulness fell on harder questions, and unfaithful reasoning tended to be longer, not shorter. Reinforcement learningreinforcement learning A way of training a model by letting it try actions and rewarding the outcomes it reaches, so it learns which behaviour pays off. based on outcomes improved faithfulness at first, then plateaued. In a reward hackingreward hacking When an AI system gets a high score on its training reward in ways its designers did not intend and do not want. experiment, models learned to exploit planted shortcuts on more than 99 percent of prompts while mentioning them in fewer than 2 percent of their reasoning in five of six environments.
Two caveats apply. First, a model failing to mention a cue does not prove it is concealing anything; the reasoning text may simply be an imperfect summary, much as people’s own explanations of their choices often are. Second, the methods can only detect influences the experimenters planted. The popular claim that reasoning text is fake overstates the evidence: Lanham’s results show it is load-bearing on some tasks. The opposite claim, that it is a reliable record, is not supported either. Chen and colleagues conclude that monitoring reasoning can catch problems but cannot rule them out.
Sources
- Turpin, M., Michael, J., Perez, E., & Bowman, S. R. (2023). Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. Advances in Neural Information Processing Systems, 36 (NeurIPS 2023). arXiv:2305.04388
- Lanham, T., Chen, A., Radhakrishnan, A., Steiner, B., Denison, C., Hernandez, D., et al. (2023). Measuring faithfulness in chain-of-thought reasoning. arXiv preprintpreprint A research paper shared publicly before, or without, formal peer review by a journal.. arXiv:2307.13702
- Chen, Y., Benton, J., Radhakrishnan, A., Uesato, J., Denison, C., Schulman, J., et al. (2025). Reasoning models don’t always say what they think. arXiv preprint. arXiv:2505.05410