Why a language model can know who Tom Cruise's mother is but not who her son is
The idea
Ask a large language modellarge language model A computer program trained on very large amounts of text to predict and produce written language. “Who is Tom Cruise’s mother?” and it will usually say Mary Lee Pfeiffer. Ask “Who is Mary Lee Pfeiffer’s son?” and it often cannot say. In a 2023 paper, Lukas Berglund, Owain Evans and five co-authors called this the reversal cursereversal curse The finding that a language model trained on a fact stated one way round, such as 'A is B', often cannot recall it when asked the other way round.: a model trained on a statement of the form “A is B” does not automatically learn “B is A”.
Their celebrity test asked GPT-4 about 1,573 child and parent pairs. It identified the parent 79 percent of the time and the child about a third of the time. That gap alone is hard to interpret, because a famous person’s name appears in far more text than their parent’s. So the authors ran a cleaner experiment. They invented facts that appear nowhere online, such as “Daphne Barrington is the director of A Journey Through Time”, and fine-tunedfine-tuning Further training of an already trained AI model on a smaller, chosen set of examples to change how it behaves. GPT-3 and Llama models on them.
The models learned the trained direction. Asked the reverse (“Who directed A Journey Through Time?”), the 175-billion-parameterparameter One of the adjustable numbers inside an AI model that training tunes; model size is usually counted in them. GPT-3 scored 0.0 and 0.1 percent exact matchexact match A way of scoring an answer that gives full credit only if it is completely right and nothing otherwise. on the two versions of the task. The probability it assigned to the correct name was no higher than for a random name. The failure held across model sizes and families. When the fact was placed directly in the prompt instead of in the training data, models could reverse it easily: what fails is the stored knowledge, not the ability to reason about a sentence in front of them.
A common explanation is the way these models are trained: they learn to predict the next tokentoken A small chunk of text, often a word or part of a word, that a language model reads or writes as one unit. given the ones before it. Learning to continue “Daphne Barrington is” with “the director of…” adjusts what comes after Daphne’s name. Nothing in that update requires the name to come out when the film is the starting point.
Why it might change a decision
The finding suggests a check anyone can run. If you rely on a chatbot to recall a fact about a less-documented entity, a person, product, regulation or gene, ask the question in both directions. The prediction is that a model will more often answer correctly when you start from the entity that the source texts usually start from, and that giving it the relevant text in the prompt will close most of the gap. For anyone building a question-answering tool on their own documents, the same logic favours supplying the documents at question time over relying on fine-tuning alone to store facts, because facts learned in only one phrasing may be retrievable from only one side.
How solid is this?
The core result is not a one-off. Berglund and colleagues’ paper is a preprintpreprint A research paper shared publicly before, or without, formal peer review by a journal. (first posted September 2023, revised in 2024) built on controlled fine-tuning experiments, where the facts are invented so that no pre-existing text can leak the answer. Two further lines of work found the same thing with different designs.
Zeyuan Allen-Zhu and Yuanzhi Li trained models from scratch on synthetic biographies in their “Physics of Language Models: Part 3.2” paper, published at ICLR 2025. They report that “inverse knowledge search”, such as recovering a person from one of their attributes, was “virtually 0%” regardless of how the question was prompted. Performance only improved when the training data itself presented the information in the reversed order.
Olga Golovneva, Allen-Zhu, Jason Weston and Sainbayar Sukhbaatar at Meta then tested a fix in a 2024 preprint. Their “reverse trainingreverse training Training a language model on each text both in its normal order and with its words reversed, so that facts are seen from both directions.” adds a reversed copy of each training text, keeping names intact. They report that this largely removes the failure on reversal tasks, and that compute-matched models did as well or better on standard benchmarksbenchmark A fixed set of tasks used to score and compare AI systems.. That result comes from one group and has not, as far as the sources here show, been independently reproduced.
What is less settled is how much the curse matters in practice. The clean zero comes from facts seen in exactly one phrasing. Real training data often states famous facts many ways round, which is presumably why GPT-4 got some reverse questions right at all. The celebrity numbers also mix direction with frequency, since the 79 and 33 percent figures compare questions about famous names with questions about obscure ones; the paper’s own figure reports 28 percent for the child direction, a reminder that the exact number depends on scoring choices. The strongest competing reading is therefore that the curse is a sharp effect under narrow conditions, and that its real-world size depends on how varied the text about a given fact is. None of the papers measures that directly for current commercial models, and newer models trained with different data or methods may behave differently.
Sources
- Berglund, L., Tong, M., Kaufmann, M., Balesni, M., Stickland, A. C., Korbak, T. and Evans, O. (2023, revised 2024). “The Reversal Curse: LLMs trained on ‘A is B’ fail to learn ‘B is A’.” arXiv preprint 2309.12288. https://arxiv.org/abs/2309.12288v4
- Allen-Zhu, Z. and Li, Y. (2025). “Physics of Language Models: Part 3.2, Knowledge Manipulation.” International Conference on Learning Representations (ICLR 2025); first posted to arXiv in 2023. https://proceedings.iclr.cc/paper_files/paper/2025/hash/d5494c8747276d3cdb2598e5617de89d-Abstract-Conference.html
- Golovneva, O., Allen-Zhu, Z., Weston, J. and Sukhbaatar, S. (2024). “Reverse Training to Nurse the Reversal Curse.” arXiv preprint 2403.13799. https://arxiv.org/abs/2403.13799v3