Why language models lose track of the middle
The idea
Modern language models can accept very long inputs, from tens of thousands to hundreds of thousands of tokenstoken A small chunk of text, often a word or part of a word, that a language model reads or writes as one unit.. Accepting text is not the same as using it well. In a paper first posted in 2023 and published in Transactions of the Association for Computational Linguistics in 2024, Nelson Liu and colleagues tested how well models find a relevant piece of information depending on where it sits in the input.
They used two tasks. In multi-document question answering, the model gets a question and 10, 20 or 30 documents, exactly one of which contains the answer; the researchers moved that document from first to last. In key-value retrieval, the model gets a long list of random keys and values and must return the value for one key. Across models including MPT-30B-Instruct, LongChat-13B, GPT-3.5-Turbo and Claude-1.3, performance was usually best when the relevant information was at the start or end and significantly worse when it was in the middle, even for versions built for long contexts. Plotted against position, accuracy formed a U shape.
A concrete example: GPT-3.5-Turbo, asked the questions with no documents, scored 56.1%. With 20 documents and the answer at position 9, near the middle, it scored 53.8%; with 30 documents and the answer at position 9 or 14, about 50.5% to 50.9%. In those positions, supplying the right document did worse than supplying none.
Why it might change a decision
The finding predicts that, for a model with this bias, the order of material in a prompt changes the answer even when the content is identical. In a retrieval system that hands a model a ranked list of documents, the paper’s case study also found that reader accuracy levelled off well before the retriever stopped finding more relevant documents: going from 20 to 50 retrieved documents improved GPT-3.5-Turbo by only about 1.5 points and Claude-1.3 by about 1. Whether it pays to retrieve more, or where to place the most important passage, is therefore something that can be measured for a given model rather than assumed.
How solid is this?
The original result comes from controlled benchmarkbenchmark A fixed set of tasks used to score and compare AI systems. experiments, not real user tasks, but it covered several open and closed models of the time and two different tasks. Some details were less uniform: an encoder-decoder modelencoder-decoder A model design with one part that reads the whole input and a second part that writes the output., Flan-UL2, was fairly robust to position when inputs stayed within its training length, and the U shape appeared only beyond it.
Later work broadly confirms the effect and suggests it can be reduced. Cheng-Yu Hsieh, Yung-Sung Chuang and colleagues linked it to a built-in U-shaped attention bias: models attend more to the beginning and end of the input regardless of relevance. Calibrating that bias improved retrieval-augmented generationretrieval-augmented generation A way of using a language model in which relevant documents are first looked up and then given to the model along with the question. by up to 15 percentage pointspercentage point The plain difference between two percentages, so a move from 40 to 45 percent is a rise of 5 percentage points. over existing methods. Shengnan An and colleagues argued the cause is training that never teaches a model that crucial information can be anywhere. Their synthetic training data produced FILM-7B, a version of Mistral-7B that retrieved reliably across its 32K window in probing tasks. Both are preprintspreprint A research paper shared publicly before, or without, formal peer review by a journal., each tested on particular models.
A broader benchmark, RULER, by Cheng-Ping Hsieh and colleagues, is a reminder that simple retrieval tests flatter models. Testing 17 long-context models on 13 tasks, they found most did well on basic needle-in-a-haystackneedle-in-a-haystack A test in which a single fact is hidden somewhere in a long text and a model is asked to find it. retrieval but declined sharply on harder tasks as inputs grew. Almost all claimed context sizescontext window The maximum amount of text a language model can take in and consider at once. of at least 32K tokens, yet only about half kept satisfactory performance at 32K. RULER measures length more than position, so it does not show directly whether the middle is still the weak spot.
The fair summary is that the positional effect is real and well replicatedreplication Running a study again, independently, to see whether the original result holds up. for the models Liu et al. tested; that it can be reduced; and that how much of it remains in the newest commercial models has not been established by the sources here. Claims that it is solved, or that it applies equally to every model, both outrun the evidence.
Sources
- Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 157-173. doi:10.1162/tacl_a_00638 (preprint: arXiv:2307.03172)
- Hsieh, C.-Y., Chuang, Y.-S., Li, C.-L., Wang, Z., Le, L. T., Kumar, A., Glass, J., Ratner, A., Lee, C.-Y., Krishna, R., & Pfister, T. (2024). Found in the middle: Calibrating positional attention bias improves long context utilization. arXiv preprint. arXiv:2406.16008
- An, S., Ma, Z., Lin, Z., Zheng, N., & Lou, J.-G. (2024). Make your LLMlarge language model A computer program trained on very large amounts of text to predict and produce written language. fully utilize the context. arXiv preprint. arXiv:2404.16811
- Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., & Ginsburg, B. (2024). RULER: What’s the real context size of your long-context language models? arXiv preprint. arXiv:2404.06654