How big should a language model be for the compute you have?
The idea
Training a large language modellarge language model A computer program trained on very large amounts of text to predict and produce written language. means making a budget decision before you start. For a fixed amount of computing power, you can train a bigger model on less text, or a smaller model on more text. Scaling lawsscaling law A formula, fitted to experiments, that predicts how a system's performance changes as it is made larger or given more resources. are attempts to answer that trade-off with formulas fitted to many smaller training runs.
In January 2020 Jared Kaplan and colleagues posted a paper on arXiv showing that a model’s test loss, a measure of how badly it predicts the next word, fell as a smooth power lawpower law A relationship in which one quantity changes in proportion to another raised to a fixed power, so it looks like a straight line on a log-log plot. as model size, dataset size and compute increased, across many orders of magnitude. Their models ranged from 768 to 1.5 billion parametersparameter One of the adjustable numbers inside an AI model that training tunes; model size is usually counted in them., trained on up to 23 billion tokenstoken A small chunk of text, often a word or part of a word, that a language model reads or writes as one unit. of web text. From those curves they derived an allocation rule: optimal model size should grow roughly as compute to the power 0.73, and data only as 0.27. In their words, when the model grows eightfold, the data needs to grow by only about five times. The practical message was to build very large models and train them on comparatively modest amounts of data.
In March 2022 Jordan Hoffmann and colleagues at DeepMind reached a different answer. Training more than 400 models, from 70 million to over 16 billion parameters on 5 to 500 billion tokens, and analysing them three separate ways, they found that model size and training data should grow in roughly equal proportion. Their exponents for model size ranged from 0.46 to 0.50, against Kaplan’s 0.73. By that reckoning, many large models of the time were undertrained. To test it they built Chinchilla: 70 billion parameters trained on 1.4 trillion tokens, using the same compute as their earlier 280 billion parameter Gopher, which had seen 300 billion tokens. Chinchilla outperformed Gopher and several other larger models on a wide range of benchmarksbenchmark A fixed set of tasks used to score and compare AI systems..
Why it might change a decision
The two rules make different, checkable predictions. Under Kaplan’s allocation, a lab with ten times more compute should mostly buy parameters. Under Hoffmann’s, it should split the increase between parameters and data, and a model trained on about 20 tokens per parameter should beat a larger model trained on fewer tokens for the same compute. The Chinchilla versus Gopher comparison was one direct test of that prediction.
How solid is this?
Both papers are large empirical studies, not single runs, but each is one lab’s set of experiments, and the laws are fitted curves rather than derived from theory. Hoffmann and colleagues themselves noted slight curvature in their results and significant uncertainty when extrapolating many orders of magnitude. Their head-to-head test rested on essentially one pair of large models.
They explained the disagreement mainly by method: Kaplan’s team used a fixed number of training tokens and learning ratelearning rate A setting that controls how large an adjustment a model makes to itself after each batch of training examples. schedule for all models, and most of their runs were small. A 2024 study by Tomer Porian and colleagues reproduced Kaplan’s law and traced the gap to three other factors: how the cost of the model’s final layer was counted, the length of the warmup period, and optimiser settings that were not adjusted with scale. With those corrected, their results agreed closely with Chinchilla. They also found that careful learning rate decay was not essential, contrary to a hypothesis Hoffmann and colleagues had offered.
The Chinchilla paper has its own problem. In 2024 Tamay Besiroglu and colleagues reconstructed the data behind Hoffmann’s third method from the paper’s figures and refitted it. The published parameters fitted the data poorly and were inconsistent with the other two methods; the reported confidence intervalsconfidence interval A range of values, calculated from data, that is meant to show how precisely a quantity has been estimated. were so narrow that they would have needed over 600,000 experiments, where Hoffmann’s team likely ran fewer than 500. The published fit implied about 70 tokens per parameter; the refit gave about 20, matching the other two methods and Chinchilla’s actual training. The replicators report that a lead author of the original paper attributed the narrow intervals to a high loss scale, a fitting setting which, the replicators argue, made the optimiser stop early. So one of three estimates was flawed, and the headline conclusion survived.
The popular shorthand that “20 tokens per parameter” is simply optimal outruns the data in another way. It minimises training compute only. Nikhil Sardana and colleagues showed that once the cost of running a model for many users is counted, it pays to train smaller models for longer, and quality kept improving in their tests up to 10,000 tokens per parameter.
Sources
- Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020). Scaling laws for neural language models. arXiv preprintpreprint A research paper shared publicly before, or without, formal peer review by a journal. 2001.08361. arxiv.org/abs/2001.08361
- Hoffmann, J., Borgeaud, S., Mensch, A., et al. (2022). Training compute-optimalcompute-optimal Describing the split between model size and amount of training text that gives the best result for a fixed amount of computing. large language models. arXiv preprint 2203.15556. arxiv.org/abs/2203.15556
- Besiroglu, T., Erdil, E., Barnett, M., & You, J. (2024). Chinchilla scaling: A replicationreplication Running a study again, independently, to see whether the original result holds up. attempt. arXiv preprint 2404.10102. arxiv.org/abs/2404.10102
- Porian, T., Wortsman, M., Jitsev, J., Schmidt, L., & Carmon, Y. (2024). Resolving discrepancies in compute-optimal scaling of language models. arXiv preprint 2406.19146. arxiv.org/abs/2406.19146
- Sardana, N., Portes, J., Doubov, S., & Frankle, J. (2023). Beyond Chinchilla-optimal: Accounting for inference in language model scaling laws. arXiv preprint 2401.00448. arxiv.org/abs/2401.00448