Why making a model bigger can make it worse, then better again
The idea
For most of the twentieth century, statistics students learned one picture of model complexity. A model that is too simple misses real patterns. A model that is too flexible starts fitting the noise in its particular sample, and its error on new data climbs. Plot test error against model size and you get a U: the best model sits somewhere in the middle. This is the bias-variance trade-offbias-variance trade-off The classical idea that a model too simple misses real patterns while a model too flexible chases noise, so the best model sits in between., and it is why overfittingoverfitting When a model fits the quirks of the particular examples it was tuned on so closely that it does worse on new examples. is treated as the main danger of a large model.
Modern neural networks seemed to break the rule. They often have far more adjustable numbers than training examples, can fit their training sets perfectly, and still predict new data well. In a 2019 paper in PNAS, Mikhail Belkin, Daniel Hsu, Siyuan Ma and Soumik Mandal proposed a single curve covering both regimes. Test error first follows the familiar U. It peaks near the interpolation thresholdinterpolation threshold The model size at which a model first has just enough capacity to fit every training example exactly., the point where the model has just barely enough capacity to fit every training example. Then, as capacity keeps growing, error falls again, sometimes below the bottom of the original U. They called this double descentdouble descent A pattern where a model's error on new data falls, rises to a peak, then falls again as the model is made larger..
A concrete example from their paper: they fitted models built from random features to 10,000 handwritten digits from the MNIST dataset. As the number of features approached 10,000, test error rose sharply; the model at exactly that size was among the worst. With many more features than examples, error dropped and ended up lower than any classical model in the sweep. They saw a similar shape in two-layer neural networks and in random forests.
Their proposed mechanism is about choice. Below the threshold there is roughly one best fit. Just at the threshold there is exactly one fit that passes through every point, and it has to contort itself to do so. Past the threshold there are many perfect fits, and the training procedure tends to land on the smoothest one, the minimum-norm solutionminimum-norm solution Among all settings of a model that fit the training data perfectly, the one with the smallest overall size of its numbers, which tends to be the smoothest.. More capacity offers smoother options.
In December 2019 Preetum Nakkiran and five co-authors, including Ilya Sutskever, reported the same pattern in deep networks: ResNets and simpler convolutional networks on the CIFAR image datasets, and transformers on translation. They added two twists. Error could also rise and fall again over training time for a large model, and in a band of model sizes, more training data made results worse. In one translation setting, 18,000 examples gave higher test loss than 4,000.
Why it might change a decision
The finding bears on a specific practical move: increasing model size in small steps and stopping when validation error rises. Under the classical picture, a rising error means you have passed the sweet spot. Under double descent, a rise near the threshold can be a temporary bump, and a much larger model could do better. The testable behaviour is this: before concluding a model is too big, check whether the size at which error peaked is close to the size at which training error first reached zero. If it is, the classical reading may be wrong for that problem.
How solid is this?
The existence of the curve is well supported. It has been reported by separate groups across several model families and datasets, and these are controlled experiments in which model size is varied directly, not observational studies. In 2020 Marco Loog and colleagues published a letter in PNAS showing the phenomenon was not new: peaks of this kind were described for linear regression in 1989 and 1990 and for a linear classifierclassifier A model that sorts inputs into categories, such as flagged or not flagged. in the late 1990s. Effect sizeeffect size How large an effect is, as opposed to whether it exists at all. A result can be statistically reliable and still be too small to matter. varies a great deal, so there is no single number to report.
Three qualifications matter. First, the peak is often weak or absent on clean data. Nakkiran’s team found the sharpest peaks with deliberately corrupted labels, and both papers note that early stoppingearly stopping Ending a model's training before it finishes fitting the training data, at the point where its error on held-out data is lowest. or other regularisation can hide it. Second, the explanation is less settled than the observation. The minimum-norm account is proven for simple models, but for deep networks both papers call it a conjecture. Third, it is not universal. Loog and colleagues point to methods that show no peak at all.
The strongest competing reading is that double descent says more about untuned training than about large models. If careful regularisation often removes the bump, the classical intuition may still guide most decisions, and the popular phrase “bigger is always better” outruns the data. What the evidence does support is narrower: perfect fit to training data is not, by itself, a reliable sign that a model will do badly on new data.
Sources
- Belkin, M., Hsu, D., Ma, S. & Mandal, S. (2019). Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, 116(32), 15849-15854. doi:10.1073/pnas.1903070116; preprintpreprint A research paper shared publicly before, or without, formal peer review by a journal. arXiv:1812.11118
- Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B. & Sutskever, I. (2019). Deep double descent: where bigger models and more data hurt. arXiv preprint. arXiv:1912.02292
- Loog, M., Viering, T., Mey, A., Krijthe, J. H. & Tax, D. M. J. (2020). A brief prehistory of double descent. Letter, Proceedings of the National Academy of Sciences. arXiv:2004.04328