Why guessing a batter's average gets better when you look at other players
The idea
Suppose you want to estimate several quantities at once: the true batting skill of eighteen baseball players, the infection rate in thirty-six towns, the quality of a dozen hospitals. The obvious method is to use each one’s own data: a player who hit .400 in his first 45 at-bats gets an estimate of .400.
In a paper presented at the Third Berkeley Symposium in 1954 and 1955 and published in 1956, the statistician Charles Stein proved that this obvious method is inadmissibleadmissible Said of an estimation method when no other method is at least as accurate in every possible situation and more accurate in some. whenever you estimate three or more means together. That means another estimator exists that has lower expected total squared errorsquared error The difference between an estimate and the true value, multiplied by itself, so that large misses count much more than small ones., no matter what the true values are. In one or two dimensions the ordinary estimator cannot be beaten this way; from three upward it always can.
In 1961 Stein and Willard James gave an explicit version, now called the James-Stein estimatorJames-Stein estimator A method that estimates several averages at once by pulling each one toward their overall mean, which lowers the total error when there are three or more.. It takes every individual estimate and pulls it toward the grand mean of all of them, by an amount that depends on how spread out the estimates are compared with the noise in each one. The more the individual values look like noise around a common centre, the harder it shrinks.
The result disturbed statisticians because the quantities do not need to be related. Erich Lehmann, who had worked on the same question, later said he was “stunned with disbelief” when Stein told him, according to a 2012 account by Lawrence Brown and Linda Zhao.
A concrete example. Bradley Efron and Carl Morris, writing in the Journal of the American Statistical Association in 1975, took eighteen major league players’ batting averages after their first 45 at-bats of the 1970 season and used them to predict each player’s average over the rest of that season. Stein’s rule put about 21 percent of the weight on each player’s own record and about 79 percent on the group mean (on a transformed scale). Its total squared prediction error was 5.01, against 17.56 for the raw early averages. It did better for 15 of the 18 players.
Why it might change a decision
Any time you rank or compare many noisy measurements, the extreme ones are partly extreme because of luck. Shrinkage turns that into a specific procedure: before acting on the top and bottom of a list (the best-scoring school, the worst-rated surgeon, the fastest-growing product), estimate how much of the spread is noise and pull each value toward the average by that amount.
A testable consequence follows. If you use first-month figures to predict the rest of the year for a set of salespeople, shops or players, the shrunken predictions should have lower total error than the raw ones, and the gap should be largest when early samples are small. Efron and Morris also showed the same logic working on toxoplasmosis rates across 36 cities in El Salvador, where a version allowing unequal sample sizes cut error by roughly two thirds.
How solid is this?
The core claim is a mathematical theorem, not an empirical finding, so the usual worries about replicationreplication Running a study again, independently, to see whether the original result holds up. do not apply to it. Stein’s 1956 proof has stood for seventy years and is the basis of a large literature on shrinkage and empirical Bayesempirical Bayes A family of methods that use the spread of many related measurements to decide how much to trust each single one. methods, the same family of ideas behind modern techniques for stabilising many noisy estimates.
What the theorem promises is narrower than the popular slogan “averaging with others always helps.” First, it guarantees lower total squared error, not better estimates for every individual. In the baseball data Roberto Clemente, whose early average of .400 was the highest, was pulled down to .290 while he went on to hit .346, so the shrunken estimate was slightly worse than his raw one. Efron and Morris note that the rule permits “considerably increased risk to individual components” and proposed a modified version that limits how far any single estimate can move.
Second, the proof assumes roughly normal errors with known, equal variance. Real data break these assumptions, which is why Efron and Morris transformed the batting averages first. Third, while the guarantee holds even for unrelated quantities, the size of the gain depends on how close the true values really are to each other. For eighteen players with similar skill it was large; for quantities with nothing in common it would be close to zero.
The strongest competing view is not that the theorem is wrong but that total squared error is the wrong target in many settings. If the cost of misjudging one exceptional case is high, a method that sacrifices that case for the group average may be the worse choice.
Sources
- Charles Stein (1956). “Inadmissibility of the usual estimator for the mean of a multivariate normal distribution.” Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, Vol. 1, 197–206. University of California Press. PDF via DTIC
- Bradley Efron and Carl Morris (1975). “Data analysis using Stein’s estimator and its generalizations.” Journal of the American Statistical Association 70(350), 311–319. PDF
- Lawrence D. Brown and Linda H. Zhao (2012). “A geometrical explanation of Stein shrinkage.” Statistical Science 27(1), 24–30. arXiv:1203.4737