Was there ever a Hawthorne effect at Hawthorne?
The idea
Between November 1924 and October 1927, engineers at Western Electric’s Hawthorne Works near Chicago changed the lighting in rooms where women assembled relays and wound coils, and recorded how much they produced. The version taught in textbooks for decades is striking: output went up when the light was raised, and also went up when it was lowered. The conclusion drawn was that the light hardly mattered. What mattered was that the workers knew they were being studied.
That interpretation, later broadened through further studies at the plant into the 1930s, became the Hawthorne effectHawthorne effect The claim that people change how they behave because they know they are being studied or watched.: the idea that people change their behaviour because they are being observed. According to McCambridge and colleagues, the term itself first appeared in a methodology textbook in 1953. It is now a standard caution in trial design, medicine and education.
The awkward part is that for most of that time nobody could check the illumination claim. The records were thought to have been destroyed.
Why it might change a decision
If observation alone reliably lifts performance, then any study that watches its participants will overstate the effect of whatever is being tested, and a control groupcontrol group The people in a study who do not receive the treatment, so that the treated group can be compared against them. that is watched just as closely is the main defence. The specific prediction from the illumination story is testable: output should jump immediately after an experimenter changes something, whatever the change, and that jump should not appear on comparable days without a change.
How solid is this?
The illumination story itself rests on accounts of a single set of poorly controlled factory experiments, and the original data were not analysed statistically until 2011.
That year Steven Levitt and John List, in American Economic Journal: Applied Economics, reported finding the records on microfilm at the University of Wisconsin-Milwaukee and Harvard’s Baker Library. They digitised output for three rooms, measured per room rather than per worker. Their verdict on the textbook pattern was blunt: the famous descriptions, they wrote, “prove to be entirely fictional.”
The main reason is mundane. Every lighting change began on a Monday, after the plant’s Sunday rest day, and output was higher on Mondays than on Fridays or Saturdays. But Monday output was equally high whether or not the lights had changed that day. In their regressions the coefficients for the days of change were not statistically significant. A fall in output after April, once read as a response to the experiment, also appeared in later years and in rooms no longer being studied, which points to seasonal patterns instead.
Levitt and List did not report a clean null. They found some subtler signs consistent with an observation effect: output in experimental rooms grew faster than at the plant as a whole, was higher while manipulations were under way, and responded more to artificial light changes than to natural ones. That last effect was small, under 1 percent for a one standard deviation change in light. They concluded that the design was weak enough to limit what anyone can learn from it.
The illumination studies were not the only evidence. Stephen Jones, in the American Journal of Sociology in 1992, analysed data from the Relay Assembly Test Room, a later and more famous Hawthorne study, and found “slender or no evidence” of a Hawthorne effect once other factors were accounted for.
The broader question, whether taking part in research changes behaviour, is separate from what happened at Hawthorne. Jim McCambridge, John Witton and Diana Elbourne’s 2014 systematic reviewsystematic review A study that searches for every piece of research on a question using set rules and then summarises what they found together. in the Journal of Clinical Epidemiology found 19 eligible studies, all but one in health sciences: 8 randomised trials, 5 quasi-experimental studies and 6 observational evaluations. Most showed some evidence of an effect, but the authors judged substantial bias likely and the studies very varied in method and in what they meant by the term. Across 14 studies with yes-or-no outcomes they reported an unweighted pooled odds ratioodds ratio A number comparing how likely an outcome is in one group with how likely it is in another, where 1 means no difference. of 1.17, while warning that the review combined “apples and oranges” and that pooled results needed caution. Their conclusion was that participation does affect behaviour, but that little is securely known about its conditions, mechanisms or size. They proposed the label “research participation effects” instead.
The strongest competing explanation for the original pattern is confoundingconfounding When a third factor that moves alongside the thing being studied makes it look as if one causes the other.: the day of the week, the season, and managerial changes at the plant all moved alongside the experiments. The popular version, that output rose with every change of light, outruns the data. The general claim that being studied can alter behaviour is plausible and partly supported, but its size and mechanism remain uncertain.
Sources
- Levitt, S. D., & List, J. A. (2011). Was there really a Hawthorne effect at the Hawthorne plant? An analysis of the original illumination experiments. American Economic Journal: Applied Economics, 3(1), 224-238. doi:10.1257/app.3.1.224
- Jones, S. R. G. (1992). Was there a Hawthorne effect? American Journal of Sociology, 98(3), 451-468. doi:10.1086/230046
- McCambridge, J., Witton, J., & Elbourne, D. R. (2014). Systematic review of the Hawthorne effect: New concepts are needed to study research participation effects. Journal of Clinical Epidemiology, 67(3), 267-277. doi:10.1016/j.jclinepi.2013.08.015