Regression to the mean, or why perfection rarely lasts

Statistics is a useful tool for understanding the patterns in the world around us. But our intuition often lets us down when it comes to interpreting those patterns. In this series we look at some of the common mistakes we make and how to avoid them when thinking about statistics, probability and risk.

Have you ever experienced the perfect evening out? The weather was great, you got the best table in the house at your favourite restaurant, the food was delicious and the wine superb, and the conversation was sparkling.

Then have you made the mistake of trying to repeat the experience and ended up disappointed? This is because your perfect evening was due to a series of chance events that all fell in your favour. A great experience is like tossing a coin and getting a long run of heads – unusual and difficult to repeat.

When you try to repeat a perfect experience, at least one thing is likely to be imperfect the second time around. The couple at the next table are loud and boorish, the waiter gets your order wrong, your jokes fall flat, and so on.

Happily, it works both ways. So if you’re forced to repeat a terrible experience, it’s likely that it won’t be so bad the second time around.

This phenomenon is called “regression to the mean” or “reversion to mediocrity”, which sums up how unusual events are likely to be followed by more typical ones.

The polymath Sir Francis Galton coined the term when he noticed that tall parents tended to have children shorter than them, whereas short parents often had children who were taller than themselves.

For a parent to be unusually tall, the genetic coin had to be “heads” many times in a row. Repeating that feat of chance for their children is not impossible, but it is unlikely.

Is that a trend?

Regression to the mean is driven by chance, and so it occurs wherever chance occurs, which means it occurs almost everywhere. It is prevalent in sport and can explain the “manager of the month curse” in football. This award is usually won by managers who have had four or more wins in a row, often because of a combination of skill and luck. When the luck runs out, the “curse” strikes.

Regression to the mean will even occur in this article as unusually long sentences will tend to be followed by shorter ones. Check if you don’t believe me. It occurs in the published literature on regression to the mean, as years with many published papers on the phenomenon tend to be followed by years with fewer papers.

This article will itself cause some regression to the mean if it spikes interest in the Wikipedia page, but that interest will inevitably wane.

Regression to the mean is mostly harmless, but it becomes a problem when the change it creates is misinterpreted.

For example, imagine you ran a hospital and were told that hospital-acquired infections were five times higher than average last month. A colleague tells you they know the cause and it can solved by using more prophylactic antibiotics.

You agree and in the following month you’re told that prophylactic antibiotic use is through the roof and infections have come down. Your mind makes a causal connection and you’re now convinced of the need for widespread prophylactic antibiotics, a potentially dangerous connection given that the unusual infection rate could have been due to chance events.

Now your hospital budget will be tighter because of the costs of using more antibiotics, and you’re contributing to serious problem of antibiotic resistance.

Making sham treatments look good

Regression to the mean is unwittingly exploited by quacks who often see patients when they are at their lowest. As many diseases have a natural ebb and flow, seeing patients when they are at their worst is the best time to exploit regression to the mean, because any treatment will appear to cause improvements in enough patients to make it look broadly effective.

Telling the difference between regression to the mean and a real change can be difficult. A chronically ill patient may have a very bad day, but is that the early warning of a downward trajectory or just a blip due to a random cluster of events, such as a bad meal, poor sleep, or an ill-judged sprint for the bus?

Gathering more data using watchful waiting can be useful, as once a clear pattern emerges in a patient’s well-being, it is less likely to be the random ups and downs of regression to the mean.

Regression to the mean is everywhere. Being aware of it might help you avoid overreacting to unusual events. However, if you didn’t find this article interesting or useful then why not read another one on The Conversation – the chances are you’ll enjoy it more.