What it means when scientists say their results are 'significant'

This article is part of the series This is research, where we ask academics to share and discuss open access articles that reveal important aspects of science. Today’s piece looks at statistics, and how to interpret them for meaning in the real world.

Let’s face it, scientific papers aren’t exactly page turners. They are written by scientists, for scientists, and often in a language that seems to only vaguely resemble English.

And perhaps one of the most daunting aspects of a scientific paper is the statistics (“stats”) section.

But what do stats really mean in the real world? Here’s an example from leukaemia research to help you break it down.

Read more: Our survey found 'questionable research practices' by ecologists and biologists – here's what that means

Strength of results

Stats are key to good research – they help researchers determine whether the results observed are strong enough to be due to an important scientific phenomenon.

As a research student I would always look for the magic number which indicates statistically significant differences in my experiments: most people agree this number to be 0.05 (you may see this in a paper written as p < 0.05).

When comparing two outcomes or groups in a scientific study, achieving a statistical significance value of less than 0.05 means there’s less than 5% chance the results are due to chance alone. Or, put another way, if you repeated that same experiment 20 times under the same conditions, you’d get the same result at least 19 times.

But the focus on looking for statistically significant differences can blind us to the bigger picture. As I advanced through my scientific training I learnt to look for biologically significant differences.

Read more: The curious case of the missing workplace teaspoons

Biological significance

Biological significance addresses the question of whether the statistical difference actually means anything in terms of a real outcome, like a disease. Can the result explain how the disease is caused? Does it provide a new avenue to treat the disease? Basically, is it relevant?

A recent paper published in the journal Leukaemia will help explain my point. The paper looked at why some people are able to stop their treatment for chronic myeloid leukaemia without the cancer coming back, while in others the cancer relapsed.

The key finding of this study was that patients who did not relapse had a higher proportion of natural killer cells compared to patients that did relapse.

Natural killer cells are a type of immune cell that controls viral infections and tumours. So, the more cells there are to kill the cancer, the less likely the cancer was to relapse – makes sense!

This finding has the potential to guide doctors in seeing which patients are likely to remain cancer-free after stopping treatment. This is definitely biologically significant.

Read more: My cancer is in remission – does this mean I'm cured?

Not so relevant

Another result from the same paper (Figure 3a if you want to click through to the data) shows a statistically significant difference in a sub-type of natural killer cells (called adaptive natural killer cells). But is this difference biologically relevant?

At this stage there is little evidence of a role for adaptive natural killer cells in the context of leukaemia. Also, the difference between the groups is relatively small, with a large variation within the groups (there are large error bars on the graph).

These factors make it more likely that the differences may be due to the mathematics involved in the statistical test rather than a biological effect. As with any new finding, time and further studies will be vital in working out whether this result actually means anything.

Act like an expert

So how do you pick if the statistical differences have biological value? Being a highly trained expert in the field certainly helps.

Another way to determine if the findings in a paper have biological relevance is to look for other papers that show similar results. If a result is “real” it should be found by other scientists who will build on it and publish more papers.

This means there will be lots of papers for you to read and apply your new-found passion for statistics.

The open access research paper for this analysis is Increased proportion of mature NK cells is associated with successful imatinib discontinuation in chronic myeloid leukemia.