Has Australia really had 60,000 undiagnosed COVID-19 cases?

A preliminary study, posted online this week by researchers at the Australian National University and elsewhere, estimates 71,000 Australians had COVID-19 by mid-July — 60,000 more than official number of cases diagnosed by that stage.

The study involved testing 2,991 elective surgery patients in ten hospitals across four states, to see whether they had antibodies against SARS-CoV-2, the virus that causes COVID-19.

The study initially found 41 positive patients (1.4%), but then adjusted for the false positives that would arise due to the imperfect specificity of the antibody test, which the researchers estimate would produce 11 false positives for every 1,000 tests. This yielded an estimated prevalence of 0.28% — or eight “true” positives from the 2,991 people sampled.

The researchers then extrapolated this estimate, including its uncertainty parameters, to the Australian population as a whole. They ultimately concluded the number of Australians with SARS-CoV-2 antibodies — and who have therefore presumably been infected with COVID-19 — is somewhere between zero and 181,050, and most likely about 71,000.

This begs two main questions: should this alter our view on how best to contain the spread of COVID-19, and are there any limitations to the study that we should be aware of?

Let’s begin with the latter question. Here are four key things to consider when interpreting the results.

1. False positives

In countries with very low COVID-19 rates, such as Australia, the key requirement of an antibody test is to be highly specific — that is, to avoid false positives. This is even more important than being highly sensitive (avoiding false negatives).

The antibody test used in the new study reportedly has a specificity of 98.9%, and a sensitivity of 100%. This means, for every 1,000 tests, we can expect 11 false positives and no false negatives.

Imagine a place with high prevalence of the virus, such as New York City, where roughly 20% of people are estimated to have had COVID-19. A sample of 1,000 would, on average, contain 200 COVID-19 positive people, of whom the test would correctly identify all 200, with no false negatives. Among the remaining 800 people it would find around nine people positive who were actually negative, giving an estimated prevalence of 209 out of 1,000, or 20.9% — which is close to the true figure.

Now imagine a sample of 1,000 Australians, with a COVID-19 prevalence of, say, 0.2%.

Just two people in this sample would correctly test positive, and among the remaining 998 we would expect the test to deliver about 11 false positives. This gives an estimated prevalence of 13 out of 1,000, or 1.3%, which is several times higher than the true figure.

Read more: Australia's coronavirus testing rates are some of the best in the world – compare our stats using this interactive

Even if you revise your estimate to account for the expected false positives, as the authors did, we can see how hard it is to estimate low prevalences accurately. The small number of real cases is liable to be lost in the noise.

2. Sample size

A larger sample size could provide improved precision. The small sample size is why the study’s estimated range is so wide. In fact, it stretches all the way down to zero, even though we know there can’t possibly have been zero COVID-19 cases in Australia. But no matter the size of the study group, the false positive problem never really goes away as long as the prevalence is low.

3. Testing method

One solution would have been to retest the samples with currently available commercial antibody tests with specificities of 99.9%. This would have offered a way to overcome the problem with false positives.

The suspicion that the sample included a substantial proportion of false positives is supported by the fact only one COVID-19 positive patient had contact with a known COVID-19 case, and none of those who tested positive had reported any COVID-19-like illness.

Read more: Why can't we use antibody tests for diagnosing COVID-19 yet?

4. Extrapolation

There are also questions over how reliably the results can be generalised to the entire Australian public. The study involved people undergoing elective surgery, who may have had different risks of exposure to the virus.

It is hard to say from the available data whether any adjustment was made for variables such as age, sex and state of residence when extrapolating to the wider Australian population.

So what can we say for sure?

What can we determine from this study about the number of people exposed to COVID-19 in Australia? Unfortunately, without a much larger sample, wider sampling of the population, and a more reliable test, we know little more about the prevalence of exposure to COVID-19 than we already did.

This means it would be unwise to use these new findings to claim COVID-19 is any less dangerous or deadly than we thought.

Rather than take these estimates at face value, what we really need is more comprehensive testing of the prevalence of SARS-CoV-2 antibodies, including studies that track this prevalence over time.

In any case, Australia can consider itself fortunate to have low enough case numbers that the issues of false positives becomes a major caveat in interpreting studies such as this. Sadly, in many other places, false positives are buried in a landslide of genuine COVID-19 cases.

Editor’s note: Ian Cockburn, one of the study’s lead authors, told The Conversation false positives are indeed more likely to be a significant factor when trying to estimate low prevalence rates, but described the study as a “best estimate” based on two separate statistical analyses, which both arrived at the same result. He added the research team plans to use further statistical methods to check the study’s results before it is accepted for full publication.

He said the study sample “is not a perfect cross-section” of the population, and the ideal study size would be 6,000-10,000 people, but obtaining blood samples from the general population poses significant logistical and cost obstacles.

He added it can be difficult to verify commercial companies’ claims to have antibody tests with higher specificities, and that patients who register a false positive may also test false positive with another test if it works in the same way.

Correction: this article was amended on September 21, 2020, to correct an arithmetical error in the calculations of hypothetical false positives. The original version said a sample of 1,000 people with 200 true positives and a 1.1% false positive rate would yield 11 false positives. The correct figure is about nine false positives ((1,000-200) x 0.011 = 8.8).