How to read results from COVID vaccine trials like a pro

It’s been a busy week or so for news about COVID vaccines. First we heard preliminary clinical trial results from the Pfizer vaccine, then the Russian Sputnik V vaccine. This week, we heard about the Moderna vaccine. All these results were shared with the media, ahead of being peer reviewed and published in a journal.

As we expect preliminary results from more vaccine trials to be released in the coming weeks and months, it’s important to understand what’s behind these announcements, what news reports don’t tell us, and what researchers don’t yet know.

This can help us identify good news when we see it, be more critical of news reports, or delay our judgement until we have more information.

Read more: We may have to accept a 'good enough' COVID-19 vaccine, at least in 2021

1. Does the news report tell me what type of trial it is?

At this stage of the pandemic, trial results making the headlines are generally the interim results of late-stage clinical trials, known as phase 3. This is when a vaccine is given to thousands of people and tested for how well it works and whether it’s safe (more on these issues later).

In these trials, volunteers are randomised into two study arms, the vaccine arm (people who get the actual vaccine), and the placebo arm (people who get the placebo, usually an inert substance, such as a saline injection). However, some vaccine trials use vaccines against other diseases as the placebo.

So, ideally, media reports should mention how the vaccine results compare with the placebo or the comparator vaccine.

Read more: Explainer: how do drugs get from the point of discovery to the pharmacy shelf?

Before the vaccine gets to this stage it will have successfully completed smaller trials (phase 1 and 2). Often, clinical trial phases are combined. So you could have results from a trial that combines phases 1 and 2, or phases 2 and 3.

2. Does the media report mention safety?

As vaccines are mainly tested on healthy volunteers, it is extremely important to demonstrate the vaccine is safe.

Side effects (also called adverse events) are reported to an independent committee — usually with two or more experts in immunology and medicine as well as a biostatistician. It’s one of the jobs of this data monitoring committee to receive and examine reports of adverse events, and to look at interim results to determine whether the trial should continue.

Sometimes, if safety concerns are raised, a trial is temporarily halted while the committee investigates. This is what happened with the University of Oxford/AstraZeneca vaccine trial, which has since recommenced.

So any media report should mention how many people are affected by side effects, the type of side effects (common/rare, serious/minor), whether they were in people in the vaccine or placebo arm of the trial, and whether the data monitoring committee is investigating. Not all these details are available to the public.

Read more: Halting the Oxford vaccine trial doesn't mean it's not safe – it shows they're following the right process

3. Does the media report mention how well the vaccine works?

Trial outcomes are measured at one or more interim time points, and at the end of the trial. This is another factor the data monitoring committee oversees.

For instance, the committee has rules about vaccine efficacy it applies part-way through the trial to work out whether the trial proceeds. So a rule might be something like “For the trial to continue, vaccine efficacy must be at least 60% after 25% of subjects have completed the trial”.

The types of results making the headlines currently come from this type of interim analysis. In other words, the committee will have assessed the results so far and will have given the trial a green light to proceed.

No phase 3 clinical trial has yet reported the full analysis from tens of thousands of study participants, but this will happen over the next few weeks.

Vaccine efficacy

Vaccine efficacy describes how well the vaccine offers protection against the target disease. The formulae and calculations can get quite complicated, so I will only give a simple example here.

One measure is based on the “attack rate”, which is the proportion of the people in the trial diagnosed with COVID-19. We measure the attack rate in the vaccine arm and the placebo arm separately, then divide one by the other to give the “attack rate ratio”. We then subtract the attack rate ratio from 1 to get one measure of vaccine efficacy.

For example, if 5% of the vaccine arm are diagnosed with COVID-19, while 40% of the placebo are diagnosed, then the attack rate ratio is (5%/40%) or 0.125 or 12.5%. That gives a vaccine efficacy of 87.5% (100% - 12.5%).

Read more: Pfizer vaccine: what an 'efficacy rate above 90%' really means

Immune response

Some vaccine trials report how well the immune system responds (immunogenicity). For example, the University of Oxford/AstraZeneca trial has reported the antibody response as well as several other measures of immunogenicity.

Some trials only report on immunogenicity. This allows the trial to be smaller, shorter, and less expensive than vaccine efficacy trials, as they use immunogenicity as a surrogate for vaccine efficacy.

Although efficacy is the preferred endpoint for vaccine trials, some regulating authorities accept evidence of immunogenicity to authorise a vaccine.

Vaccine effectiveness

Vaccine effectiveness describes how well the vaccine offers protection against the target disease in the real world, rather than in a controlled clinical trial. Vaccine trials usually include healthy volunteers, but often don’t tell us how well the vaccine works in children, elderly people, or those with compromised immune systems.

Reported vaccine efficacies of 90-95%, as we’ve heard recently, may sound impressive. However, under real-world conditions, the vaccines are likely to offer much less protection in some population groups.

Read more: 5 ways our immune responses to COVID vaccines are unique

4. What else do I need to know?

Current trials are reporting whether or not a vaccine prevents COVID-19 (in other words, symptoms), not whether it prevents the infection itself.

However, a recent media report about the Pfizer vaccine says it is likely to prevent 50% of infections, as well as 90% of symptomatic COVID-19.

If the vaccine has 90% efficacy, then 10% of vaccinated people could still get the symptomatic disease. We would hope these people would have a much milder illness, but we don’t know if this is the case.

We also don’t know how long immunity lasts or if there are any long-term side effects.

All we can do now is wait with patience for the full phase 3 trial results to come in over the next few weeks.