News about the success of a new Ebola vaccine may be too good to be true

Ebola is on the run: the number of cases dipped below 10 a week recently, and a few days ago investigators announced in the prestigious journal The Lancet that a new Ebola vaccine was “100% effective.”

In response, global health authorities are starting to sound a little giddy. “We believe that the world is on the verge of an efficacious Ebola vaccine,” said Marie Paule Kieny, the World Health Organization’s (WHO) assistant director-general for health systems and innovation (and an author on the study). “It could be a game changer.”

She’s right: this is wonderful news, and a great testament to human ingenuity. A genetically engineered hybrid of the benign vesicular stomatitis virus and the Zaire strain of Ebola, together called rVSV-ZEBOV, was tested in a multi-site clinical trial conducted amid a massive aid response in Guinea, one of the poorest countries in Africa. The scientific and logistical acrobatics required to pull this off boggle the mind.

Yet, for three reasons, we cannot know if the vaccine really worked, or how well. Those reasons are the lack of placebo comparison, the way the investigators diagnosed vaccine failure and the possibility of statistical flukes.

Reason #1: There was no placebo to test the vaccine against

In response to the challenging ethics of Ebola vaccine research, investigators studying rVSV-ZEBOV opted not for the standard placebo-controlled trial with which most vaccines are tested but instead for an innovative approach called “ring-vaccination.”

In the clinical trial, the investigators drew a “ring” of vaccination around people with known exposures to Ebola. Groups of people who were exposed to a person with Ebola virus disease, or to someone who had been in contact with someone else with Ebola virus disease, were invited to participate in the study. Researchers recruited 7,651 people who fit that profile and randomized them to receive immediate vaccination with rVSV-ZEBOV or vaccination that was delayed by 21 days.

Afterwards, investigators followed all study participants for several weeks to see if they developed Ebola virus disease. In the immediate vaccination groups, no one developed Ebola virus disease more than 10 days after vaccination, whereas 16 patients in the delayed vaccination group did. Because the number of cases of Ebola virus disease was so much lower (ie, zero) in the immediate vaccination group, the investigators concluded the vaccine was 100% effective.

This conclusion was reasonable, but it also could be wrong. Without a placebo comparison, it’s easy to get fooled. Perhaps people with deferred vaccination were more likely to engage in risky Ebola exposures, or those who developed Ebola virus disease in the immediate vaccination group were less likely to stay in study follow-up. The investigators did as good a job as possible to avoid these problems, but without a placebo arm in the study, we just cannot be sure.

Reason #2: The definition of vaccine failure isn’t failsafe

No cases of Ebola developed in the immediate vaccination group 10 days after vaccine was given. But nine cases of Ebola developed in the immediate vaccination group within 10 days of vaccination. Since the incubation period for Ebola virus disease is typically longer than 10 days, the investigators concluded that the nine cases of Ebola virus disease that developed within 10 days of getting the vaccine resulted from pre-vaccine exposure.

Again, this was a defensible conclusion, but it could also be wrong. Sometimes Ebola develops in less than 10 days, which means some or all of the nine people in the immediate vaccination group could have been infected with Ebola after being vaccinated. That means sometimes the vaccine may have failed to provide protection against Ebola, and we do not know exactly how often.

On the decline. New confirmed Ebola cases reported by WHO in 2015.Reuters

Reason #3: The possibility of statistical flukes

When investigators were following study participants to see if the vaccine prevented Ebola, the total number of Ebola cases was shrinking rapidly. That is good news for people living in West Africa but problematic for the study. With such a small number of total cases, comparisons between immediate and delayed vaccinees become harder to make with confidence, because statistical flukes can occur.

How easily could a fluke event have thrown off the results?

Let’s go back to those nine cases of Ebola in the immediate vaccination group. Imagine if just a few of them were cases of vaccine failure, and in fact, were cases where Ebola virus disease developed after vaccination. Further, what if a few immediate vaccinees lived in communities where infection control measures were better, coincidentally, than in communities where vaccination was delayed? Perhaps immediate vaccinees were less likely to encounter Ebola because burial practices were less risky in their communities, by chance, or there was a better supply of gloves and other protective equipment. If both easily imaginable circumstances occurred in just a few cases, vaccine efficacy could be much lower than 100% and our public health messaging would be considerably more muted.

The study investigators took meticulous steps to prevent each of these problems, and they argue persuasively that it is unlikely the protection they saw from rVSV-ZEBOV was entirely due to chance. I agree. The point is that it is easy to imagine ways the vaccine is far less than 100% effective, and even possible (if unlikely) the vaccine provided no protection whatsoever. The reasonable doubt that remains is why most vaccines are tested and then retested in placebo-controlled clinical trials so that the public can trust us when we say vaccines work.

Scientists at the National Microbiology Lab in Winnipeg, Manitoba, prepare an experimental Ebola vaccine for shipment to the World Health Organization in Geneva.Reuters/Public Health Agency of Canada/Handout

Despite need for more info, we aren’t likely to get it

In response to this uncertainty, Lancet editorialists were properly guarded about the vaccine. They wrote, “More data on efficacy are needed before it can be widely deployed.”

More data are needed – but we will probably never get more data. Ebola is burning out in western Africa. That means it is becoming increasingly impossible to test the efficacy of any intervention against Ebola. This is one reason that the clinical trial results for the rVSV-ZEBOV vaccine were released earlier than expected: the data safety and monitoring board for the trial recognized that with so few cases of Ebola, the trial could no longer acquire meaningful efficacy data. That means the Guinea rVSV-ZEBOV vaccine trial is likely to remain the best evidence we have for an effective Ebola vaccine for a long time.

Ordinarily, the next step would be to prove vaccine efficacy beyond a shadow of a doubt by comparing the vaccine to placebo in a large randomized clinical trial. To get this done, we would need to wait for the next time Ebola emerges in some remote village in Guinea or a neighboring country, and then mount the mother of all Ebola vaccine trials designed to prove that the magic bullet we seek is really in hand.

We will probably never take this step. Just as many scientists decried placebo-controlled studies amid the present Ebola epidemic, surely any future resurgence of Ebola cases will be accompanied by a similar hew and cry. In fact, the critique will be even more vehement: with a vaccine already touted as “100% effective,” how can we possibly deny it to people at risk of death in days? As a harbinger of this conversation to come, upon release of the rVSV-ZEBOV study results, the study data safety and monitoring board recommended and ultimately Guinean ethics authorities approved of immediate rVSV-ZEBOV vaccination for all people in the study region who were newly exposed to Ebola.

Are we lowering our standards for vaccines in the developing world?

In a time when we benefit from a panoply of effective vaccines, it is easy to assume all vaccines work well. Yet many vaccine candidates have failed. To list a few, there are failed vaccines against herpes simplex virus and Staphylococcus aureus and even two HIV vaccines that seemed to increase the risk of HIV infection. The reason we test vaccines in rigorous placebo-controlled trials is to avoid exposing the public to vaccine duds, or worse.

Does our well-founded fear of Ebola, and our equally legitimate fear of unethical placebo research in Africa, convince us that people in West Africa should be given a vaccine that hasn’t been proven effective using the same standards as vaccines given in the rich countries of the world?

It might be politically easier, and logistically expedient in the short term, to answer “yes.” But, in the long run, I fear saying “yes” will make it harder to deliver to low- and middle-income countries vaccines of the same quality as are used in rich countries.

That is why it is so important to know whether the Ebola vaccine was 100% effective, or 100% lucky. The good money is on a percentage somewhere in between, but in truth, I doubt we will ever know.

Previously Timothy P Lahey received funding from the NIH and private funders for research related to HIV immunology, and TB vaccines. None present a conflict of interest with the current article.