How re-analysing the data of scientific research can change the findings

Science, many people believe, is kept in check by scientists reviewing each other’s work. This has recently extended to re-analysis of data to see if results can be replicated, and has overturned important findings in medicine, economics, and sociology.

We re-analysed an influential randomised controlled trial of deworming in Kenyan schools. We found that even for a randomised controlled trial – lauded as the most robust method to identify impact – there are aspects of analysis and reporting where re-analysis can shed new light.

Re-analysis is a powerful tool in the review of important studies, and should be supported with data made available by researchers and with adequate funding. The publication of the replicationresults by the International Journal of Epidemiology is a watershed. To our knowledge it is very unusual for an international journal to publish a re-analysis of an already published trial.

The original authors have responded to the our re-analysis, which is also set to be published in the journal today.

Randomised controlled trials

Randomised controlled trials were primarily developed in medicine, agriculture and psychology. They evaluate programme effects by comparing a group of individuals randomly allocated to get the service with a group who were not, using data collected some time after the intervention started.

Recently, the number of randomised controlled trials has exploded in development economics. This has caught people’s attention. Economists doing trials have appeared in the Time Magazine’s 100 most influential people list, won major awards and shifted the development discourse.

Much credit for sparking this movement goes to Michael Kremer and Ted Miguel’s paper on deworming effects on school attendance in Kenya.

A quarter of the world’s population is infected with intestinal worms, including hookworm, roundworm, schistosomiasis and whipworm. It is particularly prevalent among school-aged children in developing countries. The effects of intestinal worms are especially pronounced in Africa, where nearly half of the total disease burden is due to infectious and parasitic diseases, including worm infections.

Using a trial design now known as a “stepped-wedge”, Kremer and Miguel reported the benefits of a programme of health education and mass administration of deworming drugs.

There were several results that proved especially influential. First, they found that the deworming programme reduced school absenteeism in treatment schools by one-quarter. It was cheaper than other ways of increasing school participation.

It also appeared to improve school attendance in schools where no children were treated. This is plausible since if there are fewer worms around, other children may be less likely to get infected.

The results led many, including the Copenhagen Consensus in 2012, to conclude that deworming is one of the most cost-effective development interventions. The organisation’s Deworm the World initiative received strong independent endorsement and support from governments.

A fresh look at the data

A health worker dispenses albendazole tablets to a child on National Deworming Day in Kisumu, Kenya.Evidence Action, Courtesy of Photoshare

The decision to re-analyse this research was based on broader methodological questions. There was not a particular interest in deworming. It was a highly influential study published by economists in a format unfamiliar to epidemiologists.

We suspected that the disconnect between how highly it has been praised by other economists and how it had been previously reviewed by epidemiologists working on Cochrane reviews reflected differences in analysis and reporting between the two fields. The re-analysis was intended to help bridge the differences between epidemiology and economics.

We re-ran the original analyses using files sent to us by Miguel and his team. While few mistakes were found, the authors erroneously reported evidence of effect of the intervention on anaemia. But there was no evidence of this in reanalysis.

There were also coding errors which meant that the total effect on school attendance in the paper was overestimated. There was little evidence that attendance was improved in untreated schools.

Then we re-analysed the data using the methods that we, as epidemiologists, would typically use for analysing trials of this kind. Our analysis showed some evidence that the intervention was associated with greater school attendance. However, we add the important caveat that this conclusion is at high risk of bias.

Bias often comes into a study because of ways participants are selected for measurement, including if they are lost after first being contacted, and how good those measurements are. Bias can creep in at all stages of the research process, and usually despite the best efforts of the researchers.

Our concern about the bias was triggered by uncertainty about how the school attendance data was collected. There is a possibility that the way in which the data was collected might have been systematically different in schools that did and did not get the deworming treatment.

Although it is not known if this is the case, the amount of missing data and some peculiar patterns found in the re-analysis gave us some cause for concern. And since neither anaemia, weight-for-age, nor height-for-age appear to have been affected by the treatment, we have no other evidence for any intermediate steps on a causal chain between deworming and school attendance that might have offset some of our concerns about bias.

The future of re-analysis

This deworming re-analysis exercise should not directly affect deworming programmes. Another group is updating a systematic review and will consider the wider evidence. But it shows that systematic reviews are the best way to inform policy, and avoid hype.

The explosion of randomised trials in development is exciting. Trials in this area could be improved by adopting standards from the medical field. This is already happening: pre-analysis plans are becoming more common, and have been championed by economists at Berkeley – a movement led by, among others, Ted Miguel.

Adopting – or adapting – the medical field’s standards for trial reporting may lead to more transparent conclusions and reveal risks of bias. These standards address the fact that when it comes to randomised trials, especially in low income settings, the quality of data and measurements are just as important as the statistics subsequently applied to them.

We anticipate that re-analysis will become more common, will improve transparency, accountability, and strengthen the literature that policymakers use to base decisions that affect the health and happiness of millions of people around the world.

The authors do not work for, consult, own shares in or receive funding from any company or organization that would benefit from this article, and has disclosed no relevant affiliations beyond the academic appointment above.