Explainer: the ins and outs of peer review

If you are at all familiar with the operation of the Intergovernmental Panel on Climate Change (IPCC) you will know that, while the various authors are (unpaid) professionals of one sort or another with their own research programs, the IPCC itself neither commissions nor does research. Its conclusions are based solely on the information available in the published peer-reviewed scientific literature. What does that mean?

Every research paper that appears in print and/or online will have been read and critiqued (generally with complete anonymity) by at least two scientific peers. This mechanism was first put in place by the Royal Society in the 17th century, and has endured.

Sometimes, if the reviewers are at odds, a third or even fourth opinion will be sought. Though the tradition with law journals, for example, may be that the process is double blind – with neither the reviewers nor the authors being identified – the single-blind tradition in science means we generally know who wrote the paper, but can only speculate about the critics of our wonderful manuscript.

That may sound onerous but, in practice, it’s not hard to get almost anything published at some level in what’s broadly styled as the peer-reviewed scientific literature, especially if it is well written and gives the appearance of having been done properly.

One checkpoint most editors will look for is that the paper comes from a university or research institute with some substance and that the author is not using, say, a gmail account. That doesn’t mean that the article should be immediately rejected, but it provides some measure of upfront quality evaluation.

Academic employers have standards and, in a formal sense, most research grants are awarded to the institution, not to the individual researcher.

The list

What do I mean by “published at some level”? There’s an A list of top journals – for instance, those with the word “nature” somewhere in the title – down to an E list or beyond to which researchers can drop if they are desperate to have an article appear in print and it has been rejected as it slid down the prestige ladder.

The leading journals, like Nature and Science, are read by a broad range of people from the physical to the biological sciences, while Nature Immunology and Nature Geoscience are obviously more specialised though of high stature.

In clinical medicine, the New England Journal of Medicine, The Lancet and the Journal of Clinical Investigation are very prestigious, while learned society publications like The Journal of Immunology and the Journal of Virology are thought of as “journals of record” for sound science within their particular specialty – or trade, if you like. Some of my most-cited papers are in these trade journals.

And there are journals for climate change sceptics. Editors of that mindset will, for instance, select appropriately friendly reviewers. Otherwise, the intervention of a serious scientist sceptic who makes a substantial and well-supported case will generally be welcomed by one or other leading journal.

The reasons are obvious. Informed critique and the debate that results are illuminating and interesting to read. Those of us who are outsiders both learn from, and are entertained by, reading a lively discussion. There’s nothing better than a good fight between intelligent protagonists! Then, nobody – and that includes journal editors – wants to persist with an erroneous view, so the alternatives must be aired.

Up for review

What of the anonymous, unpaid reviewers who work hard and do their best? Most are under a lot of pressure for all sorts of reasons, and they are likely to make the biggest effort with an article submitted to a high-profile journal.

With a demanding and detailed critique for a paper that does seem to have basic merit, the back and forth between authors, editors and reviewers asking for changes and/or new experiments can take a year or more. A problem with such a protracted process can be that, through a mix of exhaustion and dilution, what was a clear message will be largely obscured by the time the article appears.

Peer review should also be seen for what it is: an agreement between the editors, reviewers and authors that the article is now fit for general consumption by a quality audience, not a validation of infallibility, a concept that doesn’t exist in science anyway!

What publication following peer review actually says about a piece of work is that, so far as anyone who has been asked to critique it can detect, what looks to be an interesting study has been done using appropriate techniques, and the data looks to be both valid and to support the conclusions that are reached.

Beyond that, the reviewers should ensure that the methods used have been presented with sufficient clarity so that the study can be repeated, or at least understood, by others.

First step

Peer-reviewed publication is, in fact, just the first step in the validation of any important research finding. It’s not an absolute; just, when done properly, the best evaluation that can be made without independent verification. What matters is that the findings and conclusions hold up as the field moves forward and that the work in question played a part in achieving a useful synthesis.

Even better, the new knowledge that appears first in a research paper is truly validated when it has the power to provide, or to facilitate, a practicable solution to a real problem.

Even so, trying to stay both topical and ahead of the game does carry some inherent risks for journal editors. Many of the fraudulent papers that are eventually outed in, particularly, the biomedical sciences appear in what are generally thought of as top journals.

The reason is obvious. Editors are prepared to chance their arm to publish novel, high-impact findings. Sometimes the review process is at fault, but more often the fraud is carefully hidden.

Even so, though bad (or dishonest) science does not usually have any real long-term negative effect within science, it can cause immense damage if wrong ideas about subjects like childhood vaccination become established in the public consciousness. That’s why it is so important that reviewers and editors do their best to see that potentially emotive, but erroneous, material does not seep through the process.

This is an edited extract from The Knowledge Wars by Peter Doherty (RRP A$29.99, eBook A$19.99), out now.