Reforming Peer Review. What are the practical steps?

The text of this was written before I saw Richard Poynder’s recent piece on PLoS ONE and the responses to that. Nothing in those really changes the views I express here but this text is not a direct response to those pieces.

So my previous post on peer review hit a nerve. Actually all of my posts on peer review hit a nerve and create massive traffic spikes and I’m still really unsure why. The strength of feeling around peer review seems out of all proportion to both its importance and indeed the extent to which people understand how it works in practice across different disciplines. Nonetheless it is an important and serious issue and one that deserves serious consideration, both blue skies thinking and applied as it were. And it is the latter I will try to do here.

Let me start with a statement. Peer review at its core is what makes science work. There are essentially two logical philosophy approaches that can be used to explain why the laptop I’m using works, why I didn’t die as a child of infection, and how we are capable of communication across the globe. One of these is the testing of our working models of the universe against the universe itself. If your theory of engines produces an engine that doesn’t work then it is probable there is something wrong with your theory.

The second is that by exposing our models and ideas to the harshest possible criticism of our peers that we can stress them to see what holds up to the best logical analysis available. The motto of the Royal Society “Nullius in verba” is generally loosely translated as “take no-one’s word for it”. The central idea of the Invisible College, the group that became the Royal Society was that they would present their experiments and their explanations to each other, relying on the criticism of their peers to avoid the risk of fooling themselves. This combined both philosophical approaches, seeing the apparatus for yourself, testing the machinery against the world, and then testing the possible explanations for its behaviour against the evidence and theory available. The community was small but this was in a real sense post-publication peer review; testing and critique was done in the presence of the whole community.

The systems employed by a few tens of wealthy men do not scale to todays global scientific enterprise and the community has developed different systems to manage this. I won’t re-hash my objections to those systems except to note what I hope should be be three fairly uncontroversial issues. Firstly that pre-publication peer review as the only formal process of review runs a severe risk of not finding the correct diversity and expertise of reviewers to identify technical issues. The degree of that risk is more contentious but I don’t see any need to multiply recent examples that illustrate that it is real. Second, because we have no system of formalising or tracking post-publication peer review there is no means either to encourage high quality review after publication, nor to track the current status or context of published work beyond the binary possibility of retraction. Third, that peer review has a significant financial cost (again the actual level is somewhat contentious but significant seems fair) and we should address whether this money is being used as efficiently as it could be.

It is entirely possible to imagine utopian schemes in which these problems, and all the other problems I have raised are solved. I have been guilty of proposing a few myself in my time. These will generally involve taking a successful system from some other community or process and imagining that it can be dropped wholesale on the research community. These approaches don’t work and I don’t propose to explore them here in detail, except as ways to provoke and raise ideas.

Arguments and fears

The prospect of radical change to our current process of peer review provokes very strong and largely negative responses. Most of these are based on fears of what would happen if the protection that our current pre-publication peer review system offers us is ripped away. My personal view is that these protections are largely illusory but a) I could well be wrong and b) that doesn’t mean we shouldn’t treat these fears seriously. They are, after all, a barrier to effective change, and if we can neutralize the fears with evidence then we are also making a case for change, and in most cases that evidence will also offer us guidance on the best specific routes for change.

These fears broadly fall into two classes. The first is the classic information overload problem. Researchers already have too much to track and read. How can they be expected to deal with the apparent flood of additional information? One answer to this is to ask how much more information would be released. This is difficult to answer. Probably somewhere between 50 and 95% of all papers that are submitted somewhere do eventually get published [1, 2 (pdf), 3, 4] suggesting that the total volume would not increase radically. However it is certainly arguable that reducing barriers would increase this. Different barriers, such as cost could be introduced but since my position is that we need to reduce these barriers to minimise the opportunity cost inherent in not making research outputs public I wouldn’t argue for that. However we could imagine a world in which small pieces of research output get published for near zero cost but turning those pieces into an argument, something that would look a lot like the current formally published paper, would cost more either in terms of commitment or financial costs.

An alternative argument, and one I have made in the past is that our discovery tools are already broken and part of the reason for that is there is not enough of an information substrate to build better ones. This argument holds that by publishing more we can make discovery tools better and actually solve the overload problem by bringing the right information to each user as and when they need it. But while I make this argument and believe it, it is conceptually very difficult for most researchers to grasp. I hesitate to suggest that this has something to do with the best data scientists, the people who could solve this problem, eschewing science for the more interesting and financially rewarding worlds of Amazon, Google, and Facebook.

The second broad class of argument against change is that the currently validated and recognized literature will be flooded with rubbish. In particular a common, and strongly held, view is that the wider community will no longer be able to rely on the quality mark that the peer reviewed literature provides in making important health, environmental, and policy decisions. Putting aside the question of whether in fact peer review does achieve an increase in accuracy or reliability there is a serious issue here to be dealt with respect to how the ongoing results of scientific research are presented to the public.

There are real and serious risks in making public the results of research into medicine, public health, and the environment. Equally treating the wider community as idiots is also dangerous. The responsible media and other interested members of the community, who can’t be always be expected to delve into, or be equipped to critique, all of the detail of any specific claim, need some clear mark or statement of the level of confidence the research community has in a finding or claim. Regardless of what we do the irresponsible media will just make stuff up anyway so its not clear to me that there is much that can be done there but responsible reporters on science benefit from being able to reference and rely on the quality mark that peer review brings. It gives them an (at least from their perspectice) objective criterion on which to base the value of a story.

It isn’t of course just the great unwashed that appreciate a quality control process. For any researcher moving out of their central area of expertise to look at a new area there is a bewildering quantity of contradictory statements to parse. How much worse would this be without the validation of peer review? How would the researcher know who to trust?

It is my belief that the emotional response to criticism of traditional pre-publication peer review is tightly connected to this question of quality, and its relation to the mainstream media. Peer review is what makes us difference. It is why we have a special relationship with reporters, and by proxy the wider community, who can trust us because of their reliance on the rigour of our quality marks. Attacks on peer review are perceived as an attack at the centre of what makes the research community special.

The problem of course is that the trust has all but evaporated. Scandals, brought on in part by a reliance on the meaning and value of peer review, have taken away a large proportion of the credibility that was there. Nonetheless, there remains a clear need for systems that provide some measure of the reliability of scientific findings. At one level, this is simple. We just wait ten years or so to see how it pans out. However, there is a real tension between the needs of reporters to get there first and be timely and the impossibility of providing absolute certainty around research findings.

Equally applying findings in the real world will often mean moving before things are settled. Delays in applying the results of medical research can kill people just as much as rushing in ahead of the evidence can. There is always a choice to be made as to when the evidence is strong enough and the downside risks low enough for research results to be applied. These are not easy decisions and my own view is that we do the wider community and ourselves a disservice by pretending that a single binary criterion with a single, largely hidden, process is good enough to universally make that decision.

Confidence is always a moving target and will continue to be. That is the nature of science. However an effective science communication system will provide some guide to the current level of confidence in specific claims.  In the longer term there is a need to re-negotiate the understanding around confidence between the responsible media and the research community. In the shorter term we need to be clearer in communicating levels of confidence and risk, something which is in any case a broader issue for the whole community.

Charting a way forward

So in practical terms what are the routes forward? There is a rhetorical technique of persuasion that uses a three-part structure in arguing for change. Essentially this is to lay out the argument in three parts, firstly that nothing (important) will change, second that there are opportunities for improvement that we can take, and third that everything will change. This approach is supposed to appeal to three types of person, those who are worried about the risks of change, those in the middle who can see some value in change but are not excited by it, and finally those who are excited by the possibilities of radical change. However, beyond being a device this structure suits the issues here, there are significant risks in change, there are widely accepted problems with the current system, and there is the possibility for small scale structural changes to allow an evolution to a situation where radical change can occur if momentum builds behind it.

Nothing need change

At the core of concerns around changing peer review is the issue of validation. “Peer reviewed” is a strong brand that has good currency. It stands for a process that is widely respected and, at least broadly speaking, held to be understood by government and the media. In an environment where mis-reporting of medical or environmental research can easily lead to lost lives this element of validation and certification is critical. There is no need in any of the systems I will propose for this function to go away. Indeed we aim to strengthen it. Nor is there a need to abandon the situation where specific publication venues are marked as having been peer reviewed and only contain material that has been through a defined review process. They will continue to stand or fall on their quality and the value for money that they offer.

The key to managing the changes imposed on science communication by the rise of the web, while maintaining the trust and value of traditional review systems, is to strengthen and clarify the certification and validation provided by peer review and to retain a set of specific publication venues that guarantee those standards and procedures of review. These venues, speaking as they will to both domain specific and more general scientific audience, as well as to the wider community will focus on stories and ideas. They will, in fact look very like our current journals and have contents that look the same as our current papers.

These journals will have a defined and transparent review process with objective standards and reasonable timeframes. This will necessarily involve obtaining opinions from a relatively small number of people and a final decision made by a central editor who might be a practising researcher or a professional editor. In short all the value that is created by the current system, should and can be retained.

Room for improvement

If we are to strengthen the validation process of peer review we need to address a number of issues. The first of these is transparency. A core problem with peer review is that it is in many cases not clear what process was followed. How many external referees were used? Did they have substantive criticisms, and did disagreements remain? Did the editors over-rule the referees or follow their recommendation? Is this section of the journal peer reviewed at all?

Transparency is key. Along with providing confidence to readers such transparency could support quantitative quality control and would provide the data that would help us to identify where peer review succeeds and where it is failing. Data that we desperately need so we can move beyond assertions and anecdote that characterise the current debate.

A number of publishers have experimented with open peer review processes. While these remain largely experiments a number of journals, particularly those in medical fields, will publish all the revisions of a paper along with the review reports at each stage. For those who wish to know whether their concerns were covered in the peer review process this is a great help.

Transparency can also support an effective post publication review process. Post-publication review has occurred at ArXiv for many years where a pre-print will often be the subject of informal discussion and comment before it is submitted for formal review at a peer reviewed journal. However it could be argued that the lack of transparency that results from this review happening informally makes it harder to identify the quality papers in the ArXiv.

A more formal process of publication, then validation and certification has been adopted by Atmospheric Chemistry and Physics and other Copernicus publications. Here the submitted manuscript is published in ACP Discussions (after a “sanity check” review), and then subject to peer review, both traditional by selected referees and in an open forum. If the paper is accepted it is published, along with links to the original submission and commentary in the main journal. The validation provided by review is retained while providing enhanced transparency.

In addition this approach addresses the concerns of delays in publication, whether due to malicious referees or simply the mechanics of the process, and the opportunity costs for further research that they incur. By publishing first, in a clearly non-certificated form, the material is available for those who might find them of value but in a form that is clearly marked as non-validated, use at own risk. This is made clear by retaining the traditional journal, but adding to it at the front end. This kind of approach can even support the traditional system of tiered journals with the papers and reviews trickling down from the top forming a complete record of which journal rejected which papers in which form.

The objection to this style of approach is that this approach doesn’t support the validation needs of biomedical and chemical scientists to be “first to publish in peer reviewed journal”. There is a significant cultural distinction between the physical sciences that use ArXiv and the biosciences in particular best illustrated by a story that I think I first heard from Michael Nielsen.

A biologist is talking to a physicist and says, “I don’t understand how you can put your work in the ArXiv as a preprint. What if someone comes along and takes your results and then publishes them before you get your work to a peer reviewed journal?”

The physicist thinks a little about this before responding, “I don’t understand how you can not put your work in the ArXiv as a preprint. What if someone comes along and takes your result and then publishes them before you get your work to a peer reviewed journal?”

There is a cultural gulf here that can not be easily jumped. However this is happening by stealth anyway with a variety of journals that have subtle differences in the peer review process that are not always clearly and explicitly surfaced. It is interesting in this context that PLoS ONE and now its clones are rapidly moving to dominate the publishing landscape despite a storm of criticism around the (often misunderstood) peer review model. Even in the top tier it can be unclear whether particular classes of article are peer reviewed (see for example these comments [1, 2, 3] on this blog post from Neil Saunders). The two orthogonal concepts of “peer reviewed” and “formally published” appear to be drifting apart from what was an easy (if always somewhat lazy) assumption that they are equivalent. Priority will continue to be established by publication. The question of what kind of publication will “count” is likely to continue to shift but how fast and in what disciplines remains a big question.

This shift can already be seen in the application of DOIs to an increasingly diverse set of research outputs. The apparent desire to apply DOIs stems from the idea that a published object is “real” if it has a DOI. This sense of solidness seems to arise from the confidence that having a DOI makes an object citeable. The same confidence does not apparently apply to URLs or other identifiers, even when those URLs come from stable entities such as Institutional Repositories or recognised Data Services.

This largely unremarked shift may potentially lead to a situation where a significant proportion of the reference list of a peer reviewed paper may include non-peer reviewed work. Again the issue of transparency arises, how should this be marked? But equally there will be some elements that are not worthy of peer review, or perhaps only merit automated validation such as some types of dataset. Is every PDB or Genbank entry “peer reviewed”? Not in the commonly meant sense, but is it validated? Yes. Is an audit trail required? Yes.

A system of transparent publication mechanisms for the wide range of research objects we generate today, along with clear but orthogonal marking of whether and how each of those objects have been reviewed provides real opportunities to both encourage rapid publication, enable transparent and fair review, and to provide a framework for communicating effectively the level of confidence the wider community has in a particular claim.

These new publication mechanisms and the increasing diversity of published research outputs are occurring anyway. All I am really arguing for is a recognition and acceptance that this is happening at different rates and in different fields. The evidence from ArXiv, ACP, and to a lesser extent conferences and online notebooks is that the sky will not fall in as long as there is clarity as to how and whether review has been carried out. The key therefore is much more transparent systems for marking what is reviewed, and what is not, and how review has been carried out.

Radical Changes

A system that accepts that there is more than one version of a particularly communication opens the world up to radical change. Re-publication following (further) review becomes possible as do updates and much more sophisticated retractions. Papers where particular parts are questioned become possible as review becomes more flexible and disagreement, and the process of reaching agreement no longer need to be binary issues.

Reviewing different aspects of a communication leads in turn to the feasibility of publishing different parts for review at different times. Re-aggregating different sets of evidence and analysis to provide a dissenting view becomes feasible. The possibilities of publishing and validating portions of a whole story offer great opportunities for increased efficiency and for much more public engagement and information with the current version of the story. Much is made of poor media reporting of “X cures/causes cancer” style stories but how credible would these be if the communication in question was updated to make it clear that the media coverage was overblown or just plain wrong? Maybe this wouldn’t make a huge difference but at some level what more can we be asked to do?

Above all the blurring of the lines between what is published and what is just available and an increasing need to be transparent about what has been reviewed and how will create a market for these services. That market is ultimately what will help to both drive down the costs of scholarly communication and to identify where and how review actually does add value. Whole classes of publication will cease to be reviewed at all as the (lack of) value of this becomes clear. Equally high quality review can be re-focussed where it is needed, including the retrospective or even continuous review of important published material. Smaller ecosystems will naturally grow up where networks of researchers have an understanding of how much they trust each others results.

The cultural chasm between the pre-review publication culture by users of the ArXiv and the chemical and biomedical sciences will not be closed tomorrow but as the pressures of government demands for rapid exploitation and the possibilities of losing opportunities by failing to communicate rise there will be a gradual move towards more rapid publication mechanisms. In parallel as the pressures to quantitatively demonstrate efficient and effective use of government funding rise opportunities will arise for services to create low barrier publication mechanisms. If the case can be made for measurement of re-use then this pressure has the potential to lead to effective communication as well as just dumping of the research record.

Conclusion

Above all other things the major trend I see is the breakage of the direct link between publication and peer review. Formal publication in the print based world required a filtering mechanism to be financially viable. The web removes that requirement, but not the requirement of quality marking and control. The ArXiv, PLoS ONE and other experiments with simplifying peer review processes, Institutional Repositories, and other data repositories, the expanding use of DOIs, and the explosion of freely available research content and commentary on the web are all signs of a move towards lower barriers in publishing a much more diverse range of research outputs.

None of this removes the need for quality assurance. Indeed it is precisely this lowering of barriers that has brought such a strong focus on the weaknesses of our current review processes. We need to take the best of both the branding and the practice of these processes and adapt them or we will lose both the confidence of our own community and the wider public. Close examination of the strengths and weaknesses and serious evidence gathering is required to adapt and evolve the current systems for the future. Transparency, even radical transparency of review processes may well be something that is no longer a choice for us to make. But if we move in this direction now, seriously and with real intent, then we may as a research community be able to retain control.

The status quo is not an option unless we choose to abandon the web entirely as a place for research communication and leave it for the fringe elements and the loons. This to me is a deeply retrograde step. Rather, we should take our standards and our discourse, and the best quality control we can bring to bear out into the wider world. Science benefits from a diversity of views and backgrounds. That is the whole point of peer review. The members of the Invisible College knew that they might mislead themselves and took the then radical approach of seeking out dissenting and critical views. We need to acknowledge our weaknesses, celebrate our strengths and above all state clearly where we are unsure. It might be bad politics, but it’s good science.