What is it with researchers and peer review? or; Why misquoting Churchill does not an argument make
I’ve been meaning for a while to write something about peer review, pre and post publication, and the attachment of the research community to traditional approaches. A news article in Nature though, in which I am quoted seems to have really struck a nerve for many people and has prompted me to actually write something. The context in which the quote is presented doesn’t really capture what I meant but I stand by the statement in isolation:
“It makes much more sense in fact to publish everything and filter after the fact” – quoted in Mandavilli (2011) “Trial by Twitter” Nature 469, 286-287
I think there are two important things to tease out here, firstly a critical analysis of the problems and merits of peer review, and secondly a close look at how it could be improved, modified, or replaced. I think these merit separate posts so I’ll start here with the problems in our traditional approach.
One thing that has really started to puzzle me is how un-scientific scientists are about the practice of science. In their own domain researchers will tear arguments to pieces, critically analyse each piece for flaws, and argue incessantly over the data, the methodology, the analysis, and the conclusions that are being put forward, usually with an open mind and a positive attitude.
But shift their attention onto the process of research and all that goes out the window. Personal anecdote, gut feelings, half-baked calculations and sweeping statements suddenly become de rigueur.
Let me pick a toy example. Whenever an article appears about peer review it seems inevitably to begin or end with someone raising Churchill; something along the lines of:
“It’s exactly like what’s said about democracy,” he adds. “The peer-review process isn’t very good — but there really isn’t anything that’s better.” ibid
Now lets examine this through the lens of scientific argument. Firstly it’s an appeal to authority, not something we’re supposed to respect in science and in any case its a kind of transplanted authority. Churchill never said anything about peer review but even if he did, why should we care? Secondly it is a misquotation. In science we expect accurate citation. If we actually look at the Churchill quote we see:
“Many forms of Government have been tried and will be tried in this world of sin and woe. No one pretends that democracy is perfect or all-wise. Indeed, it has been said that democracy is the worst form of government except all those other forms that have been tried from time to time.” – sourced from Wikiquotes, which cites: The Official Report, House of Commons (5th Series), 11 November 1947, vol. 444, cc. 206–07
The key here is “…apart from all those other[s…] tried from time to time…”. Churchill was arguing from historical evidence. The trouble is when it comes to peer review we a) have never really tried any other system so the quote really isn’t applicable (actually its worse than that, other systems have been used, mostly on a small scale, and they actually seem to work pretty well but that’s for the next post) and b) what evidence we do have shows almost universally that peer review is a waste of time and resources and that it really doesn’t achieve very much at all. It doesn’t effectively guarantee accuracy, it fails dismally at predicting importance, and its not really supporting any effective filtering. If I appeal to authority I’ll go for one with some domain credibility, lets say the Cochrane Reviews which conclude the summary of a study of peer review with “At present, little empirical evidence is available to support the use of editorial peer review as a mechanism to ensure quality of biomedical research.” Or perhaps Richard Smith, a previous editor of the British Medical Journal, who describes the quite terrifying ineffectiveness of referees in finding errors deliberately inserted into a paper. Smith’s article is a good entry into to the relevant literature as is a Research Information Network study that notably doesn’t address the issue of whether peer review of papers helps to maintain accuracy despite being broadly supportive of the use of peer review to award grants.
Now does this matter? I mean in some ways people seem to feel we’re bumbling along well enough. Why change things? Well consider the following scenario.
The UK government gives £3B to a company, no real strings attached, except the expectation of them reporting back. At the end of the year the company says “we’ve done a lot of work but we know you’re worried about us telling you more than you can cope with, and you won’t understand most of it so we’ve filtered it for you.”
A reported digs a bit into this and is interested in these filters. The interview proceeds as follows:
“So you’ll be making the whole record available as well as the stuff that you’ve said is most important presumably? I mean that’s easy to do?”
“No we’d be worried about people getting the wrong idea so we’ve kept all of that hidden from them.”
“OK, but you’ll be transparent about the filtering at least?”
“No, we’ll decide behind closed doors with three of our employers and someone to coordinate the decision. We can’t really provide any information on who is making the decisions on what has been filtered out. Our employees are worried that their colleagues might get upset about their opinions so we have to keep it secret who looked at what.”
“Aaaalright so how much does this filtering cost?”
“We’re not too sure, but we think between £180M and £270M a year.”
“And that comes out of the £3B?”
“No, we bill that separately to another government department.”
“And these filters, you are sure that they work?”
“Well we’ve done a bit of work on that, but no-one in the company is especially interested in the results.”
“But what are the results?”
“Well we can’t show any evidence that the filtering is any good for deciding what is important or whether it’s accurate, but our employees are very attached to it. I can get some of them in, they’ll tell you lots of stories about how its worked for them…”
I mean seriously? They’d be ripped to shreds in moments. What if this happened within government? The media would have a field day. What makes us as a research community any different? And how are you going to explain that difference to taxpapers? Lets look at the evidence, see where the problems are , see where the good things are, and lets start taking our responsibility to the public purse seriously. Lets abandon the gut feelings and anecdotes and actually start applying some scientific thinking to the processes we used to do and communicate science. After all if science works, then we can’t lose can we?
Now simply abandoning the current system tomorrow is untenable and impractical. And there are a range of perfectly valid concerns that can be raised about moving to different systems. These are worth looking at closely and we need to consider carefully what kinds of systems and what kinds of transition might work. But that is a job for a second post.
- The case against peer review. (slate.com)
- The Necessities of a Peer Review (brighthub.com)
- Peer review: Trial by Twitter – Apoorva Mandavilli – naturenews – nature.com (richarddawkins.net)
- Strong scientific peer review leads to better science and policy formation (scienceblog.com)
- A trip through the peer review sausage grinder (arstechnica.com)