Peer review: What is it good for?
- Image by Gideon Burton via Flickr
It hasn’t been a real good week for peer review. In the same week that the Lancet fully retract the original Wakefield MMR article (while keeping the retraction behind a login screen – way to go there on public understanding of science), the main stream media went to town on the report of 14 stem cell scientists writing an open letter making the claim that peer review in that area was being dominated by a small group of people blocking the publication of innovative work. I don’t have the information to actually comment on the substance of either issue but I do want to reflect on what this tells us about the state of peer review.
There remains much reverence of the traditional process of peer review. I may be over interpreting the tenor of Andrew Morrison’s editorial in BioEssays but it seems to me that he is saying, as many others have over the years “if we could just have the rigour of traditional peer review with the ease of publication of the web then all our problems would be solved”. Scientists worship at the altar of peer review, and I use that metaphor deliberately because it is rarely if ever questioned. Somehow the process of peer review is supposed to sprinkle some sort of magical dust over a text which makes it “scientific” or “worthy”, yet while we quibble over details of managing the process, or complain that we don’t get paid for it, rarely is the fundamental basis on which we decide whether science is formally published examined in detail.
There is a good reason for this. THE EMPEROR HAS NO CLOTHES! [sorry, had to get that off my chest]. The evidence that peer review as traditionally practiced is of any value at all is equivocal at best (Science 214, 881; 1981, J Clinical Epidemiology 50, 1189; 1998, Brain 123, 1954; 2000, Learned Publishing 22, 117; 2009). It’s not even really negative. That would at least be useful. There are a few studies that suggest peer review is somewhat better than throwing a dice and a bunch that say it is much the same. It is at its best at dealing with narrow technical questions, and at its worst at determining “importance” is perhaps the best we might say. Which for anyone who has tried to get published in a top journal or written a grant proposal ought to be deeply troubling. Professional editorial decisions may in fact be more reliable, something that Philip Campbell hints at in his response to questions about the open letter [BBC article]:
Our editors [...] have always used their own judgement in what we publish. We have not infrequently overruled two or even three sceptical referees and published a paper.
But there is perhaps an even more important procedural issue around peer review. Whatever value it might have we largely throw away. Few journals make referee’s reports available, virtually none track the changes made in response to referee’s comments enabling a reader to make their own judgement as to whether a paper was improved or made worse. Referees get no public credit for good work, and no public opprobrium for poor or even malicious work. And in most cases a paper rejected from one journal starts completely afresh when submitted to a new journal, the work of the previous referees simply thrown out of the window.
Much of the commentary around the open letter has suggested that the peer review process should be made public. But only for published papers. This goes nowhere near far enough. One of the key points where we lose value is in the transfer from one journal to another. The authors lose out because they’ve lost their priority date (in the worse case giving the malicious referees the chance to get their paper in first). The referees miss out because their work is rendered worthless. Even the journals are losing an opportunity to demonstrate the high standards they apply in terms of quality and rigor – and indeed the high expectations they have of their referees.
We never ask what the cost of not publishing a paper is or what the cost of delaying publication could be. Eric Weinstein provides the most sophisticated view of this that I have come across and I recommend watching his talk at Science in the 21st Century from a few years back. There is a direct cost to rejecting papers, both in the time of referees and the time of editors, as well as the time required for authors to reformat and resubmit. But the bigger problem is the opportunity cost – how much that might have been useful, or even important, is never published? And how much is research held back by delays in publication? How many follow up studies not done, how many leads not followed up, and perhaps most importantly how many projects not refunded, or only funded once the carefully built up expertise in the form of research workers is lost?
Rejecting a paper is like gambling in a game where you can only win. There are no real downside risks for either editors or referees in rejecting papers. There are downsides, as described above, and those carry real costs, but those are never borne by the people who make or contribute to the decision. Its as though it were a futures market where you can only lose if you go long, never if you go short on a stock. In Eric’s terminology those costs need to be carried, we need to require that referees and editors who “go short” on a paper or grant are required to unwind their position if they get it wrong. This is the only way we can price in the downside risks into the process. If we want open peer review, indeed if we want peer review in its traditional form, along with the caveats, costs and problems, then the most important advance would be to have it for unpublished papers.
Journals need to acknowledge the papers they’ve rejected, along with dates of submission. Ideally all referees reports should be made public, or at least re-usable by the authors. If full publication, of either the submitted form of the paper or the referees report is not acceptable then journals could publish a hash of the submitted document and reports against a local key enabling the authors to demonstrate submission date and the provenance of referees reports as they take them to another journal.
In my view referees need to be held accountable for the quality of their work. If we value this work we should also value and publicly laud good examples. And conversely poor work should be criticised. Any scientist has received reviews that are, if not malicious, then incompetent. And even if we struggle to admit it to others we can usually tell the difference between critical, but constructive (if sometimes brutal), and nonsense. Most of us would even admit that we don’t always do as good a job as we would like. After all, why should we work hard at it? No credit, no consequences, why would you bother? It might be argued that if you put poor work in you can’t expect good work back out when your own papers and grants get refereed. This again may be true, but only in the long run, and only if there are active and public pressures to raise quality. None of which I have seen.
Traditional peer review is hideously expensive. And currently there is little or no pressure on its contributors or managers to provide good value for money. It is also unsustainable at its current level. My solution to this is to radically cut the number of peer reviewed papers probably by 90-95% leaving the rest to be published as either pure data or pre-prints. But the whole industry is addicted to traditional peer reviewed publications, from the funders who can’t quite figure out how else to measure research outputs, to the researchers and their institutions who need them for promotion, to the publishers (both OA and toll access) and metrics providers who both feed the addiction and feed off it.
So that leaves those who hold the purse strings, the funders, with a responsibility to pursue a value for money agenda. A good place to start would be a serious critical analysis of the costs and benefits of peer review.
Addition after the fact: Pointed out in the comments that there are other posts/papers I should have referred to where people have raised similar ideas and issues. In particular Martin Fenner’s post at Nature Network. The comments are particularly good as an expert analysis of the usefulness of the kind of “value for money” critique I have made. Also a paper in the Arxiv from Stefano Allesina. Feel free to mention others and I will add them here.
![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_e.png?x-id=b5743b7d-0c7c-4113-a3a2-10a6691b5e3f)
(4 votes, average: 5.00 out of 5)
Loading ...
Yes, but one bright note: the Lancet retraction made news in at least two outlets: I heard on NPR report on it and also a general radio news report that mentioned it.
This comment was originally posted on FriendFeed
It was actually quite thoroughly covered in the UK media, so in that sense good. There is a question given the issues now raised as to why it ever got through peer review in the first place though.
This comment was originally posted on FriendFeed
Liked “Peer review: What is it good for?” http://ff.im/-frRWJ
This comment was originally posted on Twitter
Science in the Open: Peer review: What is it good for? http://bit.ly/aydsKR
This comment was originally posted on Twitter
#psp10 piece by C. Neylon http://cameronneylon.net/blog/peer-review-what-is-it-good-for/ is interesting to compare with Harley’s talk
This comment was originally posted on Twitter
[...] of researchers with the peer review process is still making waves this week. Cameron Neylon gives his own take on the matter at his blog. I’m not at all sure that I agree with his analysis, having had my own [...]
Science in the Open: Peer review: What is it good for? http://bit.ly/aydsKR /via @BoraZ
This comment was originally posted on Twitter
Science in the Open: Peer review: What is it good for? http://bit.ly/aydsKR (via @BoraZ) good questioning!
This comment was originally posted on Twitter
Reflecting on 33+ yrs of publishing: yes interesting RT @phylogenomics: Liked “Peer review: What is it good for?” http://ff.im/-frRWJ
This comment was originally posted on Twitter
If it doesn’t work in the journal format, it works even less in conference format (primary publication venue for computer science). Example: from submission to notification is 6 weeks, in which time the reviewers have to process over 20 papers each. I can’t see how this allows for any meaningful comments. See http://cacm.acm.org/magazines/2009/8/34492-viewpoint-time-for-computer-science-to-grow-up/fulltext
This comment was originally posted on FriendFeed
And yes, I am bitter :)
This comment was originally posted on FriendFeed
Peer review is a complex issue and this is the best "in a nutshell" overview I’ve seen to date. Nice one.
This comment was originally posted on FriendFeed
Love this part, Cameron: "But there is perhaps an even more important procedural issue around peer review. Whatever value it might have we largely throw away. Few journals make referee’s reports available, virtually none track the changes made in response to referee’s comments enabling a reader to make their own judgement as to whether a paper was improved or made worse. Referees get no public credit for good work, and no public opprobrium for poor or even malicious work. And in most cases a paper rejected from one journal starts completely afresh when submitted to a new journal, the work of the previous referees simply thrown out of the window."
This comment was originally posted on FriendFeed
Liked “#psp10 piece by C. Neylon http://cameronneylon.net/blog/peer-review-what-is-it-good-for/ is interesting to…” http://ff.im/fs3P3
This comment was originally posted on Twitter
Peer review: What is it good for? @CameronNeylon calls for a critical analysis of peer review costs/benefits http://bit.ly/dh1KrH
This comment was originally posted on Twitter
Two points: 1) Why doesn’t computer science use its distance from the journal system to leapfrog further away from it, e.g. to wiki-based publication models? http://cacm.acm.org/magazines/2009/8/34492-viewpoint-time-for-computer-science-to-grow-up/fulltext
This comment was originally posted on FriendFeed
2) There are some journals which do the whole review process in public, notably those published by the European Geoscience Union. http://bit.ly/axjZeo
This comment was originally posted on FriendFeed
Peer review: What is it good for? http://cameronneylon.net/blog/peer-review-what-is-it-good-for/ Really worth some thought!
This comment was originally posted on Twitter
Liked “Peer review: What is it good for?” http://ff.im/-frRWJ (via @phylogenomics)
This comment was originally posted on Twitter
Liked “Peer review: What is it good for?” http://ff.im/-frRWJ
This comment was originally posted on Twitter
Peer review: What is it good for? http://ow.ly/14rEJ Conclusion: Community distorts reality. Reason to fear = community constructing reality
This comment was originally posted on Twitter
You make many good points, but the solution isn't to throw out the baby with the bath water. There are many reasons for rejecting a paper, and peer review doesn't distinguish, but if a result is not valid, it shouldn't appear anywhere (as in the Wakefield MMR paper). However, many (most?) rejections, especially from “top” journals, are because the paper isn't “good enough” by some subjective standard. The authors then have to waste time, as you say, shopping the paper around. This could be remedied by a system where the paper was published in a lesser journal without delay. The PLoS people have a partial solution to this – papers rejected by PLoS Biology are passed down – if the authors agree – to the next tier of journals, and if those journals don't want them, to PLoS ONE, which is the bottom tier.
Biology Direct is trying another model, the open peer review you propose, where all the reviews appear along with the paper. But many authors and reviewers don't want to do this.
With open access, the cream will rise to the top in many cases, so the journal itself matters less – many papers in PLoS ONE are getting loads of attention because they're good papers. But we still need a reviewer system to eliminate bogus results, and to provide feedback on how to fix not-quite-ready results.
There are good arguments here and certainly more coherent proposals than produced by 'disgruntled of Oxbridge'. I don't completely agree but I follow the logic. I did want to say that it isn't true that there is now downside to editors in rejecting papers. It isn't very immediate but I for one have always had my editorial work judged both on the papers I accepted and those that I rejected. When papers that I have rejected have appeared in other prominent journals and/or become influential I have had some explaining to do to my bosses. Editors live in as much fear of missing something good as publishing stinkers.
#OA Science in the Open » Blog Archive » Peer review: What is it good for?: http://url4.eu/1KFK3
This comment was originally posted on Twitter
The BioMedCentral Journals publish the pre-publication history of a paper, including the reviewers names: http://www.biomedcentral.com/info/about/peerreview/
This comment was originally posted on FriendFeed
I wrote about the value of peer review a few months back: http://network.nature.com/people/mfenner/blog/2009/07/13/the-value-of-peer-review
This comment was originally posted on FriendFeed
The best quote: "Whatever value [peer-review] might have we largely throw away." This is probably also why previous studies have had such a difficulty finding a measurable benefit in peer-review.Many of your suggestions have been implemented by one journal or another, but nowhere with the goal in mind to make the best out of the process for science. IMHO this is the most innovative suggestion in your post.
This comment was originally posted on FriendFeed
. @boraz “Peer review- what is it good for?” no better than rolling dice to select quality papers???http://bit.ly/aydsKR
This comment was originally posted on Twitter
Peer review: What is it good for? http://ff.im/-fuCnV
This comment was originally posted on Twitter
Peer review: What is it good for? http://tr.im/N40c
This comment was originally posted on Twitter
Martin, yes sorry I missed your post somehow originally, possibly in a post-Scifoo haze. The comments there are really thoughtful and apposite to my post as well. Particularly about potential unintended side effects and multiple rounds of peer review.
This comment was originally posted on FriendFeed
Daniel, re: computer science its an interesting question, and a comment on the post raises the issue that the problems we’re raising are much worse in a conference paper system due to time constraints. But everything I’ve seen around CS has been suggestions to move more towards a journal system as this is typical of "mature" subjects. Again its about wanting the cache that the magic dust of peer review seems to sprinkle on its recipients. It provides all this (IMO unjustified) credibility.
This comment was originally posted on FriendFeed
Daniel, re: computer science its an interesting question, and a comment on the post raises the issue that the problems we’re raising are much worse in a conference paper system due to time constraints. But everything I’ve seen around CS has been suggestions to move more towards a journal system as this is typical of "mature" subjects. Again its about wanting the cache that the magic dust of peer review seems to sprinkle on its recipients. It provides all this (IMO unjustified) credibility. The EGU system is interesting, I hadn’t come across that before. Do people seem to cope with it?
This comment was originally posted on FriendFeed
@objetpetitm – RT @SpGNo: Peer review: What is it good for? http://tr.im/N40c
This comment was originally posted on Twitter
liked ‘Peer review: What is it good for?’ @CameronNeylon calls for a critical analysis of the benefits of peer review http://bit.ly/aydsKR
This comment was originally posted on Twitter
Q: ‘Peer review: What is it good for?’ A: Narrow technical questions, not determining importance. http://bit.ly/aydsKR (via @BioMedCentral)
This comment was originally posted on Twitter
Very good look at the flaws in peer review: http://cameronneylon.net/blog/peer-review-what-is-it-good-for/
This comment was originally posted on Twitter
Cameron, what I took away from Lance Fortnow’s CACM piece was that the current state of affairs in CS does not provide reasonable publication outlets for a large body of solid science that isn’t graded as "the best" or for longer, in-depth papers. Open access is one answer to providing discoverable publication. But there’s a conflict between the growth in scientific research output and the perceived need to be published in a small number of high-profile journals. As an editor for a niche journal I’m concerned that peer review just won’t scale.
This comment was originally posted on FriendFeed
Chris, fair point, and I will accept that I overstated the case somewhat in that respect, there are certainly some consequences for editors. But to stretch the financial analogy they are not fully unwound. There are internal consequences, and no doubt these lead to some sort of external consequences in the long term, but they are hidden. In a functioning market all players need reasonably good information, including e.g. Journal X has a history of rejecting papers similar to mine which then go on to have a big impact in Journal Y – to cut my costs I should submit to Journal Y.
Steven, I agree that peer review is all we have, and in the long term it seems to work, or at least science works. But I still feel we need to ask the fundamental question. You say “we still need a reviewer”. I say, “show me the evidence that this provides any useful information at all”. More precisely lets look at the situations where we can show peer review does work and try to make them more efficient. Then toss out the rest and try to find better solutions. I like the PLoS ONE approach (and I am an academic editor) because it narrows the criteria but even here it is tough.
In principle I do like the Biology Direct approach, but in practice, being in the middle of trying to get it to work for me it is confusing and I feel even more inefficient from my perspective. And as I understand it still doesn't own up to rejecting papers or the reasons for that decision.
I think there is a more effective alternative to the trickle down approach you describe. Publishing everything as pre-prints and then select, or charge a stinging fee, for peer review. People will only put forward what they see as the best, reviewers will have more time and prestige for the work they do, and the whole process can be feed into a commentary system for a paper that already exists, solving both the priority problem and the retaining of value in the comments.
@Bill I don’t think peer review can scale and it didn’t scale well. In my opinion a good consequence of fully open and attributed peer review would be a huge reduction in the number of peer-reviewed publications submitted and published. Then, on average peer review value would not be wasted like it is now as Cameron points out.
This comment was originally posted on FriendFeed
What is peer review good for? http://tinyurl.com/yf6rscg
This comment was originally posted on Twitter
Bill, yes I can understand the motivation for the need for providing more venues for publication. Conventional (or some variant on) peer might provide a partial solution, particularly for more in depth stuff, but I guess I’m with Daniel in wondering whether as a discipline CS is better placed than most to do something different. Firstly because there is already better engagement with technology and tools than in most disciplines, but also because the idea of systems optimisation is (or at least should be!) deeply embedded
This comment was originally posted on FriendFeed
High-energy physics has longed used Arxiv to deposit papers without peer review. But for reasons that somebody involved in the field can explain better (e.g. Enrico Balli), the community still wants/needs peer-reviewed journals.
This comment was originally posted on FriendFeed
Fwd: Peer review: What is it good for? – http://cameronneylon.net/blog/peer-review-what-is-it-good-for/ (via… http://ff.im/-fvdq4
This comment was originally posted on Twitter
My understanding is that this is largely an external validation issue. So people in some theoretical physics disciplines rarely bother to go through peer review because they don’t need external validation from people who don’t "get" the Arxiv. But it would be good to get the perspective of someone on the inside on that.
This comment was originally posted on FriendFeed
Peer review: What is it good for?http://cameronneylon.net/blog/peer-review-what-is-it-good-for/
This comment was originally posted on Twitter
I thought about the scaling issue as well. For example, a typical top-level conference has ~300 submissions and 40-50 reviewers filtering to 40 accepted papers. If we ‘allowed’ all the submissions, arguably there are now too many to keep up with. But if I read the proceedings now, I only find 4-5 papers of interest. So without the gatekeepers, I would probably now see 20-30 papers of interest (by which I might say, papers I am qualified to offer comment on). There is a case for cross-fertilization opportunities, but maybe we should think about further sub-divisions.
This comment was originally posted on FriendFeed
reviewing journal paper.it’s flawed on so many levels it’s hard to know where to begin.what’s the point anyway? http://tinyurl.com/yf6rscg
This comment was originally posted on Twitter
Peer review: What is it good for? http://cameronneylon.net/blog/peer-review-what-is-it-good-for/
This comment was originally posted on Twitter
@ Cameron, authors cope by less frequently submitting low-quality manuscripts, referees by providing careful (and often signed) reviews. To get a feeling on how it works, just take a look at any recently published discussion or final paper. Mine are at http://www.biogeosciences.net/title_and_author_search.html?x=0&y=0&author=mietchen .
This comment was originally posted on FriendFeed
After they have switched from CC-BY-NC-SA to CC-BY some years ago, I see only one problem with their current system: They do not have a journal with a scope extending significantly beyond the geosciences (fair enough, given that they are a geoscientific society), which is a pity for work that does not fit that scope, since there is no real alternative out there which does the review in public (even PLoS ONE does not generally publish the review correspondence). Rejections on scope grounds, by the way, come before the discussion stage of a paper and are thus not recorded in public – I hope that this is going to change too.
This comment was originally posted on FriendFeed
After they have switched from CC-BY-NC-SA to CC-BY some years ago, I see only one problem with their current system: They do not have a journal with a scope extending significantly beyond the geosciences (fair enough, given that they are a geoscientific society), which is a pity for work that does not fit that scope, since there is no real alternative out there (even PLoS ONE does not generally publish the review correspondence). Rejections on scope grounds, by the way, come before the discussion stage of a paper and are thus not recorded in public – I hope that this is going to change too.
This comment was originally posted on FriendFeed
Another interesting aspect of their publishing – not related to peer review – is that they use http://sref.org/ instead of http://doi.org/ .
This comment was originally posted on FriendFeed
Interesting. Ulrisch Proschl has left some more detailed descriptions about the EGU approach at the blog post as well.
This comment was originally posted on FriendFeed
Another one on the "peer review week" theme: http://dx.doi.org/10.1007/s00114-010-0652-4 .
This comment was originally posted on FriendFeed
Reading: Peer review: What is it good for?http://cameronneylon.net/blog/peer-review-what-is-it-good-for/
This comment was originally posted on Twitter
Dear Cameron and All:
following up on a suggestion of Daniel Mietchen I encountered your ongoing discussion, which I find very interesting.
I agree with many of the arguments put forward, and I would like to draw your attention to a relatively new form of scientific publishing and quality assurance that solves or reduces most of the problems you addressed: interactive open access publishing and peer review as practiced by the journal Atmospheric Chemistry and Physics (ACP, http://www.atmos-chem-phys.net) and a rapidly growing number of sister journals of the European Geosciences Union (EGU, http://www.egu.eu).
Please find attached the abstract of a recent article explaining the concept, achievements and perspectives of interactive publishing, which effectively resolves the dilemma between free speech, rapid communication and thorough quality assurance as required in the scientific discourse. For more information, please visit the web pages of ACP and EGU (all freely available through open access and creative commons licensing):
http://www.atmospheric-chemistry-and-physics.ne...
http://www.atmospheric-chemistry-and-physics.ne...
http://www.atmospheric-chemistry-and-physics.ne...
With best regards,
Uli Pöschl
Interactive Open Access Publishing and Peer Review: The Effectiveness and Perspectives of Transparency and Self-Regulation in Scientific Communication and Evaluation
Ulrich Pöschl
Max Planck Institute for Chemistry, Mainz, Germany, u.poschl@mpic.de
Manuscript version of 26 October 2009, Submitted to LIBER Quarterly
Abstract
The traditional forms of scientific publishing and peer review do not live up to the demands of efficient communication and quality assurance in today’s highly diverse and rapidly evolving world of science. They need to be advanced and complemented by interactive and transparent forms of review, publication, and discussion that are open to the scientific community and to the public.
The advantages of open access, public peer review and interactive discussion can be efficiently and flexibly combined with the strengths of traditional publishing and peer review. Since 2001 the benefits and viability of this approach are demonstrated by the highly successful interactive open access journal Atmospheric Chemistry and Physics (ACP, http://www.atmos-chem-phys.net) and a growing number of sister journals of the European Geosciences Union (EGU, http://www.egu.eu) and Copernicus Publications (http://www.copernicus.org).
These journals are practicing a two-stage process of publication and peer review combined with interactive public discussion, which effectively resolves the dilemma between rapid scientific exchange and thorough quality assurance. The same or similar concepts have also been adopted in other disciplines, including the life sciences and economics. Note, however, that alternative approaches where interactive commenting and public discussion are not fully integrated with formal peer review by designated referees tend to be less successful. So far, the interactive open access peer review of ACP is arguably the most successful alternative to the closed peer review of traditional scientific journals.
The principles, key features and results of interactive open access publishing and peer review are presented and discussed in this manuscript. The achievements and statistics of ACP and its sister journals clearly prove both the scientific benefits and the financial sustainability of open access. Future perspectives and a vision of improved communication and evaluation in the global information commons are outlined with regard to the principles of critical rationalism and open societies.
P.S.: ACP and its EGU interactive open access sister journals are currently publishing about 2000 papers with a turnover of 2 MEUR per year, which the authors or their institutions are ready to cover. Moreover, they are top ranked in the citation statistics of their field (see ISI-SCIE, SCOPUS, Google Scholar, etc.). In other words, interactive open access publishing and peer review are already well established and continue to spread throughout the geosciences and beyond (see links of preceding post).
Cameron, your proposal is very close to what the interactive open access journal Atmospheric Chemistry and Physics (ACP, http://www.atmos-chem-phys.net) and a growing number of sister journals of the European Geosciences Union (EGU, http://www.egu.eu) are practicing since 2001 with great success and at fairly large scale. The results are high and steeply increasing rates of submission and publication (currently 1000 papers per year for ACP), top quality and visibility (impact factors) at low rejection rates (only 10% as opposed to 50% in traditional journals with lower impact factors), and financial sustainability at low cost (approx. 1000 EUR per paper). I am confident that interactive open access publishing is suitable for most if not all scientific disciplines, and I can only recommend this approach to all scientific publishers.
Peer review: What is it good for? http://cameronneylon.net/blog/peer-review-what-is-it-good-for/ Think the author is overstretching a bit
This comment was originally posted on Twitter
“Peer review- what is it good for?” no better than rolling dice to select quality papers???http://bit.ly/aydsKR (via @doctorblogs @boraz)
This comment was originally posted on Twitter
Great lesson Jay. verse 6 form above says: 6 “Therefore, say to the Israelites: ‘I am the LORD, and I will bring you out from under the yoke of the Egyptians. I will free you from being slaves to them, and I will redeem you with an outstretched arm and with mighty acts of judgment. 7 I will take you as my own people, and I will be your God. Then you will know that I am the LORD your God, who brought you out from under the yoke of the Egyptians. 8 And I will bring you to the land I swore with uplifted hand to give to Abraham, to Isaac and to Jacob. I will give it to you as a possession. I am the LORD.’”
You put God’s statements that “I will bring you out… I will redeem you… I will take you as my own people, and I will be your God in bold print. May I suggest that those that emphasize the sovereignty of God do exactly the same thing. They place the emphasis on what God does and regard it as primary and regard what man does as secondary and the result of what God does. Someone has said that even Pelagians like Calvinism until someone points it out and then they back off of it. Thus they take a little of the credit and glory that belongs to God and appropriate it to themselves for making the right choice and deciding to give their lives to Jesus. Granted, they don’t see it that way, but it is logical to see it like that when they say the difference between them and a lost person is their free will choice. Calvinists do not deny man making a choice and being responsible for the choice he makes. They just believe the credit for their choice belongs to God rather than themselves.
A second point has to do with Jesus drinking the cup of wrath. NT Wright seems to be the darling of many progressives in the CofC as well as the emerging church movement. I can’t get my head wrapped around his teaching that Jesus is my substitute but NOT a penal substitute. I wish someone could explain it to me better than what I’ve heard. In your post above Jesus certainly drinks the cup of wrath that is intended for, and deserved by sinners. This seems altogether consistent with the Calvinistic doctrine of penal substitution atonement. That is, he paid the penalty for my sin and his righteousness is imputed to me through faith. Did you intend for us to understand it that way?
Peace,
Randall
This comment was originally posted on One In Jesus.info
Peer-review: What is it good for? – http://tr.im/NcKn
This comment was originally posted on Twitter
NO NO NO! to quote Paul in this instance leaves us with an empty hope.
Paul was given progressive revelation. The offering of the kingdom was still in his sight. That is why is went to the synagogue first and then to the teaching of a new message that salvation was being offered to the Gentiles. Remember what he said about another gospel? That was mixing law and grace together. The old testament is used to study from a point of where have we come from. Not to be used on where are we going. I really don’t do communion any more because of its Jewish nature you just explained.
This comment was originally posted on One In Jesus.info
“Eric Weinstein provides the most sophisticated view of this that I have come across.” Wow. Thx @CameronNeylon!
http://bit.ly/PeersReviewed
This comment was originally posted on Twitter
Considering that this whole thing is just a tradition of the elders and that Jesus and the apostles had a tendency of disregarding those, I don’t think any of this is accurate.
Fascinating article, I never knew that. Um, according to your article, “one-cuppers” would be totally un-biblical in their practice.
Anyone who is literate can clearly see that he only said “This is the New Covenant ratified by my blood over one cup” regardless however many other cups they had at the Passoever. For example, with the first cup mentioned in Luke he doesn’t ascribe this meaning to it. Only to the second. Just sayin.
“This seems altogether consistent with the Calvinistic doctrine of penal substitution atonement.” Leave it to a Cavinist to make every discussion about bashing God’s grace.
“I really don’t do communion any more because of its Jewish nature you just explained.” What does that mean ‘really don’t do communion’?
This comment was originally posted on One In Jesus.info
I’m just referring to points and badges as an incentive mechanism for directing user interaction of a site in ways which provide business value.
This comment was originally posted on Hacker News
Heavenbound,
If I insisted on Passover as a condition to salvation, that might be another gospel. Understanding the Jewish history behind communion is not remotely another gospel.
It’s unfathomable to me that you reject communion because of its Jewish roots. Faith has a Jewish nature. So does the concept of a Messiah/Christ. So does Jesus. Christianity has a Jewish nature.
(Rom 11:24) After all, if you were cut out of an olive tree that is wild by nature, and contrary to nature were grafted into a cultivated olive tree, how much more readily will these, the natural branches, be grafted into their own olive tree!
We Gentiles were grafted into a Jewish tree. We therefore have and should honor our Jewish roots.
This comment was originally posted on One In Jesus.info
Peer review: What is it good for? http://bit.ly/atBZmj | (Science in the Open)
This comment was originally posted on Twitter
Exploring the linked site a little deeper, there’s this gem on making applications more game-like: http://lostgarden.com/Mixing_Games_and_Applications.pdfFrom what the author explains in the presentation, vim violates the tried-and-true video game mechanics of starting the user with only the most basic functionality. To teach vim using the author’s method, one could start by removing all but the most basic commands. Then, these missing commands could be introduced to the user one at a time, in a controlled environment where there is a clear task (eg. jump the cursor to a particular point in the text) that can be measured as success or failure.
In the linked presentation, the author draws a comparison to the game Metroid. In the game, the player falls into a deep pit and has to find a way to climb out before being able to continue on. Failure to perform the new skill (accurately timing the character’s wall-jumping) is immediately clear because the player will fall back down into the pit. When the user finally times it correctly, he’s free from the pit and the brain rewards him with a sense of accomplishment.
Back to vim, if you wanted to create such an environment for learning a new command to move the cursor around, you wouldn’t want the user to fall back on basic navigation with h, j, k, and l. You could disable these keys temporarily, or leave them but only reward the user if they accomplish the goal using the fewest key-presses possible.
Once they "win" this "level" you’ve designed, those new navigation keys should be considered part of their arsenal of skills for solving future problems. Each skill mastery could be further rewarded by filling in parts of a cheat-sheet (like this one: http://www.viemu.com/vi-vim-cheat-sheet.gif). This can be seen in the section where the author talks about Link to the Past and the picture of the player’s item inventory. The vim player’s goal could be to "unlock" and master these keyboard skills and ultimately fill in the complete chart.
This comment was originally posted on Hacker News
I don’t know if I’d call it a game, but there is vimtutor, which is a document that tells you how to edit it:http://www2.geog.ucl.ac.uk/~mdisney/teaching/unix/vimtutor
This comment was originally posted on Hacker News
Having read the comments I found it interesting that one of the features is to show you which of the features of Office other people actually use. I’ve had the same experience with blog posts or hacker news submissions about Vim or Unix tools in general. Simply being informed of their existence is a real benefit, otherwise they’d just sit there in my machine unused. (The hefty guidebook Vi IMproved by Steve Oualline was good too)
This comment was originally posted on Hacker News
The Ailing Peer Review. http://bit.ly/a9kHXf
This comment was originally posted on Twitter
“Peer Review: What Is It Good For?” http://bit.ly/bNL5hs [Science in the Open]
This comment was originally posted on Twitter
reading “Peer review: What is it good for?” by @cameronneylon – http://bit.ly/cQDCOV (via http://bit.ly/bSCxr1 via @josswinn)
This comment was originally posted on Twitter
Thanks @cameronneylon for http://bit.ly/cQDCOV (and @andypowe11 for recommending it). Feel I can take rejection better now!
This comment was originally posted on Twitter
Peer review: What is it good for? http://icio.us/hbentq
This comment was originally posted on Twitter
Cameron: You know my take on the issue, but let me briefly summarize it. The problem isn't peer review. The problem is a) that the reviewers have little incentives to do a good job and/or b) are not aware what is required of them for their review to be beneficial for progress in science.
Concretely I mean that reviewers have little time, have in the vast majority very insecure jobs and future options, so they'll fight for their own opinions in any possible way, even if they know it's unscientific. They know that, unfortunately, their colleagues' appreciation as well as their funding depends on how many people work on their own field (where there's flies, there must be shit). They rarely misunderstand the necessity of taking risk. But maybe worst of all is that the time it takes to offer thoughtful comments and constructive criticism is, the way it looks now, completely wasted. We simply have no culture in which criticism is sufficiently appreciated.
These are social problems, caused by insufficient education and external pressures (time, financial, peer pressure). The problem with peer review are symptoms, not the disease.
http://tinyurl.com/yewsg4a
Science in the Open » Blog Archive » Peer review: What is it good for?
This comment was originally posted on Twitter
“There are few studies that suggest peer review is better than throwing dice and a bunch that say it is much the same.” http://bit.ly/9DBJWx
This comment was originally posted on Twitter
Peer review: What is it good for? http://bit.ly/cBiZ0Y
This comment was originally posted on Twitter
Peer review: What is it good for? http://bit.ly/cBiZ0Y /via @razZ0r
This comment was originally posted on Twitter
Heavenbound,
You stated, “I really don’t do communion any more because of its Jewish nature you just explained.”
Is that the reason you give Jesus in the assembly when you decide not to take communion? Do you express that thought to Jesus when communion is served? Are you telling me that . . . had you been in the upper room when Jesus was serving the communion meal . . . that you would have said, “No thanks.”
I don’t understand your reasoning. Anyway, didn’t Jesus say, “Do this in memory of me.” Isn’t the bread and the wine something to embrace? Something To cherish? I would think that someone who was heavenbound would cherish this gift Jesus gave us.
Some other questions for you:
My god is the god of Abraham, Isaac, and Jacob. Is not my god your god? You wouldn’t reject my god would you . . . seeing that he was their god first?
Wasn’t the “prophet like me” prophesied by the prophet speaking in Deuteronomy 18 . . . Moses the lawgiver? You wouldn’t reject The Prophet Greater Than Moses would you . . . seeing that the very god at work in Moses brought to completion that which Moses prophesied?
Didn’t Paul tell us to study the scriptures . . . the Jewish scriptures? You wouldn’t reject the Psalms would you?
This comment was originally posted on One In Jesus.info
Bishop Jay,
Back to the lesson on the 5 cups. I really found myself nourished by this information, as I do in all of the summaries you are giving us on the faith lessons by Ray Vander Laan.
After reading the lesson above, it seems to me that the bread that represented his body and the wine that represented his blood were incorporated into a passover meal that was already in practice. So instead of these two things taking the place ot the passover meal, or being instituted as something separate, they were made a part of the passover meal. The passover meal was “upgraded” or reshaped into the communion meal Christ initiated. Am I reading correctly?
We see this practice in other aspects of what Jesus gave us. Christian baptism is an example that comes to mind. Baptism wasn’t new in Acts 2. There was already a baptism for forgivenss of sins. Jesus took the baptism that John the Baptizer was administering to the people and “upgraded” it. Not only was there foregiveness of sins as John taught . . . but also the gift of the Spirit was promised. A pretty good upgrade!
So God’s practice of taking existing expressions of faith . . . and incorporating them into new expressions of faith . . . is well established. Did I get this right?
Is it a stretch to conclude that it was Jesus’ intent to incorporate the symbolism from the original passover meal into the new meal that Christians would partake? This seems reasonable because, as the lesson illustrates, it would help us “understand our own Christianity in much greater depth.” and help us “understand the Lord’s Supper in greater depth.” By dropping the original components of the meal, we have lost the symbolism that Jesus intended to hand to us. Did I read this correctly? Is that what Ray Vander Laan is thinking? Is that where you are leading? I like it.
This comment was originally posted on One In Jesus.info
Good article! See also our recent First Monday paper and the following month's suggested solution to achieve what Andrew Morrison suggests:
http://firstmonday.org/htbin/cgiwrap/bin/ojs/in...
http://www.uic.edu/htbin/cgiwrap/bin/ojs/index….
It is about time this discussion was engaged.
Brian Whitworth
http://bit.ly/peerReviewSux It does. I’m feeling the pain as I get DL + AAAI assignments, a journal review, SUM starts up, and ESWC rages on
This comment was originally posted on Twitter
Bee, I agree with everything you say here (as you know) and I think we're coming from the same perspective. How do we align incentives so that important things are done? The market is clearly broken but how do we fix it? But I still think there is a more fundamental problem.
As far as I am aware this is no evidence that traditional peer review (prior to publication/decision, with a limited number of people) would be effective even if people had all the incentives in the world to do it properly. It doesn't matter how much the incentives are fixed if the wheels are still square.
[...] 9, 2010 · Leave a Comment Cameron Neylon has written a post on the problems of peer review. From the post: Whatever value it might have we [...]
[...] Neylon’s recent post about peer review is pretty damning. It’s interesting to compare his description of peer review’s faults [...]
Prodam byt 4+1 v Pacove ( okres Pelhrimov ) – Pacov 1300000,- http://realcr.cz/detail/prodam-byt-41-v-pacove-okres-pelhrimov-pacov/id/74683
This comment was originally posted on Twitter
Lots of discussion around Peer Review lately | Science in the Open » Peer review: What is it good for? http://bit.ly/d165va
This comment was originally posted on Twitter
Interesting to think about this. The two obvious factors are first the “open” bit. One of the serious problems with traditional peer review is that it is necessarily limited by the small number of people involved – this is probably what maximises the random effects. The flip side, making it open, obviously has both the advantage and disadvantage of allowing anyone who is interested to comment. Central challenge is setting the procedural barriers to comment at a level which maximises important signal to noise.
From a naive external perspective to the other advantage code has is that you can make “objective” assessments and test them against reality through explicit tests. Does this modification pass the unit tests? Is there evidence that speaks to whether this is real? Or just the imagination of us non-computational people. Comparison of code peer review in closed vs open systems?
This comment was originally posted on The Third Bit
Will O’Reilly start releasing the reviewer notes for its Conference proposals (rejected and accepted)?
This comment was originally posted on O’Reilly Radar Insight, analysis, and research about emergin…
I’d love to see all journals adopt the Nature policy of listing the contribution of each author at the end of the paper. I’ve published papers where I know that some of the co-authors haven’t even read the paper, let alone contributed text or figures to it. Ok, so they contributed a bit of data or something but I think that this should spelled out so those who really contribute can take the credit for it.
[This is the example from the first Nature paper from 1999 that used this protocol: "R.R. conceived the experiment, and together with A.H. and L.L. carried it out; C.D.B. designed and carried out the data analysis; R.R. and C.D.B. co-wrote the paper."]
This comment was originally posted on SomeBeans
With all the recent scandal, the question needs to be asked. Is the peer review process actually helping? http://goo.gl/eiEZ
This comment was originally posted on Twitter
@andyrussell – it certainly is nice to know, it’s one of the things you tend to pick up at conferences but should really be recorded in the journal.
This comment was originally posted on SomeBeans
I like the way publishing organisations also make you responsible for layout, and make you use their own templates, often with tortuous styles for drawings, chemical reaction schemes etc. I sometimes wonder if they employ any sub-editors any more. That’s probably why their profit margins are so huge. We’re all so obsessed with high-impact publication, which means choice of particular journals, that they can dictate all this. Maybe we should have the courage to publish differently.
This comment was originally posted on SomeBeans
Very funny video, very interesting post for a non-scientist who’d always vaguely wondered what a "paper" was.
Surprising that there’s so much unpaid work.
This comment was originally posted on SomeBeans
@SimonHiggins – I think the astrophysics and high energy physics communities have made some progress on doing their own thing at http://arxiv.org/ I think they have an advantage with being a relatively small community focused around a smaller number of facilities which makes reputation easy to develop personally.
@Nora_lumiere – glad you enjoyed it! I had an inkling that for a lot of people the idea of an academic paper is pretty alien, which was why I wrote the post.
This comment was originally posted on SomeBeans
I think the peer review must not be used for the selection of papers to be published, but used to improve them, enrich them with further points of view, further contrast and verification.
Furthermore, I agree that the burden of peer reviewing can be reduced by 90-95% as you claim. I have a mechanish in IEEE Intelligent Systems, Nov/Dc 2007, Vol. 22 num. 6 http://www.computer.org/portal/web/csdl/doi/10….. I named it “citation auctions” and the good thing is that peer review is applied for improving the papers prior their submission to publication, but not as the selective method for thos 90% of papers that cannot get the threshold and are automatically rejected. The remaining 10% still are reviewed to verify other editorial requirements.
@gdelfino http://is.gd/87kff
This comment was originally posted on Twitter
Brian, interesting papers though I've only had a chance to skim them so far. Have you looked at the Frontiers series of Journals or the EGU journals mentioned below in the comments? How do they map on to your thinking?
I think I've seen some similar ideas mooted but not as an explicit auction. More a pay and return scheme, you can only be peer reviewed once you've put in a certain number of peer reviews. I think I first saw something like that suggested by Jeremiah Faith – but I'd have to dig deep to find the reference now. Is there a version of your paper full text online somewhere? Would be interested to read more.
a valuable contribution from @cameronneylon to the peer review debate: http://tinyurl.com/yewsg4a
This comment was originally posted on Twitter
[...] review and have offered a developmental model of scholarship. Which means I’m predisposed to finding articles like this very satisfying: Scientists worship at the altar of peer review, and I use that metaphor [...]
I agree with Steven; while it is easy to identify flaws in the peer review process, and exciting and spectacular examples when it fails, I believe it provides a critical role in the larger process of scientific discovery. It may be useful to distinguish between “peer” and “review” — whether one gives up on the “review” process altogether, because it has a random component and may not be reliably reproducible, or whether one focuses on how to make it better.
It is not surprising that there is a random component to success in peer review, there is a random component in every review process (the New Yorker recently had a compelling story on how professional sports teams spend zillions of dollars deciding who to recruit, with tons of statistics and performance information, and yet still do a terrible job of predicting success). Similar problems are encountered in college admissions. Reviewing with perfect consistency and accuracy is impossible, even in fields that do not try to create new knowledge.
But I would argue that giving up on review is even worse. In some fields I am familiar with, there are many papers published that I think are misleading in an important way. Yes, the experiments were done as described, and the results collected properly, but the explanation for the results was not unique, and the conclusions drawn were overstated. It is hard for me to see the benefit of more misleading papers; I think reducing review barriers will reduce the signal to noise ratio dramatically, because it is much easier to produce a “novel” mistaken result than a correct one.
Bill, thanks for the comments. I don't think anyone is giving up on review altogether. Peer review in its general form is what makes science work in the long term. But this kind of review appears to work over a longer time and require more diverse input than the traditional pre-publication peer review process. My opinion/feeling is that if we could harness this kind of post-publication peer review effectively and make it more efficient then we could do much better than we ever can trying to improve the traditional pre-publication approach.
There are risks here – potentially uncertified research being picked up by a credulous media outlet and amplified or taken out of context but there are risks and damages and expenses with the current system. What I'm really arguing for is a serious cost-benefit analysis of the different approaches we could take. I still feel we need much less journal-formal-publication which I think lines up with what you are saying. Another problem here is that the published article becomes the finished article – rather than constantly evolving, or being rejected.
All these papers you mention that are misleading – why can't you mark them up with your comments? Would that not be more useful peer review? Might not another reviewer raise a point that you hadn't thought of? We used to do this before publication because the act of changing the printed copy was way too expensive to even consider. Now it costs almost nothing (to change that is – publishing still costs money obviously)
I guess at core I'm asking for evidence and clarification to support your first statement. Does traditional pre-publication peer review provide a critical role in the process. Or does review in general? The second I will agree with whole heartedly. The first I am much more sceptical.
For me, the evidence that peer review serves a central role in publication is two-fold. First, several of my best papers improved dramatically in the review process, either because I was forced to provide more data (and often do more analyses) or to present it in a more clear or convincing way. Second, papers I review often have serious mis-statements that need correction or reflect misunderstandings of the literature or the resources upon which the studies were based. Investigators, particularly in a young and rapidly changing field, often make mistakes, and the literature is more useful when there are fewer mistakes.
Indeed, I would argue that the papers demonstrating the “random” nature of peer review have some issues of their own, as those papers freely admit. We cannot know the “true” distribution of “good” and “bad” papers, and it is not clear to me that consistency is the appropriate surrogate. And then, of course, there is the irony that these papers questioning the value of peer-review were in fact peer-reviewed.
As has been pointed out elsewhere, selection of papers for publication balances competing priorities: validity, significance, “sexiness”, politics, controversy. Scientists have been complaining about literature overload for decades, which suggests to me that we need more review, not less. And it's hard for me to imagine an alternative to peer review, despite its shortcomings.
Well said, as are the comments! My take; we've depended to much in the past on a lone scientist model that ignores the social nature to knowledge. Your ideas and many comments reference the social nature of knowledge production in the process, but peer review operates without acknowledgement of this social nature. There would be benefits to reducing the cascading wall of publications, but at the same time increasing the ability of more people to contribute and to understand the process. The web has created a new publication world, but the old infrastructure is insufficient for taking advantage of the potential that exists.
Peer review is broken, http://cameronneylon.net/blog/peer-review-what-is-it-good-for/
This comment was originally posted on Twitter
Peer review: What is it good for? http://bit.ly/a9kHXf /via @glynmoody #ency10
This comment was originally posted on Twitter
Science in the Open » Peer review: What is it good for? http://bit.ly/bFD63C #science
This comment was originally posted on Twitter
Bookmarked: Peer review: What is it good for? Science in the Open http://bit.ly/bV3IOB
This comment was originally posted on Twitter
[...] blogging regarding university tenure processes and journal peer review processes are a reminder of how contestable knowledge production can be; especially if knowledge is not used [...]
[...] Siemens sent me a link to a post by Cameron Neylon that attempts to pound yet another nail in the coffin of peer review. As an editor of a peer [...]
At Neil, above: Couldn’t the gatekeeping function (somewhat necessary if space is limited at the conference) be fulfilled equally well by doing the review in public, in combination with a karma system?
This comment was originally posted on FriendFeed
Science in the Open » Blog Archive » Peer review: What is it good for? http://bit.ly/cxzR6o
This comment was originally posted on Twitter
2 thoughtful & thought-provoking pieces about peer review & its vagaries http://bit.ly/cxzR6o & http://bit.ly/1PIqDK (thnx @michael_rowe)
This comment was originally posted on Twitter
[...] 5, 2010 (my 41st birthday!): Cameron Neylon posts “Peer Review: What is it good for?” The basic premise of the article is that the process needs to be more open. Journals [...]
[...] present data seemingly supported by others' findings. That may not always be wise of me, given the faults of peer review and other supposedly rational and effective academic processes. Regardless, many tenets of science [...]
[...] First, it is important to remind everyone that peer-review is a very new thing. Only one minor paper by Einstein went through peer-review. Nature only started experimenting with it in the late 1960s. Yet lots and lots of great science was published before this was instituted. There is no data supporting the view that peer-review actually does much good. [...]
Leave your response!
Additional comments powered by BackType
License
Recent Friendfeed activity
Going into hibernation for the weekend. This week has been insane. Next week looks worse and has a long haul flight at the end of it.
Friday 18:59
I _think_ our glutamate responsive ion channel actually responds to glutamate...now for bed!
Friday 0:47
Something evil happens when people who think in Fortran write python....
Wednesday 14:40
Shared: My qualitative study of science blogging
Wednesday 10:12
Shared: Truly Open Data
Tuesday 14:39
My FriendFeed profile …
Recent posts on Twitter
Calendar
I am also found at...
Tags
Recent Posts
Most Commented
Most Viewed