Thought-provoking post, as always. There is so much I want to say/ask, but not sure where to start. Oh, and I may come off as overly negative, but it done out of a sincere interest in understanding how Plos ONE publishing model will overcome their issues and succeed. I personally have a great interest in new peer review/ scholarly communication models, so that may be why I am asking these questions.

‘we have in principle the ability to do peer review differently, more effectively, and more efficiently’ is a bold statement. Being able to improve peer review that distinguishably would truly be the holy grail of scholarly communication (and likely that of science as well). I am not sure we currently have any examples of such a new revolutionary system, though.

I agree that Plos ONE’s publication policy (the scientific/methodological soundness of the paper being the deciding factor for publication) is certainly a more efficient utilization of peer reviewers to get sound papers out there. However, is it also more effective quality wise? Since the peer review system itself does not provide a quality/significance filter, the question is whether their “post publication comments/ratings to determine significance/originality of a paperâ€ can at least provide that same level of effectivity in filtering quality, while maintaining that advantage in efficiency. I have some serious doubts about that. Mostly it has, again, to do with scaling. Concerning both the time and the (initial) amount of people required to (thoroughly) read the papers to be able to rate/comment on it. And the efficiency and effectivity of validating that rating/commenting process itself.

But let me start in a more logical order. First of all, it is just me, or is it not possible to search/browse Plos ONE’s publications based on their ratings/comments? If not, would that not be a good function to provide for their readers? Considering that is Plos ONE’s system to help readers find original/significant papers? That actually replaces the traditional “significance/originalityâ€ check by peer reviewers of other publishers? Otherwise, I do not really see the point in having ratings in the first place. Of course, when you do have such a search/browse filter system in place, you may also need to analyze the (potentially negative) effect that will have on having scholars reach and rate/comment on the unrated papers. But either way, you cannot expect them to just search/browse their fields and randomly read and rate/comment papers they find (whenever they arrive on result pages with papers absent of any ratings/comments), do you? Because in that case I seriously doubt it is a more efficient system. It certainly is not more effective to find papers that are perceived as more significant by peers.

So I wanted to find some papers with comments/ratings to see what they are like. Which brings me to another point: in the traditional peer review system, you have “qualified peersâ€ and the editors (also somewhat qualified as a “peerâ€ I guess) of the journals to assess the significance of the papers. I am now at the registration page of Plos ONE and I am wondering if there is anything that prevents me from signing up and voting articles up/down? Since I am definitely not qualified to assess most, if not all, of Plos ONE’s papers on quality/significance? In other words: how is Plos ONE planning to verify the quality of the quality assessments of the papers expressed through the ratings? In fact, how many ratings per publication would you find accurate or at least as accurate as the traditional means of (initially) determining its significance? Excluding biased ratings from family, colleagues, students etc.?

In the traditional sense, the editor, and thus the journal publisher, is answerable to both the soundness of the science and the significance of it. Which is there for all the readers to witness and judge for themselves. Not doing their job right means an end to their income/continuity. Since Plos ONE’s peer reviewers (and editors?) are only accountable for its scientific soundness, who will be responsible for the assessment of its significance? And what will be the consequences of that? And, to not ignore the all important scaling factor, remember that we are now talking about ALL the published papers (of Plos ONE), like they are in the traditional publishing models.

Anyway, having expressed some of my concerns concerning this model and with it in my mind, onto your data analysis.

‘a little more than 13% (380 of 2773) of all papers have ratings’

How many of them have more than 1 rating? Consider 2 peer reviewers and an editor for the traditional peer review to judge this, would you say at least 3 objective assessments by (qualified) peers are at least required to be on a somewhat even level?

‘around 23% have comments, notes, or replies to either (647 of 2773, not including any from PLoS ONE staff)’

Idem ditto for this statistic. And again, how well will this scale? When the time comes that all the papers need this metric early after their publications??

Personally, I think PLoS’ success is largely based on it being a high profile name, with no small help from funding institutions, and being Open Access. I do not think they necessarily stand out as a qualitative journal publisher per se because, as they have stated themselves: they are not explicitly looking for significant/original papers. And it is somewhat demonstrated that Open Access journals do get more cites than non OA. In that sense, I think it is not entirely fair to compare PLoS’ “successâ€ with commercial publishers. Based on OA’s citation advantage, one could state that PLoS journals’ impact ratings (i.e. a metric of its “successâ€) would have been lower (and its “successâ€ less significant) if they had gone commercial.

And that is all fine and well, but what happens if we ever reach 100% Open Access? Then PLoS’ OA advantage is gone, and then they have to do something else to stand out and maintain (or increase) their impact factor (and the reasons to keep on receiving the funding that they do, compared to the other OA publishers/ journals). And when that time comes, I think one of the key factors is how well their filter for quality/ significance of papers works in comparison to other OA journals/publishers. And to me, that is still the biggest issue with Plos ONE’s different publishing model. End rant.

PS. open peer review is not radical per se. The British Medical Journal (pretty high profile in the medical fields) has been toying with the concept of open peer review since 1999 (http://www.bmj.com/cgi/content/full/318/7175/4) and now it has an open peer review process (http://resources.bmj.com/bmj/authors). And the idea itself has been discussed a lot, and they usually run into the same type of for/against arguments. Nature’s experiment did not receive a whole lot of comments and was not seen as a success by them.