Home » Blog

Thinking about peer review of online material: The Peer Reviewed Journal of Open Science Online

21 September 2008 27 views 28 Comments

I hold no particular candle for traditional peer review. I think it is inefficient, poorly selective, self reinforcing, often poorly done, and above all, far too slow. However I also agree that it is the least worst system we have available to us.  Thus far, no other approaches have worked terribly well, at least in the communication of science research. And as the incumbent for the past fifty years or so in the post of ‘generic filter’ it is owed some respect for seniority alone.

So I am considering writing a fellowship proposal that would be based around the idea of delivering on the Open Science Agenda via three independent projects, one focussed on policy and standards development, one on delivering a shareable data analysis pipeline for small angle scattering as an exemplar of how a data analysis process can be shared, and a third project based around building the infrastructure for embedding real science projects involving chemistry and drug discovery in educational and public outreach settings. I think I can write a pretty compelling case around these three themes and I think I would be well placed to deliver on them, particularly given the collaborative support networks we are already building in these areas.

The thing is I have no conventional track record in these areas. There are a bunch of papers currently being written but none that will be out in print by the time the application is supposed to go in. My recorded contribution in this area is in blog posts, blog comments, presentations and other material, all of which are available online. But none of which are peer-reviewed in the conventional sense.

One possibility is to make a virtue of this – stating that this is a rapidly moving field – that while papers are in hand and starting to come out that the natural medium for communication with the specific community is online through blogs and other media. There is an argument that conventional peer review simply does not map on to the web of data, tools, and knowledge that is starting to come together and that measuring a contribution in this area by conventional means is simply misguided.  All of which I agree with in many ways.

I just don’t think the referees will buy it.

Which got me thinking. It’s not just me, many of the seminal works for the Open Science community are not peer reviewed papers. Bill Hooker‘s three parter [1, 2, 3] at Three Quarks Daily comes to mind, as does Jean-Claude’s presentation on Nature Precedings on Open Notebook Science, Michael Nielsen’s essay The Future of Science, and Shirley Wu’s Envisioning the scientific community as One Big Lab (along with many others). It seems to me that these ought to have the status of peer reviewed papers which raises the question. We are a community of peers, we can referee, we can adopt some sort of standard of signficance and decide to apply that selectively to specific works online. So why can’t we make them peer reviewed?

What would be required? Well a stable citation obviously, so probably a DOI and some reasonably strong archival approach, probably using WebCite.  There would need to be a clear process of peer review, which need not be anonymous, but there would have to be a clear probity trail to show that an independent editor or group of referees made a decision and that appropriate revisions had been made and accepted. The bar for acceptance would also need to be set pretty high to avoid the charge of simply rubber stamping a bunch of online material. I don’t think open peer review is a problem for this community so many of the probity questions can be handled by simply having the whole process out in the open.

One model would be for an item to be submitted by posting a link on a new page on an independent Wiki . This would then be open to peer review. Once three (five?) independent reviewers had left comments and suggestions – and a version of the document created that satisfied them posted – then the new version could be re-posted at the author’s site, in a specified format which would include the DOI and arhival links, along with a badge that would be automatically aggregated to create the index a la researchblogging.org. There would need to be a charge, either for submission or acceptance – submission would keep volume down and (hopefully) quality up.

How does this differ from setting up a journal? Well two major things – one is that the author remains the publisher so the costs of publication per se are taken out of the equation. This is important as it keeps costs down – not zero, there is still the cost of the DOI and (even if it is donated) the time of editors and referees in managing the process and giving a stamp of authority. The main cost is in maintaining some sort of central index and server pointing out at the approved items. It would also be appropriate to support WebCite if that is the backstop archive. But the big costs for journals are in providing storage that is stable in the long term and managing peer review. If the costs of storage are offloaded and  the peer review process can be self organised then the costs drop significantly.

The second major advantage is that, as a community we already do a lot of this, looking over blog posts, linking to presentations, writing commentary or promoting them on FriendFeed. The reason why ArXiv worked was that there was already a culture of preprints amongst that community. The reason why commenting, rating,  and open peer review trials have not been as successful as people had hoped is because there is no pre-existing culture of doing these things. We already have a culture of open peer review in our community. Is it worth formalising it for the really high quality material that’s already out there?

I am aware that this goes against many of the principles of open and continuous review that many of you hold dear but I think it could serve two useful purposes. First it means that members of the community, particularly younger members, can bolster their CV with peer reviewed papers. Come the revolution this won’t matter but we’re not there yet. Making these contributions tangible for people could be quite powerful. Secondly it takes the important material out of the constant stream of objects flitting past on our screens and gives them a static (I won’t say permanent) priviledged place as part of the record of this field.  Many of them perhaps already have this but I think there is a value in formalising it. Is it worth considering? This proposal is out for review.

 

1 Star2 Stars3 Stars4 Stars5 Stars (No Ratings Yet)
Loading ... Loading ...

  • Euan

    I think this is a great idea.

    “One model would be for an item to be submitted by posting a link on a new page on an independent Wiki . This would then be open to peer review. Once three (five?) independent reviewers had left comments and suggestions”

    But what would stop you telling three of your friends / students to go online and write positive reviews? I wasn’t sure how this bit worked re: the independent editors part (which I think you need, to ensure that the reviewers *are* independent) in the paragraph above…

  • Euan

    I think this is a great idea.

    “One model would be for an item to be submitted by posting a link on a new page on an independent Wiki . This would then be open to peer review. Once three (five?) independent reviewers had left comments and suggestions”

    But what would stop you telling three of your friends / students to go online and write positive reviews? I wasn’t sure how this bit worked re: the independent editors part (which I think you need, to ensure that the reviewers *are* independent) in the paragraph above…

  • http://openwetware.org/wiki/User:Cameron_Neylon Cameron Neylon

    That’s a fair concern which is difficult to answer simply . It is partly answered by a completely open approach – if the referees names are against the reviews then anyone can judge whether they are just mates giving a positive review or not.

    Overall I would say the worse case scenario is like a small society journal where the editor knows everyone and everyone knows the editor. There are plenty of cosy little fields where everyone scratches each others back.

    But if the reviewing is done openly, and bearing in mind that this particular community has an interest in making sure it _doesn’t_ look like back scratching because that will be the automatic assumption, then I think at least some of that charge can be answered. Even if there were an independent editor (say me) I’m still going to try and get the same relatively small subset of people to review. I think that actually having to get the ok from all referees may also help.

    But its an interesting question – what would actually be required to persuade e.g. PubMed that this was ‘proper’ peer review? Because that is the key issue. ISI have drawn their line in the sand before PLoS ONE so this is unlikely to make it for them – but PubMed is frankly far more important.

  • http://openwetware.org/wiki/User:Cameron_Neylon Cameron Neylon

    That’s a fair concern which is difficult to answer simply . It is partly answered by a completely open approach – if the referees names are against the reviews then anyone can judge whether they are just mates giving a positive review or not.

    Overall I would say the worse case scenario is like a small society journal where the editor knows everyone and everyone knows the editor. There are plenty of cosy little fields where everyone scratches each others back.

    But if the reviewing is done openly, and bearing in mind that this particular community has an interest in making sure it _doesn’t_ look like back scratching because that will be the automatic assumption, then I think at least some of that charge can be answered. Even if there were an independent editor (say me) I’m still going to try and get the same relatively small subset of people to review. I think that actually having to get the ok from all referees may also help.

    But its an interesting question – what would actually be required to persuade e.g. PubMed that this was ‘proper’ peer review? Because that is the key issue. ISI have drawn their line in the sand before PLoS ONE so this is unlikely to make it for them – but PubMed is frankly far more important.

  • http://www.sennoma.net/ bill

    Partial answer to Euan: positive reviews are not really the point, it’s the cricisisms that matter and they matter on the substance, regardless of who makes them. But this leaves open for gaming the “once 3-5 reviewers have commented” part. I’d be happier with an ongoing process like PLoS ONE, where an initial review (be it peer review, like P.ONE, or simply an editor’s decision) screens out the obvious whackos and then we let the community have at it. (This, imo, is the real reason why ISI baulks at PLoS ONE: they are afraid of the model, since it is likely the one which will render them finally and completely irrelevant.)

    Cam mentions my old Open Sci articles, so two maybe-relevant points about those. First, they were in a very real sense peer reviewed, though I didn’t alter the original text in response, in the comments section — I have been careful always to point to the original comments when repurposing the essays since there’s quite a bit of additional info there. Second, fwiw, those essays were cited by this peer-reviewed article: http://www.openmedicine.ca/article/view/205/104

  • http://www.sennoma.net bill

    Partial answer to Euan: positive reviews are not really the point, it’s the cricisisms that matter and they matter on the substance, regardless of who makes them. But this leaves open for gaming the “once 3-5 reviewers have commented” part. I’d be happier with an ongoing process like PLoS ONE, where an initial review (be it peer review, like P.ONE, or simply an editor’s decision) screens out the obvious whackos and then we let the community have at it. (This, imo, is the real reason why ISI baulks at PLoS ONE: they are afraid of the model, since it is likely the one which will render them finally and completely irrelevant.)

    Cam mentions my old Open Sci articles, so two maybe-relevant points about those. First, they were in a very real sense peer reviewed, though I didn’t alter the original text in response, in the comments section — I have been careful always to point to the original comments when repurposing the essays since there’s quite a bit of additional info there. Second, fwiw, those essays were cited by this peer-reviewed article: http://www.openmedicine.ca/article/view/205/104

  • http://openwetware.org/wiki/User:Cameron_Neylon Cameron Neylon

    Bill, I would argue that we already have the equivalent of PLoS ONE in the existing web. What I am proposing is an effectively an overlay journal with very high standards – so that perhaps only a few of my posts would actually go in.

    The fundamental reason for this is that if the criteria aren’t set pretty fierce then it would be difficult to argue for inclusion in PubMed which to me is critical. The reason why PubMed accept P.ONE is because of the traditional peer review stage, not the community review. I agree there is the potential for gaming the refereeing so getting that process right is very important – we certainly don’t want retractions.

    Actually in one sense what I am proposing _is_ post publication peer review anyway. The idea isn’t so much that people would write new posts for consideration but that they would take an existing post and refine and tighten it up and put it up for community consideration as to whether it is sufficiently important or not. If there isn’t reasonable acceptance that it was an important post (which can be partly measured by comments, links, page rank etc) and that it was novel at the time it was posted then it would be considered rejected.

  • http://openwetware.org/wiki/User:Cameron_Neylon Cameron Neylon

    Bill, I would argue that we already have the equivalent of PLoS ONE in the existing web. What I am proposing is an effectively an overlay journal with very high standards – so that perhaps only a few of my posts would actually go in.

    The fundamental reason for this is that if the criteria aren’t set pretty fierce then it would be difficult to argue for inclusion in PubMed which to me is critical. The reason why PubMed accept P.ONE is because of the traditional peer review stage, not the community review. I agree there is the potential for gaming the refereeing so getting that process right is very important – we certainly don’t want retractions.

    Actually in one sense what I am proposing _is_ post publication peer review anyway. The idea isn’t so much that people would write new posts for consideration but that they would take an existing post and refine and tighten it up and put it up for community consideration as to whether it is sufficiently important or not. If there isn’t reasonable acceptance that it was an important post (which can be partly measured by comments, links, page rank etc) and that it was novel at the time it was posted then it would be considered rejected.

  • http://usefulchem.blogspot.com/ Jean-Claude Bradley

    The definition of peer review is going to depend on the purpose.

    I still like the idea we discussed with Tony about having some molecule characterizations in ChemSpider peer reviewed so that they get a CAS number.

    For the purpose of being able to cite it on a CV under the peer reviewed heading that is trickier. Setting up a new journal is hard – look at the SJI controversy… If there was such a mechanism I would be happy to use it though. But if it requires me to spend significant time reformatting my blog posts then I would probably just as soon include it in a submission to an existing journal like PLoS ONE, which is already part of my ongoing research dissemination process.

  • http://usefulchem.blogspot.com Jean-Claude Bradley

    The definition of peer review is going to depend on the purpose.

    I still like the idea we discussed with Tony about having some molecule characterizations in ChemSpider peer reviewed so that they get a CAS number.

    For the purpose of being able to cite it on a CV under the peer reviewed heading that is trickier. Setting up a new journal is hard – look at the SJI controversy… If there was such a mechanism I would be happy to use it though. But if it requires me to spend significant time reformatting my blog posts then I would probably just as soon include it in a submission to an existing journal like PLoS ONE, which is already part of my ongoing research dissemination process.

  • http://openwetware.org/wiki/User:Cameron_Neylon Cameron Neylon

    Yes, we need to get back to the idea of peer reviewing those molecules!

    I agree setting up any journal is hard. That is why I am interested in thinking about the minimum required to be credible. I think the aim would be to pick out specific posts that don’t fit well onto any existing journals – and to make it worth the effort of refining and reformatting them. That’s why the quality criterion is important in my view.

  • http://openwetware.org/wiki/User:Cameron_Neylon Cameron Neylon

    Yes, we need to get back to the idea of peer reviewing those molecules!

    I agree setting up any journal is hard. That is why I am interested in thinking about the minimum required to be credible. I think the aim would be to pick out specific posts that don’t fit well onto any existing journals – and to make it worth the effort of refining and reformatting them. That’s why the quality criterion is important in my view.

  • http://www.researchremix.org/ Heather Piwowar

    Interesting idea, well described. Glad you’ve got the thinking rolling, Cameron.
    Quick thoughts:

    I agree with most of the previous comments: the implementation would have to depend on the core purpose. Can’t do everything (PubMed, ISI, high standards, open access, journal-not-conference, “real” peer review, almost-zero costs, fast turn around…) without a lot of work.

    There are existing mechanisms that meet many of the needs already (PLoS ONE, BMC Research Notes, App notes or Policy forums in existing journals, streams in conferences, others? maybe it would help if we made a list of existing peer-review forums that have published open-science stuff? ….). What are the fundamental drawbacks to these for the open science purposes? It sounds like turn-around time? Cost? Work in formalizing into a paper???. Can these be worked-around through some of the existing choices? My guess is yes…

    John Willinsky says, “only look like you are changing one thing at a time.” Since the open science ideas are considered radical in some circles, I think publishing in traditional formats is pretty important, at least initially.

    I think the risks of it being a “by advocates, reviewed by advocates, read by advocates” only journal is real. For effective dissemination it seems to me we are best off describing the ideas in existing channels, read by many.

    For it to be a journal with very high standards it requires hard reviewing and/or editing work, I think a significant step up from the type of reviewing that is done now within the open science community on the web (though I may be uninformed).

    That said, I think the physics people are an inspiring model for a future of scientific publishing, and that there is nothing sacrosanct about our current model of articles… it will and should evolve, so why not be on the cutting edge of that. It does sound like a great experiment! I’d publish there :)

  • http://www.researchremix.org Heather Piwowar

    Interesting idea, well described. Glad you’ve got the thinking rolling, Cameron.
    Quick thoughts:

    I agree with most of the previous comments: the implementation would have to depend on the core purpose. Can’t do everything (PubMed, ISI, high standards, open access, journal-not-conference, “real” peer review, almost-zero costs, fast turn around…) without a lot of work.

    There are existing mechanisms that meet many of the needs already (PLoS ONE, BMC Research Notes, App notes or Policy forums in existing journals, streams in conferences, others? maybe it would help if we made a list of existing peer-review forums that have published open-science stuff? ….). What are the fundamental drawbacks to these for the open science purposes? It sounds like turn-around time? Cost? Work in formalizing into a paper???. Can these be worked-around through some of the existing choices? My guess is yes…

    John Willinsky says, “only look like you are changing one thing at a time.” Since the open science ideas are considered radical in some circles, I think publishing in traditional formats is pretty important, at least initially.

    I think the risks of it being a “by advocates, reviewed by advocates, read by advocates” only journal is real. For effective dissemination it seems to me we are best off describing the ideas in existing channels, read by many.

    For it to be a journal with very high standards it requires hard reviewing and/or editing work, I think a significant step up from the type of reviewing that is done now within the open science community on the web (though I may be uninformed).

    That said, I think the physics people are an inspiring model for a future of scientific publishing, and that there is nothing sacrosanct about our current model of articles… it will and should evolve, so why not be on the cutting edge of that. It does sound like a great experiment! I’d publish there :)

  • http://openwetware.org/wiki/User:Cameron_Neylon Cameron Neylon

    Lots more relevant commentary at Friendfeed

  • http://openwetware.org/wiki/User:Cameron_Neylon Cameron Neylon

    Lots more relevant commentary at Friendfeed

  • http://openwetware.org/wiki/User:Cameron_Neylon Cameron Neylon

    Heather, I certainly agree that at the moment people should be aiming to publish things in the peer reviewed literature to get coverage to a wider audience (and not even necessarily only in the OA literature either).

    I think the fundamental disadvantage of existing journals for the kinds of things I am thinking about is the work involved in formalising it into a paper. As Jean-Claude says maybe this would be required anyway – so is it worth the effort? A fair question. Maybe we should try the experiment?

    The other is whether an appropriate journal exists – PLoS Biol seems to be the default high profile journals for Open Access/Open Science material – PLoS ONE probably isn’t appropriate because in most cases these posts aren’t strictly research – BMC Research Notes would be an appropriate place for some. An Open Science journal could in fact work as an overlay of all of these plus Blog Posts etc.

    The risk of it being as you say a ‘love in’ journal are very real. No worse than many existing journals but the key is all about perception. And such a venture would be starting from a difficult position in terms of perception obviously.

  • http://openwetware.org/wiki/User:Cameron_Neylon Cameron Neylon

    Heather, I certainly agree that at the moment people should be aiming to publish things in the peer reviewed literature to get coverage to a wider audience (and not even necessarily only in the OA literature either).

    I think the fundamental disadvantage of existing journals for the kinds of things I am thinking about is the work involved in formalising it into a paper. As Jean-Claude says maybe this would be required anyway – so is it worth the effort? A fair question. Maybe we should try the experiment?

    The other is whether an appropriate journal exists – PLoS Biol seems to be the default high profile journals for Open Access/Open Science material – PLoS ONE probably isn’t appropriate because in most cases these posts aren’t strictly research – BMC Research Notes would be an appropriate place for some. An Open Science journal could in fact work as an overlay of all of these plus Blog Posts etc.

    The risk of it being as you say a ‘love in’ journal are very real. No worse than many existing journals but the key is all about perception. And such a venture would be starting from a difficult position in terms of perception obviously.

  • http://www.researchremix.org/ Heather Piwowar

    Thanks for the friendfeed link, good stuff.

    Hrm, good on you, I’m still struggling with the not-only-in-the-OA-literature question, though I think I agree with you (esp if preprint can be made avail ;) )

    Agreed, love-in possibility for existing journals too. But if they aren’t trying to change the world, then it is ok if they have a limited involvement/readership. Less so with this open science stuff?

    I think the work required in making it a real journal article is needed, regardless. That work is what a peer reviewed paper represents, at least to your funders.

    I do think overlay journals have value, agreed.

    If we can figure out a way to make it work given all time constraints (nontrivial), then yes, worth the experiment to create the change we want to see.

  • http://www.researchremix.org Heather Piwowar

    Thanks for the friendfeed link, good stuff.

    Hrm, good on you, I’m still struggling with the not-only-in-the-OA-literature question, though I think I agree with you (esp if preprint can be made avail ;) )

    Agreed, love-in possibility for existing journals too. But if they aren’t trying to change the world, then it is ok if they have a limited involvement/readership. Less so with this open science stuff?

    I think the work required in making it a real journal article is needed, regardless. That work is what a peer reviewed paper represents, at least to your funders.

    I do think overlay journals have value, agreed.

    If we can figure out a way to make it work given all time constraints (nontrivial), then yes, worth the experiment to create the change we want to see.

  • http://openwetware.org/wiki/User:Cameron_Neylon Cameron Neylon

    Re: Non OA – plenty of fields have limited penetration of OA journals, particularly journals which people actually pay any attention to articles outside their immediate sphere of interest. Hence articles in Nature/Science/Cell/JACS/EMBO J etc. have an important role to play IMO.

    Re: time contstraints – non-trival, very much so – blub..blub…..bl…!

  • http://openwetware.org/wiki/User:Cameron_Neylon Cameron Neylon

    Re: Non OA – plenty of fields have limited penetration of OA journals, particularly journals which people actually pay any attention to articles outside their immediate sphere of interest. Hence articles in Nature/Science/Cell/JACS/EMBO J etc. have an important role to play IMO.

    Re: time contstraints – non-trival, very much so – blub..blub…..bl…!

  • http://gunther-eysenbach.blogspot.com/ Gunther Eysenbach

    I couldn’t agree more – and thanks for mentioning WebCite (http://www.webcitation.org)
    See also: Eysenbach, Gunther. The paradox of the current state of scholarly communication in the age of Web 2.0 Posted at http://gunther-eysenbach.blogspot.com/2008/01/paradox-of-current-state-of-scholarly.html Jan 18, 2008. Archived at http://www.webcitation.org/5Ux2jEb70

  • http://gunther-eysenbach.blogspot.com Gunther Eysenbach

    I couldn’t agree more – and thanks for mentioning WebCite (http://www.webcitation.org)
    See also: Eysenbach, Gunther. The paradox of the current state of scholarly communication in the age of Web 2.0 Posted at http://gunther-eysenbach.blogspot.com/2008/01/paradox-of-current-state-of-scholarly.html Jan 18, 2008. Archived at http://www.webcitation.org/5Ux2jEb70

  • http://openwetware.org/wiki/User:Cameron_Neylon Cameron Neylon

    Gunther, thanks for dropping by and providing the link. On the associated Friendfeed commentary I expressed the view that it probably wouldn’t be appropriate to put the burden of providing the primary guarantee of stability onto WebCite because this seems philosophically inappopriate (if the argument is we can keep costs down by not being the ‘publisher’ then passing that on to WebCite seems cheating, even if there is a payment that goes with it). Do you have a feeling on this?

  • http://openwetware.org/wiki/User:Cameron_Neylon Cameron Neylon

    Gunther, thanks for dropping by and providing the link. On the associated Friendfeed commentary I expressed the view that it probably wouldn’t be appropriate to put the burden of providing the primary guarantee of stability onto WebCite because this seems philosophically inappopriate (if the argument is we can keep costs down by not being the ‘publisher’ then passing that on to WebCite seems cheating, even if there is a payment that goes with it). Do you have a feeling on this?

  • http://sharescienceideas.wikispaces.com/ Noam

    Hi Cameron,

    You and your ideas are inspirational. Collaborative online peer-review ‘publishing’ is the way forward. I do agree that a few more converts outside the current choir of Open Science are needed before your approach would have more credibility.

    In the long run, I think all grant/research proposals should be written openly and collaboratively in a Wiki-style format, with an entirely revised system of apportioning credit and dividing funding/labor among different labs… I would love to help, but have only sporadically had the time to work on something along those lines, at http://sharescienceideas.wikispaces.com/

    Thanks! Noam

  • http://sharescienceideas.wikispaces.com/ Noam

    Hi Cameron,

    You and your ideas are inspirational. Collaborative online peer-review ‘publishing’ is the way forward. I do agree that a few more converts outside the current choir of Open Science are needed before your approach would have more credibility.

    In the long run, I think all grant/research proposals should be written openly and collaboratively in a Wiki-style format, with an entirely revised system of apportioning credit and dividing funding/labor among different labs… I would love to help, but have only sporadically had the time to work on something along those lines, at http://sharescienceideas.wikispaces.com/

    Thanks! Noam