Category: Blog

Posted on

Won’t someone please think of the policy wonks?

I wouldn’t normally try to pick a fight with Chad Orzel, and certainly not over a post which I mostly agree with, but I wanted to take some issue with the slant in his relatively recent post We are science (see also a good discussion in the comments).  Chad makes a cogent argument that there is a lot of whining about credit and rewards and that ‘Science’ or ‘The Powers That Be’ are blamed for a lot of these things. His point is that ‘We are science’ – and that it is the community of scientists or parts of it that makes these apparently barmy decisions. And as part of the community, if we want to see change, it is our responsibility to get on and change things. There is a strong case for more grass-roots approaches and in particular for those of us with some influence over appointments and promotions procedures to make the case in those committees for widening criteria. I certainly don’t have any problem with his exhortation to ‘be the change’ rather than complain about it. I would hope that I do reasonably good, though by no means perfect,  job of trying to live by the principles I am advocate.

Yet at the same time I think his apparent wholesale rejection of top-down approaches is a bit too much. There is a place for advocating changes in policy and standards. It is not always the right approach, either because it is not the right time or place, but sometimes it is. One reason for advocating changes in policy and standards is to provide a clear message about what the aspirations of a community or funding body are. This is particularly important in helping younger researchers assess the risks and benefits of taking more open approaches. Many of the most energetic advocates of open practice are actually in no position to act on their beliefs because as graduate students and junior postdocs they have no power to make the crucial decisions over where to publish and how (and whether) to distribute data and tools.

Articulations of policy such as the data sharing statements required by the UK BBSRC make it clear that there is an aspiration to move in this direction, that funding will be linked to delivering on these targets. This will both encourage young scientists to make the case to their PIs that this is the way forward and will also influence hiring committees. Chad makes the point that a similar mandate on public outreach for NSF grants has not been taken seriously by grantees. Here I would agree. There is no point having such policies if they are not taken seriously. But everything I have seen and heard so far suggests that the BBSRC does intend to take their policy and delivery on the data sharing statements very seriously indeed.

Top down initiatives are also sometimes needed to drive infrastructure development. The usability of the tools that will be needed to deliver on the potential of data and process sharing is currently woefully inadequate. Development is necessary and funding for this is required. Without the clear articulation from funders that this is the direction in which they wish to go, that they expect to see standards rising year on year, and without them backing that up with money, then nothing much will happen. Again BBSRC has done this, explicitly stating that it expects funding requestst to include support for data availability. The implication is that if people haven’t thought about the details of what they will do and its costs there will be questions asked. I wonder whether this was true of the NSF outreach scheme?

Finally, policy and top-down fiat has the potential, when judiciously applied to accelerate change. Funders, and indeed governments want to see better value for money on the research investment and see greater data availability as one way of achieving that. Chad actually provides an example of this working. The NIH deposition mandate has significantly increased the proportion of NIH funded papers available in PubMedCentral (I suspect Chad’s figure of 30% is actually taken from my vague recollection of the results of the Wellcome Trust Mandate – I think that current evidence suggests that the NIH mandate is getting about 50% now – just saw a graph of this somewhere but can’t find it now! Mark Siegal in the comments provided a link to the data in an article in Science (sorry behind paywall) – here). Clearly providing funding and policy incentives can move us further in that direction quicker. Anybody who doesn’t believe that funding requirements can drive the behaviour of research communities extremely effectively clearly hasn’t applied for any funding in the last decade.

But it remains the case that policy and funding is a blunt instrument. It is far more effective in the long term to bring the community with you by persuasion than by force. A community of successful scientists working together and changing the world is a more effective message in many ways than a fiat from the funding body. All I’m saying is that a combination of both is called for.

Posted on

Thinking about peer review of online material: The Peer Reviewed Journal of Open Science Online

I hold no particular candle for traditional peer review. I think it is inefficient, poorly selective, self reinforcing, often poorly done, and above all, far too slow. However I also agree that it is the least worst system we have available to us.  Thus far, no other approaches have worked terribly well, at least in the communication of science research. And as the incumbent for the past fifty years or so in the post of ‘generic filter’ it is owed some respect for seniority alone.

So I am considering writing a fellowship proposal that would be based around the idea of delivering on the Open Science Agenda via three independent projects, one focussed on policy and standards development, one on delivering a shareable data analysis pipeline for small angle scattering as an exemplar of how a data analysis process can be shared, and a third project based around building the infrastructure for embedding real science projects involving chemistry and drug discovery in educational and public outreach settings. I think I can write a pretty compelling case around these three themes and I think I would be well placed to deliver on them, particularly given the collaborative support networks we are already building in these areas.

The thing is I have no conventional track record in these areas. There are a bunch of papers currently being written but none that will be out in print by the time the application is supposed to go in. My recorded contribution in this area is in blog posts, blog comments, presentations and other material, all of which are available online. But none of which are peer-reviewed in the conventional sense.

One possibility is to make a virtue of this – stating that this is a rapidly moving field – that while papers are in hand and starting to come out that the natural medium for communication with the specific community is online through blogs and other media. There is an argument that conventional peer review simply does not map on to the web of data, tools, and knowledge that is starting to come together and that measuring a contribution in this area by conventional means is simply misguided.  All of which I agree with in many ways.

I just don’t think the referees will buy it.

Which got me thinking. It’s not just me, many of the seminal works for the Open Science community are not peer reviewed papers. Bill Hooker‘s three parter [1, 2, 3] at Three Quarks Daily comes to mind, as does Jean-Claude’s presentation on Nature Precedings on Open Notebook Science, Michael Nielsen’s essay The Future of Science, and Shirley Wu’s Envisioning the scientific community as One Big Lab (along with many others). It seems to me that these ought to have the status of peer reviewed papers which raises the question. We are a community of peers, we can referee, we can adopt some sort of standard of signficance and decide to apply that selectively to specific works online. So why can’t we make them peer reviewed?

What would be required? Well a stable citation obviously, so probably a DOI and some reasonably strong archival approach, probably using WebCite.  There would need to be a clear process of peer review, which need not be anonymous, but there would have to be a clear probity trail to show that an independent editor or group of referees made a decision and that appropriate revisions had been made and accepted. The bar for acceptance would also need to be set pretty high to avoid the charge of simply rubber stamping a bunch of online material. I don’t think open peer review is a problem for this community so many of the probity questions can be handled by simply having the whole process out in the open.

One model would be for an item to be submitted by posting a link on a new page on an independent Wiki . This would then be open to peer review. Once three (five?) independent reviewers had left comments and suggestions – and a version of the document created that satisfied them posted – then the new version could be re-posted at the author’s site, in a specified format which would include the DOI and arhival links, along with a badge that would be automatically aggregated to create the index a la researchblogging.org. There would need to be a charge, either for submission or acceptance – submission would keep volume down and (hopefully) quality up.

How does this differ from setting up a journal? Well two major things – one is that the author remains the publisher so the costs of publication per se are taken out of the equation. This is important as it keeps costs down – not zero, there is still the cost of the DOI and (even if it is donated) the time of editors and referees in managing the process and giving a stamp of authority. The main cost is in maintaining some sort of central index and server pointing out at the approved items. It would also be appropriate to support WebCite if that is the backstop archive. But the big costs for journals are in providing storage that is stable in the long term and managing peer review. If the costs of storage are offloaded and  the peer review process can be self organised then the costs drop significantly.

The second major advantage is that, as a community we already do a lot of this, looking over blog posts, linking to presentations, writing commentary or promoting them on FriendFeed. The reason why ArXiv worked was that there was already a culture of preprints amongst that community. The reason why commenting, rating,  and open peer review trials have not been as successful as people had hoped is because there is no pre-existing culture of doing these things. We already have a culture of open peer review in our community. Is it worth formalising it for the really high quality material that’s already out there?

I am aware that this goes against many of the principles of open and continuous review that many of you hold dear but I think it could serve two useful purposes. First it means that members of the community, particularly younger members, can bolster their CV with peer reviewed papers. Come the revolution this won’t matter but we’re not there yet. Making these contributions tangible for people could be quite powerful. Secondly it takes the important material out of the constant stream of objects flitting past on our screens and gives them a static (I won’t say permanent) priviledged place as part of the record of this field.  Many of them perhaps already have this but I think there is a value in formalising it. Is it worth considering? This proposal is out for review.

 

Posted on

Q&A in this week’s Nature – one or two (minor) clarifications

So a bit of a first for me. I can vaguely claim to have contributed to two things into the print version of Nature this week. Strictly speaking my involvement in the first, the ‘From the Blogosphere‘ piece on the Science Blogging Challenge, was really restricted to discussing the idea (originally from Richard Grant I believe) and now a bit of cheerleading and ultimately some judging. The second item though I can claim some credit for in as much as it is a Q&A with myself and Jean-Claude Bradley that was done when we visited Nature Publishing Group in London a few weeks back.

It is great that a journal like Nature views the ideas of data publication, open notebook science, and open science in general as worthy of featuring. This is not an isolated instance either, as we can point to the good work of the Web Publishing Group, in developing useful resources such as Nature Precedings, as well as previous features in the print edition such as the Horizons article (there is also another version on Nature Precedings) written by Peter Murray-Rust. One thing I have heard said many times in recent months is that while people who advocate open science may not agree with everything NPG with respect to copyright and access, people are impressed and encouraged by the degree of engagement that they maintain with the community.

I did however just want to clarify one or two of the things I apparently said. I am not claiming that I didn’t say those things – the interview was recorded after all – but just that on paper they don’t really quite match what I think I meant to say. Quoting from the article:

CN-Most publishers regard what we do as the equivalent of presenting at a conference, or a preprint. That hasn’t been tested across a wide range of publishers, and there’s at least one — the American Chemical Society — that doesn’t allow prepublication in any form whatsoever.

That sounds a little more extreme than what I meant to say – there are a number of publishers that don’t allow submission of material that has appeared online as a pre-print and the ACS has said that they regard online publication as equivalent to a pre-print. I don’t have any particular sympathy for the ACS but I think they probably do allow publication of material that was presented at ACS conferences.

CN-Open notebooks are practical but tough at the moment. My feeling is that the tools are not yet easy enough to use. But I would say that a larger proportion of people will be publishing electronically and openly in ten years.

Here I think what I said is too conservative on one point and possibly not conservative enough on the other. I did put my neck out and say that I think the majority of scientists will be using electronic lab notebooks of one sort or another in ten years. Funder data sharing policies will drive a much greater volume of material online post publication (hopefully with higher quality description) and this may become the majority of all research data. I think that more people will be making more material available openly as it is produced as well but I doubt that this will be a majority of people in ten years – I hope for a sizeable and significant minority and that’s what we will continue to work towards.

Posted on

The distinction between recording and presenting – and what it means for an online lab notebook

Something that has been bothering me for quite some time fell into place for me in the last few weeks. I had always been slightly confused by my reaction to the fact that on UsefulChem Jean-Claude actively works to improve and polish the description of the experiments on the wiki. Indeed this is one of the reasons he uses a wiki as the process of making modifications to posts on blogs is generally less convenient and in most cases there isn’t a robust record of the different versions. I have always felt uncomfortable about this because to me a lab book is about the record of what happened – including any mistakes in recording you make along the way. There is some more nebulous object (probably called a report) which aggregates and polishes the description of the experiments together.

Now this is fine, but point is that the full history of a UsefulChem page is immediately available from the history. So the full record is very clearly there – it is just not what is displayed. In our system we tend to capture a warts and all view of what was recorded at the time and only correct typos or append comments or observations to a post. This tends not be very human readable in most cases – to understand the point of what is going on you have to step above this to a higher level – one which we are arguably not very good at describing at the moment.

I had thought for a long time that this was a difference between our respective fields. The synthetic chemistry of UsefulChem lends itself to a slightly higher level description where the process of a chemical reaction is described in a fairly well defined, community accepted, style. Our biochemistry is more a set of multistep processes where each of those steps is quite stereotyped. In fact for us it is difficult to define where the ‘experiment’ begins and end. This is at least partly true, but actually if you delve a little deeper and also have a look at Jean-Claude’s recent efforts to use a controlled vocabulary to describe the synthetic procedures a different view arises. Each line of one these ‘machine readable’ descriptions actually maps very well onto each of our posts in the LaBLog. Something that maps on even better is the log that appears near the bottom of each UsefulChem page. What we are actually recording is rather similar. It is simply that Jean-Claude is presenting it at a different level of abstraction.

And that I think is the key. It is true that synthetic chemistry lends itself to a slightly different level of abstraction than biochemistry and molecular biology, but the key difference actually comes in motivation. Jean-Claude’s motivation from the beginning has been to make the research record fully available to other scientists; to present that information to potential users. My focus has always been on recording the process that occurs in the lab and particular to capture the connections between objects and data files. Hence we have adopted a fine grained approach that provides a good record, but does not necessarily make it easy for someone to follow the process through. On UsefulChem the ideal final product contains a clear description of how to repeat the experiment. On the LaBLog this will require tracking through several posts to pick up the thread.

This also plays into the discussion I had some months ago with Frank Gibson about the use of data models. There is a lot to be said for using a data model to present the description of an experiment. It provides all sorts of added value to have an agreed model of what these descriptions look like. However it is less clear to me that it provides a useful way of recording or capturing the research process as it happen, at least in a general case. Stream of consciousness recording of what has happened, rather than stopping halfway through to figure out how what you are doing fits into the data model, is what is required at the recording stage. One of the reasons people feel uncomfortable with electronic lab notebooks is that they feel they will lose the ability to scribble such ‘free form’ notes – the lack of any presuppositions about what the page should loook like is one of the strengths of pen and paper.

However, once the record, or records, have been made then it is appropriate to pull these together and make sense of them – to present the description of an experiment in a structured and sensible fashion. This can of course be linked back to the primary records and specific data files but it provides a comprehensible and fine grained descriptionof the rationale for and conduct of the experiment as well as placing the results in context. This ‘presentation layer’ is something that is missing from our LaBLog but could relatively easily be pulled together by writing up the methodology section for a report. This would be good for us and good for people coming into the system looking for specific information.

Person Frank Gibson

Right click for SmartMenu shortcuts

Posted on

The long slow catchup…

I’m a little shell shocked really. I’ve spent the last couple of weeks running around like a lunatic, being at meetings, organising meetings, flying out to other meetings. And then flying back to try and catch up with all the things that need doing before the next flurry of activity strikes (which involves less travel and more experiments you will be pleased to know). There are two things I desperately need to write up.

The Open Science workshop at Southampton on September 1 seemed to be well received and was certainly interesting for me.  Despite having a very diverse group of people we did seem to manage to have a sensible discussion that actually came to some conclusions. This was followed up by discussions with the web publishing group at Nature where some of these ideas were refined – more on this will follow!

Following on from this (and with a quick afternoon jaunt to Bristol for the Bristol Knowlege Unconference on the evening of September 5 I flew to Toronto en route to Waterloo for Science in the 21st Century, allowing for a brief stop for a Nature Network Toronto pub night panel session with Jen Dodd, Michael Nielsen, and Timo Hannay. The organisers of Science21, but in particular Sabine Hossenfelder, deserve huge congratulations for putting together one of the most diverse and exciting conferences I have ever been to. With speakers from historians to sociologists, hedge fund managers to writers, and even the odd academic scientist the sheer breadth of material covered was quite breathtaking.

You can see most of the talks and associated material on the Perimeter Institute Seminar Archive page here. The friendfeed commentary is also available in the science21 room. Once again it was a great pleasure to meet people I kind of knew but hadn’t ever actually met such as Greg Wilson and John Dupuis as well as to meet new people including (but by no means limited to) Harry Collins, Paul Guinnessy, and David Kaiser. We have yet to establish whether I knew Jen Dodd in a previous life…

Very many ideas will come out of this meeting I think – and I have no doubt you will see some interesting blog posts from others with the science21 tag coming out over the next few weeks and months. A couple of particular things I will try to follow up on;

  • Harry Collins spoke about categorisations of tacit (i.e. non-communicated) knowledge and how these relate to different categories of expertise. This has obvious implications for our mission to describe our experiments to a level where there is ‘no insider information’. The idea that we may be able to rationally describe what we can and cannot expect to be able to communicate and that we can therefore concentrate on the things that we can is compelling.
  • Greg Wilson made a strong case for the fully supported experiment that echoed my own thoughts about the recording of data analysis procedures. He was focussed on computational science but I think his point goes much wider than that. This requires some thought and processing but for me it is clear that the big challenge in communicating the details of our experiments now clearly lies in communicating process rather than data.

Each of these deserves its own post and will hopefully get it. And I am also aware that I owe many of you comments, replies, or other things – some more urgent than others. I’ll be getting to them as soon as I can dig myself out from under this pile of……

Posted on

The trouble with semantics…

…is knowing what you mean…

I posted last week about the spontaneous CMLReact hackfest held around Peter Murray-Rust’s dining room table the day after Science Blogging in London. There were a number of interesting things that came out of the exercise for me. The first was that it would be relatively easy to design a moderately strict, but pretty standard, description format for a synthetic chemistry lab notebook that could be automatically scraped into CMLReact.

Automatic conversions from lab book to machine readable XML

CMLReact files have (roughly) three sections. In the first, all the molecules that are relevant to the description are described, or in the ideal semantic web world pointed to at an external authority such as Chemspider, PubChem, or other source. In the second section the relationships between input materials, solvents, products, and samples are described. In general all of these will be molecules which are referred to in the first session but this is not absolutely required (and this will be important later). The final section describes observables, procedures, yields, and other descriptions of what happened or what was measured.

If we take a look at the UsefulChem experiment that we converted to CMLReact you can see that most of this information is available in one form or another. The molecules are described via InChi/InChiKey at the bottom of the page. This could be used as they are to populate the molecules section. A little additional markup to distinguish between reactants, solvents, reagents, and products would make it possible to start populating the second section describing the relationships between these molecules.

The third section is the most tricky, and this will always be an 80:20 game. The object is to abstract as much information as can be reasonably garnered without putting in the vast amount of work required to get close to 100% retrieval. At the end of the day, if someone wants the real detail they can go back to the lab book. Peter has demonstrated text scraping tools that do a pretty good job of extracting a lot of this information. In combination with a bit of markup it is reasonable to expect that some basic information (amounts of reagents, yield, temperature of reaction, some descriptive terms) could reasonably be extracted. Again, getting 80-90% of a subset of regularly used terms  would be very powerful.

But what are we describing?

There is a problem with grabbing this descriptive information from the lab notebook however, and it is a problem that is very general and something I believe we need to grapple with urgently. There is a fundamental question as to what it is that this file is describing. Does it describe the plan of the experiment? The record of carrying out a specific example of this experiment? An ‘averaged’ description of a set of equivalent experiments? A general description of the reaction? Or a description of a model of what we expect or think is happening?

If you look closely at the current version of the CMLReact file you will see that the yield is expressed as a percentage with a standard deviation. This is actually describing the average of three independent reactions but that is not actually made explicit anywhere in this file. Is this important? Well I think it is because it has an effect on what any outward links back to the lab book mean. There is a significant difference between – ‘this link points to an example of this kind of reaction’ (which might in fact be significantly different in the details) and ‘this link points to this exact experiment’ or indeed ‘this link points to an index of relevant experimental results’. Those distinctions need to be encoded in the links, or perhaps more likely made explicit in the abstracted file.

The CMLReact file is an abstraction of the experimental record. It is therefore important to make it clear what the level of abtraction is and what has been abstracted out of that description. This relates to the distinction I have made before between the flexibility required to record an experiment versus the ability to use a more structured vocabulary to describe the experiment after it has happened. My impression is that people who work in developing these controlled vocabularies are focussed on description rather than recording and don’t often make the distinction between the two. There is also often a lack of distinction between describing an experiment and describing a model of what happened in that experiment.  This is important because the model may need to be modified in the future whereas the description of the experiment should be accurate.

Summary

My view remains that when recording an experiment the system used should be as flexible as possible. Structure can be added to this primary record when convenient to make the process of abstracting from this primary record to a controlled vocabulary easier. The primary goal for me, for the moment, remains making a human readable record available. The process of converting the primary record into a controlled vocabulary, such as CMLReact, FuGE, or workflow system such as Taverna, should be enabled via domain specific automated or semi-automated tools that help the user to structure their description of the experiment in a way that makes it more directly useful to them but maintains the links with the primary record. Where the same controlled vocabulary is used for more abstracted descriptions of studies, experiments, or the models that purport to describe them, this distinction must be made clear.

Semantics depends absolutely on being clear about what you are describing. There is absolutely no point in having absolute clarity about the description of an object if the nature of that object is fuzzy. Get it right and we could have a very sophisticated description of the scientific record. Get it wrong and that description could be at best unclear and at worst downright misleading.

Posted on

The Open Science Endurance Event – Team JC-C

So far we’ve had a fun week. Jean-Claude arrived in the UK on Thursday last, followed up with a talk at Bath University to people at UKOLN on Friday. The talk kicked off an extended conversation which meant we were very late to lunch but it was great to follow up on issues from a different perspective to that. Jean-Claude will be making a screencast of the talk available on his Drexel-CoAs Podcast blog.

On Friday afternoon we headed into London in preparation for Science Blogging 2008 which was a blast. A very entertaining keynote by Ben Goldacre of Bad Science was followed up by a fascinating series of sessions on everything from Open Notebooks to connecting up conversations separated in time and space to creativity, blogging boredom, and unicycling giraffes. The sessions were great fun and there was lots of back chat on friendfeed but in many ways the best part of this for me was the chance to meet people (if in many cases very briefly) that I knew well but had never actually met in person. Too many to mention all but in particular it was a pleasure to finally meet Michael Barton, Richard Grant, Heather Etchevers, Matt Wood, Graham Steel, and many people from NPG as well as to meet many old friends. There is pretty good coverage of the meeting itself so I will simply point at the technorati tag and be done with that.

I was one of the people on the final panel with Peter Murray-Rust, Richard Grant, and moderation from Timo Hannay. This was a fun conversation with lots of different perspectives. I think the overall conclusion was that the idea that ‘blogging is bad for your career’ is shifting towards ‘why do you put all this work in?’ There was a strong sense that some people had made real personal gains out of blogging and online activities and that many organisations are starting to see it as a valuable contribution. Nonetheless it is not an activity that is widely valued, or indeed even known about. To this end the panel offered a challenge – to persuade a senior scientist to start writing a blog. One prize will be to be featured in next year’s Open Lab 2008 – the best of science writing on the web. The other prize – which caused an extensive collective intake of breath – will be an all expenses paid trip to Scifoo next year for both blogger and the encourager. The announcement will probably be made with details by the time I get to post this.

In the pub on the Saturday night, Peter M-R grabbed me and JC and Egon Willighagen and said ‘why don’t you come up to Cambridge tomorrow?’ So we all did and I and Egon have written briefly about that already. More work to do, and some interesting things to discuss, which I hope to follow up later.

Sunday afternoon- dash back to Southampton for the introductory dinner for the Open Science Workshop held at Southampton Uni on Monday. This was a really great meeting from my perspective with a real mix of tools people and ‘practicing’ scientists, computer scientists, chemists, biologists, and people with business degrees. There is more at the Wiki and on Friendfeed – but this will need a write up of its own. Hopefully slides will be made available for most of the talks and we will point to them from both Wiki and Friendfeed.

Tuesday, more meetings and planning, with a great meeting with Dave De Roure of Southampton Electronics and Computer Science and in particular the PI for the MyExperiment project.  Some good stuff will come out of this – and the contact between Dave and Chemspider has been made. The MyExperiment team are keen on delivering more for chemistry so that link will be important. However I was particularly taken with a throw away comment Dave made that workflows (and makefiles) have a direct equivalence with spreadsheets. This made me think immediately of that great divide between ‘those who use excel for everything’ and ‘those who run screaming in the other direction and would rather hard code in perl on a clay tablet’ for analysis. If we could actually leverage the vast number of analytical spread sheets sitting on a million hard drives we might be able to do some very interesting stuff. Hopefully more on this in a future post.

Wednesday we did some experiments – its mostly up online now so you can go see if you are interested. And today we are heading up to London to see the folks at Nature Publishing Group which should be fun. More opportunity to talk in detail about ideas from Saturday and the role of the publisher and papers in the future.  Had a lovely lunch with the NPG web publishing people, talks seemed to go reasonably well, and a quick chat with the people from Nature Chemistry

But it doesn’t stop there. Tomorrow JC goes to Manchester to give a talk, then heads to Edinburgh for the e-Science All Hands Meeting, including a workshop on ‘The Global Data Centric View’. Then he heads to Oxford for another meeting  and talk before finally heading back to Philadelphia. On Sunday I fly to Toronto, with a Nature Network pub session on the Sunday evening (wow I am going to be on scintillating form for that!) followed by Science in the 21st Century for the following week.

I think we are going to need a rest when we get to our respective homes again…

Posted on

Linking up open science online

I am currently sitting at the dining table of Peter Murray-Rust with Egon Willighagen opposite me talking to Jean-Claude Bradley. We pulling together sets of data from Jean-Claude’s UsefulChem project into CML to make it more semantically rich and do a bunch of cool stuff. Jean-Claude has a recently published preprint on Nature Precedings of a paper that has been submitted to JoVE. Egon was able to grab the InChiKeys from the relevant UsefulChem pages and passing those to CDK via a script that he wrote on the spot (which he has also just blogged) generated CML react for those molecules.

Peter at the same time has cut and pasted an existing example of a CML react XML document into a GoogleDoc which we will then modify to represent one example of the Ugi reactions that Jean-Claude reported in the precedings paper. You will be able to see all of these documents. The only way we would be able to do this is with four laptops all online – and for all the relevant documents and services to be available from where we are sitting. I’ve never been involved in a hackfest like this before but actually the ability to handle different aspects of the same document via GoogleDocs is a very powerful way of handling multiple processes at the same time.

Posted on

Can post publication peer review work? The PLoS ONE report card

This post is an opinion piece and not a rigorous objective analysis. It is fair to say that I am on the record as and advocate of the principles behind PLoS ONE and am also in favour of post publication peer review and this should be read in that light. [ed I’ve also modified this slightly from the original version because I got myself mixed up in an Excel spreadsheet]

To me, anonymous peer review is, and always has been, broken. The central principle of the scientific method is that claims and data to support those claims are placed, publically, in the view of expert peers. They are examined, and re-examined on the basis of new data, considered and modified as necessary, and ultimately discarded in favour of an improved, or more sophisticated model. The strength of this process is that it is open, allowing for extended discussion on the validity of claims, theories, models, and data. It is a bearpit, but one in which actions are expected to take place in public (or at least community) view. To have as the first hurdle to placing new science in the view of the community a process which is confidential, anonymous, arbitrary, and closed, is an anachronism.

It is, to be fair, an anachronism that was necessary to cope with rising volumes of scientific material in the years after the second world war as the community increased radically in size. A limited number of referees was required to make the system manageable and anonymity was seen as necessary to protect the integrity of this limited number of referees. This was a good solution given the technology of the day. Today, it is neither a good system, nor an efficient system, and we have in principle the ability to do peer review differently, more effectively, and more efficiently. However, thus far most of the evidence suggests that the scientific community dosen’t want to change. There is, reasonably enough, a general attitude that if it isn’t broken it doesn’t need fixing. Nonetheless there is a constant stream of suggestions, complaints, and experimental projects looking at alternatives.

The last 12-24 months have seen some radical experiments in peer review. Nature Publishing Group trialled an open peer review process. PLoS ONE proposed a qualitatively different form of peer reivew, rejecting the idea of ‘importance’ as a criterion for publication. Frontiers have developed a tiered approach where a paper is submitted into the ‘system’ and will gradually rise to its level of importance based on multiple rounds of community review. Nature Precedings has expanded the role and discipline boundaries of pre-print archives and a white paper has been presented to EMBO Council suggesting that the majority of EMBO journals be scrapped in favour of retaining one flagship journal for which papers would be handpicked from a generic repository where authors would submit, along with referees’ reports and author’s response, on payment of a submission charge. Of all of these experiments, none could be said to be a runaway success so far with the possible exception of PLoS ONE. PLoS ONE, as I have written before, succeeded precisely because it managed to reposition the definition of ‘peer review’. The community have accepted this definition, primarily because it is indexed in PubMed. It will be interesting to see how this develops.

PLoS has also been aiming to develop ratings and comment systems for their papers as a way of moving towards some element of post publication peer review. I, along with some others (see full disclosure below) have been granted access to the full set of comments and some analytical data on these comments and ratings. This should be seen in the context of Euan Adie’s discussion of commenting frequency and practice in BioMedCentral journals which broadly speaking showed that around 2% of papers had comments and that these comments were mostly substantive and dealt with the science. How does PLoS ONE compare and what does this tell us about the merits or demerits of post publication peer review?

PLoS ONE has a range of commenting features, including a simple rating system (on a scale of 1-5) the ability to leave freetext notes, comments, and questions, and in keeping with a general Web 2.o feel the ability to add trackbacks, a mechanism for linking up citations from blogs. Broadly speaking a little more than 13% (380 of 2773) of all papers have ratings and around 23% have comments, notes, or replies to either (647 of 2773, not including any from PLoS ONE staff) . Probably unsurprisingly most papers that have ratings also have comments. There is a very weak positive correlation between the number of citations a paper has received (as determined from Google Scholar) and the number of comments (R^2 = 0.02, which is probably dominated by papers with both no citations and no comments, which are mostly recent, none of this is controlled for publication date).

Overall this is consistent with what we’d expect. The majority of papers don’t have either comments or ratings but a significant minority do. What is slightly suprising is that where there is arguably a higher barrier to adding something (click a button to rate versus write a text comment) there is actually more activity. This suggests to me that people are actively uncomfortable with rating papers versus leaving substantive comments. These numbers compare very favourably to those reported by Euan on comments in BioMedCentral but they are not yet moving into the realms of the majority. It should also be noted that there has been a consistent  programme at PLoS ONE with the aim of increasing the involvement of the community. Broadly speaking I would say that the data we have suggest that that programme has been a success in raising involvement.

So are these numbers ‘good’? In reality I don’t know. They seem to be an improvement on the BMC numbers arguing that as systems improve and evolve there is more involvement. However, one graph I received seems to indicate that there isn’t an increase in the frequency of comments within PLoS ONE over the past year or so which one would hope to see. Has this been a radical revision of how peer review works? Well not yet certainly, not until the vast majority of papers have ratings, but more importantly not until we have evidence that people are using those ratings. We are not yet in a position where we are about to see a stampede towards radically changed methods of peer review and this is not surprising. Tradition changes slowly – we are still only just becoming used to the idea of the ‘paper’ being something that goes beyond a pdf, embedding that within a wider culture of online rating and the use of those ratings will take some years yet.

So I have spent a number of posts recently discussing the details of how to make web services better for scientists. Have I got anything useful to offer to PLoS ONE? Well I think some of the criteria I suggested last week might be usefully considered. The problem with rating is that it lies outside the existing workflow for most people. I would guess that many users don’t even see the rating panel on the way into the paper. Why would people log into the system to look at a paper? What about making the rating implicit when people bookmark a paper in external services? Why not actually use that as the rating mechanism?

I emphasised the need for a service to be useful to the user before there are any ‘social effects’ present. What can be offered to make the process of rating a paper useful to the single user in isolation? I can’t really see why anyone would find this useful unless they are dealing with huge number of papers and can’t remember which one is which from day to day. It may be useful within groups or journal clubs but all of these require a group to sign up.  It seems to me that if we can’t frame it as a useful activity for a single person then it will be difficult to get the numbers required to make this work effectively on a community scale.

In that context, I think getting the numbers to around the 10-20% level for either comments or ratings has to be seen as an immense success. I think it shows how difficult it is to get scientists to change their workflows and adopt new services. I also think there will be a lot to learn about how to improve these tools and get more community involvement. I believe strongly that we need to develop better mechanisms for handling peer review and that it will be a very difficult process getting there. But the results will be seen in more efficient dissemination of information and more effective communication of the details of the scientific process. For this PLoS, the PLoS ONE team, as well as other publishers, including BioMedCentral, Nature Publishing Group, and others, that are working on developing new means of communication and improving the ones we have deserve applause. They may not hit on the right answer first off, but the current process of exploring the options is an important one, and not without its risks for any organisation.

Full disclosure: I was approached along with a number of other bloggers to look at the data provided by PLoS ONE and to coordinate the release of blog posts discussing that data. At the time of writing I am not aware of who the other bloggers are, nor have I read what they have written. The data that was provided included a list of all PLoS ONE papers up until 30 July 2008, the number of citations, citeulike bookmarks, trackbacks, comments, and ratings for each paper. I also received a table of all comments and a timeline with number of comments per month. I have been asked not to release the raw data and will honour that request as it is not my data to release. If you would like to see the underlying data please get in contact with Bora Zivkovic.