Category: Blog

Posted on

Wears the passion? Yes it does rather…

Leukemia cells.
Image via Wikipedia

Quite some months ago an article in Cancer Therapy and Biology by Scott Kern of Johns Hopkins kicked up an almighty online stink. The article entitled “Where’s the passion” bemoaned the lack of hard core dedication amongst the younger researchers that the author saw around him…starting with:

It is Sunday afternoon on a sunny, spring day.

I’m walking the halls—all of them—in a modern $59 million building dedicated to cancer research. A half hour ago, I completed a stroll through another, identical building. You see, I’m doing a survey. And the two buildings are largely empty.

The point being that if they really cared, those young researchers would be there day in-day out working their hearts out to get to the key finding. At one level this is risible, expecting everyone to work 24×7 is not a good or efficient way to get results. Furthermore you have to wonder why these younger researchers have “lost their passion”. Why doesn’t the environment create that naturally, what messages are the tenured staff sending through their actions. But I’d be being dishonest if there wasn’t a twinge of sympathy for me as well. Anyone who’s run a group has had that thought that the back of their mind; “if only they’d work harder/smarter/longer we’d be that much further ahead…”.

But all of that has been covered by others. What jumped out of the piece for me at the time were some other passages, ones that really got me angry.

When the mothers of the Mothers March collected dimes, they KNEW that teams, at that minute, were performing difficult, even dangerous, research in the supported labs. Modern cancer advocates walk for a cure down the city streets on Saturday mornings across the land. They can comfortably know that, uh…let’s see here…, some of their donations might receive similar passion. Anyway, the effort should be up to full force by 10 a.m. or so the following Monday.

[…]

During the survey period, off-site laypersons offer comments on my observations. “Don’t the people with families have a right to a career in cancer research also?” I choose not to answer. How would I? Do the patients have a duty to provide this “right”, perhaps by entering suspended animation?

Now these are all worthy statements. We’d all like to see faster development of cures and I’ve no doubt that the people out there pounding the streets are driven to do all they can to see those cures advance. But is the real problem here whether the postdocs are here on a Sunday afternoon or are there things we could do to advance this? Maybe there are other parts of the research enterprise that could be made more efficient…like I don’t know making the results of research widely available and ensuring that others are in the best position possible to build on their results?

It would be easy to pick on Kern’s record on publishing open access papers. Has he made all the efforts that would enable patients and doctors to make the best decisions they can on the basis of his research? His lab generates cell lines that can support further research. Are those freely available for others to use and build on? But to pick on Kern personally is to completely miss the point.

No, the problem is that this is systemic. Researchers across the board seem to have no interest whatsoever in looking closely at how we might deliver outcomes faster. No-one is prepared to think about how the system could be improved so as to deliver more because everyone is too focussed on climbing up the greasy pole; writing the next big paper and landing the next big grant. What is worse is that it is precisely in those areas where there is most public effort to raise money, where there is a desperate need, that attitudes towards making research outputs available are at their worse.

What made me absolutely incandescent about this piece was a small piece of data that some of use have known about for a while but has only just been published. Heather Piwowar, who has done a lot of work on how and where people share, took a close look at the sharing of microarray data. What kind of things are correlated with data sharing. The paper bears close reading (Full disclosure: I was the academic editor for PLoS ONE on this paper) but one thing has stood out from me as shocking since the first time I heard Heather discuss it: microarray data linked to studies of cancer is systematically less shared.

This is not an isolated case. Across the board there are serious questions to be asked about why it seems so difficult to get the data from studies that relate to cancer. I don’t want to speculate on the reasons because whatever they are, they are unnacceptable. I know I’ve recommended this video of Josh Sommer speaking many times before, but watch it again. Then read Heather’s paper. And then decide what you think we need to do about it. Because this can not go on.

Enhanced by Zemanta
Posted on

How to waste public money in one easy step…

Oleic Acid
Image via Wikipedia

Peter Murray-Rust has sparked off another round in the discussion of the value that publishers bring to the scholarly communication game and told a particular story of woe and pain inflicted by the incumbent publishers. On the day he posted that I had my own experience of just how inefficient and ineffective our communication systems are by wasting the better part of the day trying to find some information. I thought it might be fun to encourage people to post their own stories of problems and frustrations with access to the literature and the downstream issues that creates, so here is mine.

I am by no means a skilled organic chemist but I’ve done a bit of synthesis in my time and I certainly know enough to be able to read synthetic chemistry papers and decide whether a particular synthesis is accessible. So on this particular day I was interested in deciding whether it was easy or difficult to make deuterated mono-olein. This molecule can be made by connecting glycerol to oleic acid. Glycerol is cheap and I should have in my hands some deuterated oleic acid in the next month or so. The chemistry for connecting acids to alcohols is straightforward, I’ve even done it myself, but this is a slightly special case. Firstly the standard methods tend to be wasteful of the acid, which in my case is the expensive bit. The second issue is that glycerol has three alcohol groups. I only want to modify one, leaving the other two unchanged, so it is important to find a method that gives me mostly what I want and only a little of what I don’t.

So the question for me is: is there a high yielding reaction that will give me mostly what I want, while wasting as little as possible of the oleic acid? And if there is a good technique is it accessible given the equipment I have in the lab? Simple question, quick trip to Google Scholar, to find reams of likely looking papers, not one of which I had full text access to. The abstracts are nearly useless in this case because I need to know details of yields and methodology so I had several hundred papers, and no means of figuring out which might be worth an inter-library loan. I spent hours trying to parse the abstracts to figure out which were the most promising and in the end I broke…I asked someone to email me a couple of pdfs because I knew they had access. Bear in mind what I wanted to do was spend a quick 30 minutes or so to decide whether this was pursuing in detail. What is took was about three hours, which at full economic cost of my time comes to about £250. That’s about £200 of UK taxpayers money down the toilet because, on the site of the UKs premiere physical and biological research facilities I don’t have access to those papers. Yes I could have asked someone else to look but that would have taken up their time.

But you know what’s really infuriating. I shouldn’t even have been looking at the papers at all when I’m doing my initial search. What I should have been able to do was ask the question:

Show me all syntheses of mono-olein ranked first by purity of the product and secondly by the yield with respect to oleic acid.

There should be a database where I can get this information. In fact there is. But we can’t afford access to the ACS’ information services here. These are incredibly expensive because it used to be necessary for this information to be culled from papers by hand. But today that’s not necessary. It could be done cheaply and rapidly. In fact I’ve seen it done cheaply and rapidly by tools developed in Peter’s group that get around ~95% accuracy and ~80% recall over synthetic organic chemistry. Those are hit rates that would have solved my problem easily and effectively.

Unfortunately despite the fact those tools exist, despite the fact that they could be deployed easily and cheaply, and that they could save researchers vast amounts of time research is being held back by a lack of access to the literature, and where there is access by contracts that prevent us collating, aggregating, and analysing our own work. The public pays for the research to be done, the public pays for researchers to be able to read it, and in most cases the public has to pay again if they should want to read it. But what is most infuriating is the way the public pays yet again when I and a million other scientists waste our time, the public’s time, because the tools that exist and work cannot be deployed.

How many researchers in the UK or world wide are losing hours or even days every week because of these inefficiencies. How many new tools or techniques are never developed because they can’t legally be deployed? And how many hundreds of millions of dollars of public money does that add up to?

Enhanced by Zemanta
Posted on

A new sustainability model: Major funders to support OA journal

Open Access logo and text
Image via Wikipedia

”The Howard Hughes Medical Institute, the Max Planck Society and the Wellcome Trust announced today that they are to support a new, top-tier, open access journal for biomedical and life sciences research. The three organisations aim to establish a new journal that will attract and define the very best research publications from across these fields. All research published in the journal will make highly significant contributions that will extend the boundaries of scientific knowledge.” [Press Release]

It has been clear for some time that the slowness of the adoption of open access publication models by researchers is in large part down to terror that we have of stepping out of line and publishing in the ‘wrong’ journals. More radical approaches to publication will clearly lag even further behind while this inherent conservatism is dominant. Publishers like PLoS and BMC have tackled this head on by aiming to create prestigous journals but the top of the pile has remained the traditional clutch of Nature, Science, and Cell.

The incumbent publishers have simultaneously been able to sit back due to a lack of apparent demand from researchers. As the demand from funders has increased they have held back, complaining the business models to support Open Access publication are not clear. I’ve always found the ‘business model’ argument slightly specious. Sustainability is important but scholarly publishing has never really had a viable business model, it has had a subsidy from funders. Part of the problem has been the multiple layers and channels that subsidy has gone through but essentially funders, through indirect funding of academic libraries, have been footing the bill.

Some funders, and the Wellcome Trust has lead on this, have demanded that their researchers make their outputs accessible while simultaneously requiring publishers comply with their requirements on access and re-use rights. But progress has been slow, particularly in opening up what is perceived as the top of the market. Despite major inroads made by PLoS Biology and PLos Medicine those journals perceived as the most prestigious have remain resolutely closed.

Government funders are mostly constrained in their freedom to act but the Wellcome Trust, HHMI, and Max Planck Society have the independence to take the logical step. They are already paying for publication, why not actively support the formation of a new journal, properly open access, and at the same time lend the prestige that their names can bring?

This will send a very strong message, both to researchers and publishers, about what these funders value, and where they see value for money. It is difficult to imagine this will not lead to a seismic shift in the publishing landscape, at least from a political and financial perspective. I don’t believe this journal will be as technically radical as I would like, but it is unlikely it could be while achieving the aims that it has. I do hope the platform it is built on enables innovation both in terms of what is published and the process by which it is selected.

But in a sense that doesn’t matter. This venture can remain incredibly conservative and still have a huge impact on taking the research communication space forward. What it means is that three of the worlds key funders have made an unequivocal statement that they want to see Open Access, full open access on publication without restrictions on commercial use, or text-mining, or re-use in any form, across the whole of the publication spectrum. And if they don’t get it from the incumbent publishers they’re prepared to make it happen themselves.

Full Disclosure: I was present at a meeting at Janelia Farm in 2010 where the proposal to form a journal was discussed by members of the Wellcome, HHMI, and MPG communities.

Enhanced by Zemanta
Posted on

Cultural Infrastructure: Or why the ‘Big Society’ and post-pub peer review will fail in their current forms

As a wooly headed social liberal with a strong belief in the power of the internet to connect people it is probably natural for to me have some sympathy for the ideas behind the “Big Society” espoused by the UK Prime Minister. The concept here is that many of the support roles traditionally taken by or at least mediated by government can be more effectively and efficiently provided in the community by direct interactions between members of the public. The potential of the web to inform, support critical debate, and above all connect people is enormous. There is a real potential to create markets for exchange of resources, primarily time and money, between those who have and those who need, while naturally tailoring these services to what is really needed at a local level. The efficiency of markets can be combined with a socialist vision of community responsibility combining the best of both political visions and jettisoning the failures of each.

There is of course trouble in utopia. What actually happens as government withdraws as the guarantor or service provision and hands over the reins to local community groups is a mixture of triumph and farce. A few communities are strong enough to engage their members and deliver services competently and effectively but these are usually the wealthy areas that are already well served. More widely a mixture of incompetence (in a purely technical sense) and a simple lack of people prepared or able to carry this load leads to those areas with poor service provision to receive less. There are groups that will step in to take up the slack but they tend to fall into two groups, either wealthy benefactors or groups with a specific agendas, sometimes with extreme viewpoints. Regardless of motivation, these groups come into communities from the outside. In that sense they are just like government, only with less regulation, probably less experience of service provision, and above all, not part of the community.

Building the technical infrastructure to support the Big Society is easy. But just because technical infrastructure is there doesn’t mean that community members have the time, the resources, or most importantly the motivation to use it. The cultural infrastructure, the assumptions that would motivate people to get involved, the support for people to take time off work (or work shorter hours) so as to spend time in the community, and the trust in systems for pooling the necessary resources simply doesn’t exist.  This cultural infrastructure will take a generation to build and will require a focused effort from government, communities, and industry. And it will require money.

So what does this have to do with peer review? Researchers contribute to traditional peer review because it something we expect to do. It is often claimed that we do peer review because we expect the favour of high quality review back in return. I just don’t buy it. If this were true we’d find it much easier to get people to share data (because they’d get data shared back in the future) or materials (similarly). None of this happens because we don’t have a cultural infrastructure that supports and values these behaviours. We do however have an existing cultural infrastructure which supports the need for traditional peer review to be done.

Naturally it follows that those things that aren’t seen as an assumed part of the job don’t get done. Post-publication peer review, despite its potential is one of these things. Data and process sharing are others. Data sharing plans and funder mandates are really just scaffolding. They support the structure that we can see that we want but they aren’t the foundations and pillars that will support a robust community. The moral debate on these issues has fundamentally been won, it is merely a question of implementation and managing the transition. But we now need to look beyond the technical problems and the application of big sticks to understand how we can build the cultural infrastructure that supports a world where reproducibility, effective communication of all research outputs, and ongoing critique of the outputs of others are assumed to be part of the job.

 

Enhanced by Zemanta
Posted on

Evidence to the European Commission Hearing on Access to Scientific Information

European Commission
Image by tiseb via Flickr

On Monday 30 May I gave evidence at a European Commission hearing on Access to Scientific Information. This is the text that I spoke from. Just to re-inforce my usual disclaimer I was not speaking on behalf of my employer but as an independent researcher.

We live in a world where there is more information available at the tips of our fingers than even existed 10 or 20 years ago. Much of what we use to evaluate research today was built in a world where the underlying data was difficult and expensive to collect. Companies were built, massive data sets collected and curated and our whole edifice of reputation building and assessment grew up based on what was available. As the systems became more sophisticated new measures became incorporated but the fundamental basis of our systems weren’t questioned. Somewhere along the line we forgot that we had never actually been measuring what mattered, just what we could.

Today we can track, measure, and aggregate much more, and much more detailed information. It’s not just that we can ask how much a dataset is being downloaded but that we can ask who is downloading it, academics or school children, and more, we can ask who was the person who wrote the blog post or posted it to Facebook that led to that spike in downloads.

This is technically feasible today. And make no mistake it will happen. And this provides enormous potential benefits. But in my view it should also give us pause. It gives us a real opportunity to ask why it is that we are measuring these things. The richness of the answers available to us means we should spend some time working out what the right questions are.

There are many reasons for evaluating research and researchers. I want to touch on just three. The first is researchers evaluating themselves against their peers. While this is informed by data it will always be highly subjective and vary discipline by discipline. It is worthy of study but not I think something that is subject to policy interventions.

The second area is in attempting to make objective decisions about the distribution of research resources. This is clearly a contentious issue. Formulaic approaches can be made more transparent and less easy to legal attack but are relatively easy to game. A deeper challenge is that by their nature all metrics are backwards looking. They can only report on things that have happened. Indicators are generally lagging (true of most of the measures in wide current use) but what we need are leading indicators. It is likely that human opinion will continue to beat naive metrics in this area for some time.

Finally there is the question of using evidence to design the optimal architecture for the whole research enterprise. Evidence based policy making in research policy has historically been sadly lacking. We have an opportunity to change that through building a strong, transparent, and useful evidence base but only if we simultaneously work to understand the social context of that evidence. How does collecting information change researcher behavior? How are these measures gamed? What outcomes are important? How does all of this differ cross national and disciplinary boundaries, or amongst age groups?

It is my belief, shared with many that will speak today, that open approaches will lead to faster, more efficient, and more cost effective research. Other groups and organizations have concerns around business models, quality assurance, and sustainability of these newer approaches. We don’t need to argue about this in a vacuum. We can collect evidence, debate what the most important measures are, and come to an informed and nuanced inclusion based on real data and real understanding.

To do this we need to take action in a number areas:

1. We need data on evaluation and we need to able to share it.

Research organizations must be encouraged to maintain records of the downstream usage of their published artifacts. Where there is a mandate for data availability this should include mandated public access to data on usage.

The commission and national funders should clearly articulate that that provision of usage data is a key service for publishers of articles, data, and software to provide, and that where a direct payment is made for publication provision for such data should be included. Such data must be technically and legally reusable.

The commission and national funders should support work towards standardizing vocabularies and formats for this data as well critiquing it’s quality and usefulness. This work will necessarily be diverse with disciplinary, national, and object type differences but there is value in coordinating actions. At a recent workshop where funders, service providers, developers and researchers convened we made significant progress towards agreeing routes towards standardization of the vocabularies to describe research outputs.

2. We need to integrate our systems of recognition and attribution into the way the web works through identifying research objects and linking them together in standard ways.

The effectiveness of the web lies in its framework of addressable items connected by links. Researchers have a strong culture of making links and recognizing contributions through attribution and citation of scholarly articles and books but this has only recently being surfaced in a way that consumer web tools can view and use. And practice is patchy and inconsistent for new forms of scholarly output such as data, software and online writing.

The commission should support efforts to open up scholarly bibliography to the mechanics of the web through policy and technical actions. The recent Hargreaves report explicitly notes limitations on text mining and information retrieval as an area where the EU should act to modernize copyright law.

The commission should act to support efforts to develop and gain wide community support for unique identifiers for research outputs, and for researchers. Again these efforts are diverse and it will be community adoption which determines their usefulness but coordination and communication actions will be useful here. Where there is critical mass, such as may be the case for ORCID and DataCite, this crucial cultural infrastructure should merit direct support.

Similarly the commission should support actions to develop standardized expressions of links, through developing citation and linking standards for scholarly material. Again the work of DataCite, CoData, Dryad and other initiatives as well as technical standards development is crucial here.

3. Finally we must closely study the context in which our data collection and indicator assessment develops. Social systems cannot be measured without perturbing them and we can do no good with data or evidence if we do not understand and respect both the systems being measured and the effects of implementing any policy decision.

We need to understand the measures we might develop, what forms of evaluation they are useful for and how change can be effected where appropriate. This will require significant work as well as an appreciation of the close coupling of the whole system.
We have a generational opportunity to make our research infrastructure better through effective evaluation and evidence based policy making and architecture development. But we will squander this opportunity if we either take a utopian view of what might technically feasible, or fail to act for a fear of a dystopian future. The way to approach this is through a careful, timely, transparent and thoughtful approach to understanding ourselves and the system we work within.

The commission should act to ensure that current nascent efforts work efficiently towards delivering the technical, cultural, and legal infrastructure that will support an informed debate through a combination of communication, coordination, and policy actions.

Enhanced by Zemanta
Posted on

Michael Nielsen, the credit economy, and open science

No credit cards please.......

Michael Nielsen is a good friend as well as being an inspiration to many of us in the Open Science community. I’ve been privileged to watch and in a small way to contribute to the development of his arguments over the years and I found the distillation of these years of effort into the talk that he recently gave at TEDxWaterloo entirely successful. Here is a widely accesible and entertaining talk that really pins down the arguments, the history, the successes and the failures of recent efforts to open up science practice.

Professional scientific credit is the central issue

I’ve been involved in many discussions around why the potential of opening up research practice hasn’t lead to wider adoption of these approaches. The answer is simple, and as Michael says very clearly in the opening section of the talk, the problem is that innovative approaches to doing science are not going to be adopted while those that use them don’t get conventional scientific credit. I therefore have to admit to being somewhat nonplussed by GrrlScientist’s assessment of the talk that “Dr Nielsen has missed — he certainly has not emphasised — the most obvious reason why the Open Science movement will not work: credit.”

For me, the entire talk is about credit. He frames the discussion of why the Qwiki wasn’t a huge success, compared to the Polymath project, in terms of the production of conventional papers, he discusses the transition from Galileo’s anagrams to the development of the scientific journal in terms of ensuring priority and credit. Finally he explicitly asks the non-scientist members of the audience to do something that even more closely speaks to the issue of credit, to ask their scientist friends and family what they are doing to make their results more widely available. Remember this talk is aimed at a wider audience, the TEDxWaterloo attendees and the larger audience for the video online (nearly 6,000 when I wrote this post). What happens when taxpayers start asking their friends, their family, and their legislative representatives how scientific results are being made available? You’d better believe that this has an affect on the credit economy.

Do we just need the celebrities to back us?

Grrl suggests that the answer to pushing the agenda forward is to enlist Nobelists to drive projects in the same way that Tim Gowers pushed the Polymath project. While I can see the logic and there is certainly value in moral support from successful scientists we already have a lot of this. Sulston, Varmus, Michael and Jon Eisen, and indeed Michael himself just to name a few are already pushing this agenda. But moral support and single projects are not enough. What we need to do is hack the underlying credit economy, provide proper citations for data and software, exploit the obsession with impact factors.

The key to success in my view is a pincer movement. First, showing that more (if not always completely) open approaches can outcompete closed approaches on traditional assessment measures, something demonstrated successfully by Galaxy Zoo, the Alzeimers Disease Neuroimaging Initiative, and the Polymath Projects. Secondly changing assessment policy and culture itself, both explicitly by changing the measures by which researchers are ranked, and implicitly by raising the public expectation that research should be open.

The pendulum is swinging and we’re pushing it just about every which-way we can

I guess what really gets my back up is that Grrl sets off with the statement that “Open Science will never work” but then does on to put her finger on exactly the point where we can push to make it work. Professional and public credit is absolutely at the centre of the challenge. Michael’s talk is part of a concerted, even quite carefully coordinated, campaign to tackle this issue at a wide range of levels. Michael’s tour of his talk, funded by the Open Society Institute seeks to raise awareness. My recent focus on research assessment (and a project also funded by OSI) is tackling the same problem from another angle. It is not entirely a coincidence that I’m writing this in a hotel room in Washington DC and it is not at all accidental that I’m very interested in progress towards widely accepted researcher identifiers. The development of Open Research Computation is a deliberate attempt to build a journal that exploits the nature of journal rankings to make software development more highly valued. 

All of these are part of a push to hack, reconfigure, and re-assess the outputs and outcomes that researchers get credit for and the the outputs and outcomes that are valued by tenure committees and grant panels. And from where I stand we’re making enough progress that Grrl’s argument seems a bit tired and outdated. I’m seeing enough examples of people getting credit and reward for being open and simply doing and enabling better science as a result that I’m confident the pendulum is shifting. Would I advise a young scientist that being open will lead to certain glory? No, it’s far from certain, but you need to distinguish yourself from the crowd one way or another and this is one way to do it. It’s still high risk but show me something in a research career that is low risk and I’ll show something that isn’t worth doing.

What can you do?

If you believe that a move towards more open research practice is a good thing then what can you do to make this happen? Well follow what Michael says, give credit to those who share, explicitly acknowledge the support and ideas you get from others. Ask researchers how they go about ensuring that their research is widely available and above all used. The thing is, in the end changing the credit economy itself isn’t enough, we actually have to change the culture that underlies that economy. This is hard but it is done by embedding the issues and assumptions in the everyday discourse about research. “How useable are your research outputs really?” is the question that gets to the heart of the problem. “How easily can people access, re-use, and improve on your research? And how open are you to getting the benefit of other people’s contribution?” are the questions that I hope will become embedded in the assumptions around how we do research. You can make that happen by asking them.

 

Related articles
Enhanced by Zemanta
Posted on

Best practice in Science and Coding. Holding up a mirror.

The following is the text from which I spoke today at the .Astronomy conference. I think there is some video available on the .Astronomy UStream account and I also have audio which I will put up somewhere soon.

There’s a funny thing about the science and coding communities. Each seems to think that the other has all the answers. Maybe the grass is just greener…For many years as an experimental scientist I looked jealously at both computational scientists and coders in general.  Wouldn’t it be so much less stressfull, I naively thought, to have systems that would do what they were told, to be easily able to re-run experiments and to be able to rely on getting the same answer.  Above all, I thought, imagine the convenience of just being able to take someone else’s work and being able to easily and quickly apply it to my own problems.

There is something of a mythology around code, and perhaps more so around open source, that it can be relied on, that there is a toolkit out there already for every problem. That there is a Ruby Gem, or an R library for every problem, or most memorably that I can sit on a python command line and just demand antigravity by importing it. Sometimes these things are true, but I’m guessing that everyone has experience of it not being true. Of the python library that looks as though it is using dictionaries but is actually using some bizarre custom data type, the badly documented ruby gem, or the perl…well, just the perl really. The mythology doesn’t quite live up to the hype. Or at least not as often as we might like.

But if us experimental scientists have an overoptimistic view of how repeatable and reliable computational tools are then computer scientists have an equally unrealistic view of how experimental science works. Greg Wilson, one of the great innovators in computer science education once said, while criticizing documentation and testing standards of scientific code “An experimental scientist would never get away with not providing their data, not providing their working. Experimental science is expected to be reproducible from the detailed methodology given….”….data provided…detailed methodology…reproducible…this doesn’t really sound like the experimental science literature that I know.

Ted Pedersen in an article with the wonderful title “Empiricism is not a matter of faith” excoriates computational linguistics by holding it up to what he sees as the much higher standards of reproducibility and detailed description of methodology in experimental science. Yet I’ve never been able to reproduce an experiment based only on a paper in my life.

What is interesting about both of these view points is that we are projecting our very real desire to raise standards against a mythology of someone else’s practice. There seems to be a need to view some other community’s practice as the example rather than finding examples within our own. This is odd because it is precisely the best examples, within each community, that inspire the other. There are experimental scientists that give detailed step by step instructions to enable others to repeat their work, who make the details of the protocols available online, and who work within their groups to the highest standards of reproducibility that are possible in the physical world.  Equally there are open source libraries and programmes with documentation that are both succinct and detailed, that just works when you import the library, that is fully tested and comes with everything you need to make sure it will work with your systems. Or that breaks in an informative way, making it clear what you need to do with your own code to get it working.

If we think about what makes science work; effective communication, continual testing and refinement, public criticism of claims and ideas; the things that make up good science, and mean that I had a laptop to write this talk on this morning, that meant the train and taxi I caught actually run, that, more seriously a significant portion of the people in this room did not in fact die in childhood. If we look at these things then we see a very strong correspondence with good practice in software development. High quality and useful documentation is key to good software libraries.  You can be as open source as you like but if no-one can understand your code they’re not going to use it. Controls, positive and negative, statistical and analytical are basically unit tests. Critique of any experimental result comes down to asking whether each aspect of the experiment is behaving the way it should, has each process been tested that a standard input gives the expected output. In a very real sense experiment is an API layer we use to interact with the underlying principles of nature.

So this is a nice analogy, but I think we can take this further, in fact I think that code and experiment are actually linked at a deeper level. Both are an instantiation of process that take inputs and generate outputs. These are (to a first approximation – good enough for this discussion) deterministic in any given instance. But they are meaningless without context. Useless without the meaning that documentation and testing provide.

Let me give you an example. Ana Nelson has written a lovely documentation tool called Dexy. This builds on concepts of literate programming in a beautifully elegant and simple way. Take a look for the details but in essence it enables you to directly incorporate the results of arbitrary running code into your documentation. As you document what your code does you provide examples, parts of the process that are actively running, and testing the code as you go. If you break the method you break your documentation. It is also not an accident that if you are thinking about documentation as you build your code then it helps to create good modular structures that are easy to understand and therefore both easy to use and easy to communicate. They may be a little more work to write but the value you are creating by thinking about the documentation up front means you are motivated to capture this up front. Design by contract and test driven development are tough, Documentation Driven Development can really help drive good process.

Too often when we write a scientific paper it’s the last part of the process. We fabricate a story that makes sense so that we can fit in the bits we want to. Now there’s nothing wrong with this. Humans are narrative processing systems, we need stories to make sense of the world. But its not the whole story. What if, as we collect and capture the events that we ultimately use to tell our story, that we also collect and structure the story of what actually happened? Of the experiments that didn’t work, of the statistical spread of good and bad results. There’s a sarcastic term in synthetic organic chemistry, the “American Yield” in which we imagine that 20 PhD students have been tasked with making a compound and the one who manages to get the highest overall yield gets to be first author. This isn’t actually a particularly useful number. Much more useful to the chemist who wants to use this prep is the spread of values, information that is generally thrown away. The difference between actually incorporating the running of the code into the documentation, and just showing one log file, cut and pasted, from when it worked well. You lose the information about when it doesn’t work.

Other tools from coding can also provide inspiration. Tools like Hudson for continuous integration. Everytime the code base is changed everything gets re-built, dependencies are tested, unit tests run, and a record of what gets broken. If you want to do X you do want to use this version of that library.  This isn’t a problem. In any large codebase things are going to get broken as changes are made, you change something, see what is broken, then go back and gradually fix those things until you’re ready to create commit to the main branch (at which point someone else has broken something…)

Science is continuous integration. This is what we do, we make changes , we check what they break, see if the dependencies still hold and if necessary go back and fix them. This is after all where the interesting science is. Or it would be if we did it properly. David Shotton and others have spoken about the question of “citation creep” or “hedging erosion” [see for example this presentation by Anita de Waard]. This is where something initially reported in one paper as a possibility, or even just a speculation gets converted into fact by a process of citation. What starts as “…it seems possible that…” can get turned into “…as we know that X causes Y (Bloggs et al, 2009)…” within 18 months or a couple of citations. Scientists are actually not very good at checking their dependencies. And they have a tendency of coming back to bite us in exactly the same way as a quick patch that wasn’t properly tested can.

Just imagine if we could do this. If everytime a new paper was added to the literature we could run a test against the rest. Check all the dependencies…if this isn’t true then all of these other papers in doubt as well…indeed if we could unit test papers would it be worth peer reviewing them? There is good evidence that pair coding works, and little evidence that traditional peer review does. What can we learn from this to make the QA processes in science and software development better?

I could multiply examples. What would an agile lab look like? What would be needed to make it work? What can successful library development communities tell us about sharing samples, and what can the best data repositories tell us about building the sites for sharing code? How can we apply the lessons of StackOverflow to a new generation of textbooks and how can we best package up descriptions of experimental protocols in a way that provides the same functionality as sharing an Amazon Machine Image.

Best practice in coding mirrors best practice in science. Documentation, testing, integration are at the core. Best practice is also a long way ahead of common practice in both science and coding. Both, perhaps are driven increasingly by a celebrity culture that is more dependent on what your outputs look like (and where they get published) than whether anyone uses them. Testing and documentation are hardly glamorous activities.

So what can we do about it? Improving practice is an arduous task. Many people are doing good work here with training programmes, tools, standards development and calls to action online and in the scientific literature. Too many people and organizations for me to call out and none of them getting the credit they deserve.

One of the things I have been involved with is to try and provide a venue, a prestigious venue, where people can present code that has been developed to the highest standards. Open Research Computation, a new Open Access journal from BioMedCentral, will publish papers that describe software for research. Our selection criteria don’t depend on how important the research problem is, but on the availability, documentation, and testing of the code. We expect the examples given in these papers to be reproducible, by which we mean that the software, the source code, the data, and the methodology are provided and described well enough that it is possible to reproduce those examples.  By applying high standards, and by working with authors to help them reach those standards we aim to provide a venue which is both useful and prestigious. Think about it, a journal that contains papers describing the most useful and useable tools and libraries is going to get a few citations and (whisper it) ought to get a pretty good impact factor. I don’t care about impact factors but I know the reality on the ground is that that those of you looking for jobs or trying to keep them do need to worry about them.

In the end, the problem with a journal, or with code, or with science is that we want everyone else to provide the best documentation, the best tested code and procedures, but its hard to justify doing it ourselves. I mean I just need something that works; yesterday. I don’t have time to write the tests in advance, think about the architecture, re-read all of that literature to check what it really said. Tools that make this easier will help, tools like Dexy and Hudson, or lab notebooks that capture what we’re doing and what we are creating, rather than what we said we would do, or what we imagine we did in retrospect.

But it’s motivation that is key here. How do you motivate people do the work up front? You can tell them that they have to of course but really these things work best when people want to make the effort. The rewards for making your work re-usable can be enormous but they are usually further down the road than the moment where you make the choice not to bother. And those rewards are less important to most people than getting to the Nature paper, or getting mentioned in Tim O’Reilly’s tweet stream.

It has to be clear that making things re-usable is the highest contribution that you can make, and for it to be rewarded accordingly.  I don’t even really care what forms of re-use are counted, re-use in research, re-use in education, in commerce, in industry, in policy development. ORC is deliberately – very deliberately – intended to hack the impact factor system by featuring highly re-usable tools that will gain lots of citations. We need more of these hacks.

I think this shift is occurring. It’s not widely know just how close UK science funding went to being slashed in the comprehensive spending review. That it wasn’t was due to a highly coherent and well organized campaign that convinced ministers and treasury that the re-use of UK research outputs generated enormous value, both economic, social and educational for the country and indeed globally. That the Sloan Digital Sky Survey was available in a form that could be re-used to support the development of something like Galaxy Zoo played a part in this. The headlong rush of governments worldwide to release their data is a massive effort to realize the potential value of the re-use of that data.

This change in focus is coming. It will no longer be enough in science to just publish. As David Willetts said in [answer to a question] in his first policy speech, “I’m very much in favour of peer review, but I worry when the only form of review is for journals”.  Government wants evidence of wider use. They call it impact, but its basically re-use. The policy changes are coming, the data sharing policies, the public engagement policies, the impact assessments. Just showing outputs will no be enough, showing that you’ve configured those outputs so that the potential for re-use is maximized will be an assumption of receiving funding.

William Gibson said the future is already here, its just unevenly distributed. They Might Be Giants asked, not quite in response, “but where’s my jetpack?”  The jetpacks, the tools, are around us and being developed if you know where to look. Best practice is unevenly distributed both in science and in software development but it’s out there if you want to go looking. The motivation to adopt it? The world around us is changing. The expectations of the people who fund us are changing. Best practice in code and in science have an awful lot in common. If you can master one you will have to tools to help you with the other. And if you have both then you’ll be well positioned to ride the wave of change as it sweeps by.


Enhanced by Zemanta
Posted on

A return to “bursty work”

Parris Island, S.C., barrage balloon (LOC)
Image by The Library of Congress via Flickr

What seems like an age ago a group of us discussed a different way of doing scientific research. One partly inspired by the modular building blocks approach of some of the best open source software projects but also by a view that there were tremendous efficiency gains to be found in enabling specialisation of researchers, groups, even institutes, while encouraging a shared technical and social infrastructure that would help people identify the right partners for the very specific tasks that they needed doing today.

“Bursty work” is a term first used by Chris Messina but introduced to the online community of scientists by Deepak Singh. At the time it seemed obvious that with enough human and financial capital that a loose network of specialist groups could do much better science, and arguably much more effective exploitation of that science, than isolated groups perpetually re-inventing the wheel.

The problem of course is that science funding is not configured that way, a problem that is that bane of any core-facility manager’s existence. Maintaining a permanent expert staff via a hand to mouth existence of short term grants is tough. Some succeed but more probably fail, and there is very little glory in this approach. Once again it is prestige that gets promotion, not effective and efficient use of resources.

But the world is changing, a few weeks ago I got a query from a commercial partner interested in whether I could solve a specific problem. This is a small “virtual company” that aims to target the small scale, but potentially high value, innovations that larger players don’t have the flexibility to handle.  Everything is outsourced, samples prepared and passed from contractor to contractor. Turns out I think we can solve their problem and it will be exciting to see this work applied. What is even more gratifying is that the company came across this work in an Open Access journal which made it easier both to assess how useful it was and whether to get in touch. In the words of my contact:

“The fact that your work was in an open access journal certainly made it easier for me to access. I guess the same google search would have found it in a different journal, but it might have required a subscription for access. In that case I would have used the free info available (corresponding authors, university addresses etc) to try and get in touch based on the abstract.”

The same problems of course remain. How do I reasonably cost this work? What is the value of being involved vs just being a contractor. And of course, where will I find the time, or the pair of hands, to get the work done. People with the right expertise don’t grow on trees, and it’s virtually impossible to get people on short contracts at the moment. Again, in the words of our collaborator:

“Bursty work” sounds a little like how [our company] is trying to operate. One problem is moving from an investment environment where investors invest in companies to one where they invest in projects. Has any work been done to identify investors who like the idea of bursty work?

Nonetheless, its exciting to me that some elements of what was beginning to seem like a pipe dream are coming to pass. It takes time for the world to catch up, but where there is a demand for innovation, and an effective market, the opportunities are there for the people who can make them work.

[It won’t escape anyone’s notice that I’ve given no details of either the project or the company. We are doing this under an NDA and as this is someone else’s project I’m not going to be difficult about it. We make progress one small step at a time]

Enhanced by Zemanta
Posted on

Open Source, Open Research and Open Review

Logo Open Source Initiative
Image via Wikipedia

The submissions for Open Research Computation (which I blogged about a month or so back) are starting to come in and we hope to be moving towards getting those initial papers out soon. One of the things we want the journal to do is bring more of the transparency and open critique that characterises the best Open Source Software development processes into the scholarly peer review process. The journal will have an open review process in which reviews and the versions of the manuscript they refer to will be available.

One paper’s authors however have taken matters into their own hands and thrown the doors completely open. With agreement from the editorial board Michael Barton and Hazel Barton have  asked the community on the BioStar site, a bioinformatics focussed member of the StackExchange family of Q&A websites, how the paper and software could be improved. They have published a preprint of the paper and the source code was obviously already available on Github. You can see more at Michael’s blog post. We will run a conventional peer review process in parallel and the final decision on whether the paper is ready to publish will rest with the ORC editors but we will take into account the comments on BioStar and of course the authors will be free to use those comments to improve on their software and documentation.

 

This kind of approach goes a long way towards dealing with the criticisms I often level at conventional peer review processes. By making the process open there is the opportunity for any interested party to offer constructive critique and help to improve the code and the paper. By not restricting commentary to a small number of people we stand a better chance of getting all the appropriate points of view represented. And by (hopefully, we may have some niggling licence issues with copying content from BioStar’s CC-BY-SA to BioMedCentral’s CC-BY) presenting all of that commentary and critique along with the authors responses we can offer a clear view of how effective the review process was and what the final decisions were based on. I’ve talked about what we can do to improve peer review. Michael and Hazel have taken action to make it happen. You can be a part of it.

Enhanced by Zemanta
Posted on

Reforming Peer Review. What are the practical steps?

Peer Review Monster

The text of this was written before I saw Richard Poynder’s recent piece on PLoS ONE and the responses to that. Nothing in those really changes the views I express here but this text is not a direct response to those pieces.

So my previous post on peer review hit a nerve. Actually all of my posts on peer review hit a nerve and create massive traffic spikes and I’m still really unsure why. The strength of feeling around peer review seems out of all proportion to both its importance and indeed the extent to which people understand how it works in practice across different disciplines. Nonetheless it is an important and serious issue and one that deserves serious consideration, both blue skies thinking and applied as it were. And it is the latter I will try to do here.

Let me start with a statement. Peer review at its core is what makes science work. There are essentially two logical philosophy approaches that can be used to explain why the laptop I’m using works, why I didn’t die as a child of infection, and how we are capable of communication across the globe. One of these is the testing of our working models of the universe against the universe itself. If your theory of engines produces an engine that doesn’t work then it is probable there is something wrong with your theory.

The second is that by exposing our models and ideas to the harshest possible criticism of our peers that we can stress them to see what holds up to the best logical analysis available. The motto of the Royal Society “Nullius in verba” is generally loosely translated as “take no-one’s word for it”. The central idea of the Invisible College, the group that became the Royal Society was that they would present their experiments and their explanations to each other, relying on the criticism of their peers to avoid the risk of fooling themselves. This combined both philosophical approaches, seeing the apparatus for yourself, testing the machinery against the world, and then testing the possible explanations for its behaviour against the evidence and theory available. The community was small but this was in a real sense post-publication peer review; testing and critique was done in the presence of the whole community.

The systems employed by a few tens of wealthy men do not scale to todays global scientific enterprise and the community has developed different systems to manage this. I won’t re-hash my objections to those systems except to note what I hope should be be three fairly uncontroversial issues. Firstly that pre-publication peer review as the only formal process of review runs a severe risk of not finding the correct diversity and expertise of reviewers to identify technical issues. The degree of that risk is more contentious but I don’t see any need to multiply recent examples that illustrate that it is real. Second, because we have no system of formalising or tracking post-publication peer review there is no means either to encourage high quality review after publication, nor to track the current status or context of published work beyond the binary possibility of retraction. Third, that peer review has a significant financial cost (again the actual level is somewhat contentious but significant seems fair) and we should address whether this money is being used as efficiently as it could be.

It is entirely possible to imagine utopian schemes in which these problems, and all the other problems I have raised are solved. I have been guilty of proposing a few myself in my time. These will generally involve taking a successful system from some other community or process and imagining that it can be dropped wholesale on the research community. These approaches don’t work and I don’t propose to explore them here in detail, except as ways to provoke and raise ideas.

Arguments and fears

The prospect of radical change to our current process of peer review provokes very strong and largely negative responses. Most of these are based on fears of what would happen if the protection that our current pre-publication peer review system offers us is ripped away. My personal view is that these protections are largely illusory but a) I could well be wrong and b) that doesn’t mean we shouldn’t treat these fears seriously. They are, after all, a barrier to effective change, and if we can neutralize the fears with evidence then we are also making a case for change, and in most cases that evidence will also offer us guidance on the best specific routes for change.

These fears broadly fall into two classes. The first is the classic information overload problem. Researchers already have too much to track and read. How can they be expected to deal with the apparent flood of additional information? One answer to this is to ask how much more information would be released. This is difficult to answer. Probably somewhere between 50 and 95% of all papers that are submitted somewhere do eventually get published [1, 2 (pdf), 3, 4] suggesting that the total volume would not increase radically. However it is certainly arguable that reducing barriers would increase this. Different barriers, such as cost could be introduced but since my position is that we need to reduce these barriers to minimise the opportunity cost inherent in not making research outputs public I wouldn’t argue for that. However we could imagine a world in which small pieces of research output get published for near zero cost but turning those pieces into an argument, something that would look a lot like the current formally published paper, would cost more either in terms of commitment or financial costs.

An alternative argument, and one I have made in the past is that our discovery tools are already broken and part of the reason for that is there is not enough of an information substrate to build better ones. This argument holds that by publishing more we can make discovery tools better and actually solve the overload problem by bringing the right information to each user as and when they need it. But while I make this argument and believe it, it is conceptually very difficult for most researchers to grasp. I hesitate to suggest that this has something to do with the best data scientists, the people who could solve this problem, eschewing science for the more interesting and financially rewarding worlds of Amazon, Google, and Facebook.

The second broad class of argument against change is that the currently validated and recognized literature will be flooded with rubbish. In particular a common, and strongly held, view is that the wider community will no longer be able to rely on the quality mark that the peer reviewed literature provides in making important health, environmental, and policy decisions. Putting aside the question of whether in fact peer review does achieve an increase in accuracy or reliability there is a serious issue here to be dealt with respect to how the ongoing results of scientific research are presented to the public.

There are real and serious risks in making public the results of research into medicine, public health, and the environment. Equally treating the wider community as idiots is also dangerous. The responsible media and other interested members of the community, who can’t be always be expected to delve into, or be equipped to critique, all of the detail of any specific claim, need some clear mark or statement of the level of confidence the research community has in a finding or claim. Regardless of what we do the irresponsible media will just make stuff up anyway so its not clear to me that there is much that can be done there but responsible reporters on science benefit from being able to reference and rely on the quality mark that peer review brings. It gives them an (at least from their perspectice) objective criterion on which to base the value of a story.

It isn’t of course just the great unwashed that appreciate a quality control process. For any researcher moving out of their central area of expertise to look at a new area there is a bewildering quantity of contradictory statements to parse. How much worse would this be without the validation of peer review? How would the researcher know who to trust?

It is my belief that the emotional response to criticism of traditional pre-publication peer review is tightly connected to this question of quality, and its relation to the mainstream media. Peer review is what makes us difference. It is why we have a special relationship with reporters, and by proxy the wider community, who can trust us because of their reliance on the rigour of our quality marks. Attacks on peer review are perceived as an attack at the centre of what makes the research community special.

The problem of course is that the trust has all but evaporated. Scandals, brought on in part by a reliance on the meaning and value of peer review, have taken away a large proportion of the credibility that was there. Nonetheless, there remains a clear need for systems that provide some measure of the reliability of scientific findings. At one level, this is simple. We just wait ten years or so to see how it pans out. However, there is a real tension between the needs of reporters to get there first and be timely and the impossibility of providing absolute certainty around research findings.

Equally applying findings in the real world will often mean moving before things are settled. Delays in applying the results of medical research can kill people just as much as rushing in ahead of the evidence can. There is always a choice to be made as to when the evidence is strong enough and the downside risks low enough for research results to be applied. These are not easy decisions and my own view is that we do the wider community and ourselves a disservice by pretending that a single binary criterion with a single, largely hidden, process is good enough to universally make that decision.

Confidence is always a moving target and will continue to be. That is the nature of science. However an effective science communication system will provide some guide to the current level of confidence in specific claims.  In the longer term there is a need to re-negotiate the understanding around confidence between the responsible media and the research community. In the shorter term we need to be clearer in communicating levels of confidence and risk, something which is in any case a broader issue for the whole community.

Charting a way forward

So in practical terms what are the routes forward? There is a rhetorical technique of persuasion that uses a three-part structure in arguing for change. Essentially this is to lay out the argument in three parts, firstly that nothing (important) will change, second that there are opportunities for improvement that we can take, and third that everything will change. This approach is supposed to appeal to three types of person, those who are worried about the risks of change, those in the middle who can see some value in change but are not excited by it, and finally those who are excited by the possibilities of radical change. However, beyond being a device this structure suits the issues here, there are significant risks in change, there are widely accepted problems with the current system, and there is the possibility for small scale structural changes to allow an evolution to a situation where radical change can occur if momentum builds behind it.

Nothing need change

At the core of concerns around changing peer review is the issue of validation. “Peer reviewed” is a strong brand that has good currency. It stands for a process that is widely respected and, at least broadly speaking, held to be understood by government and the media. In an environment where mis-reporting of medical or environmental research can easily lead to lost lives this element of validation and certification is critical. There is no need in any of the systems I will propose for this function to go away. Indeed we aim to strengthen it. Nor is there a need to abandon the situation where specific publication venues are marked as having been peer reviewed and only contain material that has been through a defined review process. They will continue to stand or fall on their quality and the value for money that they offer.

The key to managing the changes imposed on science communication by the rise of the web, while maintaining the trust and value of traditional review systems, is to strengthen and clarify the certification and validation provided by peer review and to retain a set of specific publication venues that guarantee those standards and procedures of review. These venues, speaking as they will to both domain specific and more general scientific audience, as well as to the wider community will focus on stories and ideas. They will, in fact look very like our current journals and have contents that look the same as our current papers.

These journals will have a defined and transparent review process with objective standards and reasonable timeframes. This will necessarily involve obtaining opinions from a relatively small number of people and a final decision made by a central editor who might be a practising researcher or a professional editor. In short all the value that is created by the current system, should and can be retained.

Room for improvement

If we are to strengthen the validation process of peer review we need to address a number of issues. The first of these is transparency. A core problem with peer review is that it is in many cases not clear what process was followed. How many external referees were used? Did they have substantive criticisms, and did disagreements remain? Did the editors over-rule the referees or follow their recommendation? Is this section of the journal peer reviewed at all?

Transparency is key. Along with providing confidence to readers such transparency could support quantitative quality control and would provide the data that would help us to identify where peer review succeeds and where it is failing. Data that we desperately need so we can move beyond assertions and anecdote that characterise the current debate.

A number of publishers have experimented with open peer review processes. While these remain largely experiments a number of journals, particularly those in medical fields, will publish all the revisions of a paper along with the review reports at each stage. For those who wish to know whether their concerns were covered in the peer review process this is a great help.

Transparency can also support an effective post publication review process. Post-publication review has occurred at ArXiv for many years where a pre-print will often be the subject of informal discussion and comment before it is submitted for formal review at a peer reviewed journal. However it could be argued that the lack of transparency that results from this review happening informally makes it harder to identify the quality papers in the ArXiv.

A more formal process of publication, then validation and certification has been adopted by Atmospheric Chemistry and Physics and other Copernicus publications. Here the submitted manuscript is published in ACP Discussions (after a “sanity check” review), and then subject to peer review, both traditional by selected referees and in an open forum. If the paper is accepted it is published, along with links to the original submission and commentary in the main journal. The validation provided by review is retained while providing enhanced transparency.

In addition this approach addresses the concerns of delays in publication, whether due to malicious referees or simply the mechanics of the process, and the opportunity costs for further research that they incur. By publishing first, in a clearly non-certificated form, the material is available for those who might find them of value but in a form that is clearly marked as non-validated, use at own risk. This is made clear by retaining the traditional journal, but adding to it at the front end. This kind of approach can even support the traditional system of tiered journals with the papers and reviews trickling down from the top forming a complete record of which journal rejected which papers in which form.

The objection to this style of approach is that this approach doesn’t support the validation needs of biomedical and chemical scientists to be “first to publish in peer reviewed journal”. There is a significant cultural distinction between the physical sciences that use ArXiv and the biosciences in particular best illustrated by a story that I think I first heard from Michael Nielsen.

A biologist is talking to a physicist and says, “I don’t understand how you can put your work in the ArXiv as a preprint. What if someone comes along and takes your results and then publishes them before you get your work to a peer reviewed journal?”

The physicist thinks a little about this before responding, “I don’t understand how you can not put your work in the ArXiv as a preprint. What if someone comes along and takes your result and then publishes them before you get your work to a peer reviewed journal?”

There is a cultural gulf here that can not be easily jumped. However this is happening by stealth anyway with a variety of journals that have subtle differences in the peer review process that are not always clearly and explicitly surfaced. It is interesting in this context that PLoS ONE and now its clones are rapidly moving to dominate the publishing landscape despite a storm of criticism around the (often misunderstood) peer review model. Even in the top tier it can be unclear whether particular classes of article are peer reviewed (see for example these comments [1, 2, 3] on this blog post from Neil Saunders). The two orthogonal concepts of “peer reviewed” and “formally published” appear to be drifting apart from what was an easy (if always somewhat lazy) assumption that they are equivalent. Priority will continue to be established by publication. The question of what kind of publication will “count” is likely to continue to shift but how fast and in what disciplines remains a big question.

This shift can already be seen in the application of DOIs to an increasingly diverse set of research outputs. The apparent desire to apply DOIs stems from the idea that a published object is “real” if it has a DOI. This sense of solidness seems to arise from the confidence that having a DOI makes an object citeable. The same confidence does not apparently apply to URLs or other identifiers, even when those URLs come from stable entities such as Institutional Repositories or recognised Data Services.

This largely unremarked shift may potentially lead to a situation where a significant proportion of the reference list of a peer reviewed paper may include non-peer reviewed work. Again the issue of transparency arises, how should this be marked? But equally there will be some elements that are not worthy of peer review, or perhaps only merit automated validation such as some types of dataset. Is every PDB or Genbank entry “peer reviewed”? Not in the commonly meant sense, but is it validated? Yes. Is an audit trail required? Yes.

A system of transparent publication mechanisms for the wide range of research objects we generate today, along with clear but orthogonal marking of whether and how each of those objects have been reviewed provides real opportunities to both encourage rapid publication, enable transparent and fair review, and to provide a framework for communicating effectively the level of confidence the wider community has in a particular claim.

These new publication mechanisms and the increasing diversity of published research outputs are occurring anyway. All I am really arguing for is a recognition and acceptance that this is happening at different rates and in different fields. The evidence from ArXiv, ACP, and to a lesser extent conferences and online notebooks is that the sky will not fall in as long as there is clarity as to how and whether review has been carried out. The key therefore is much more transparent systems for marking what is reviewed, and what is not, and how review has been carried out.

Radical Changes

A system that accepts that there is more than one version of a particularly communication opens the world up to radical change. Re-publication following (further) review becomes possible as do updates and much more sophisticated retractions. Papers where particular parts are questioned become possible as review becomes more flexible and disagreement, and the process of reaching agreement no longer need to be binary issues.

Reviewing different aspects of a communication leads in turn to the feasibility of publishing different parts for review at different times. Re-aggregating different sets of evidence and analysis to provide a dissenting view becomes feasible. The possibilities of publishing and validating portions of a whole story offer great opportunities for increased efficiency and for much more public engagement and information with the current version of the story. Much is made of poor media reporting of “X cures/causes cancer” style stories but how credible would these be if the communication in question was updated to make it clear that the media coverage was overblown or just plain wrong? Maybe this wouldn’t make a huge difference but at some level what more can we be asked to do?

Above all the blurring of the lines between what is published and what is just available and an increasing need to be transparent about what has been reviewed and how will create a market for these services. That market is ultimately what will help to both drive down the costs of scholarly communication and to identify where and how review actually does add value. Whole classes of publication will cease to be reviewed at all as the (lack of) value of this becomes clear. Equally high quality review can be re-focussed where it is needed, including the retrospective or even continuous review of important published material. Smaller ecosystems will naturally grow up where networks of researchers have an understanding of how much they trust each others results.

The cultural chasm between the pre-review publication culture by users of the ArXiv and the chemical and biomedical sciences will not be closed tomorrow but as the pressures of government demands for rapid exploitation and the possibilities of losing opportunities by failing to communicate rise there will be a gradual move towards more rapid publication mechanisms. In parallel as the pressures to quantitatively demonstrate efficient and effective use of government funding rise opportunities will arise for services to create low barrier publication mechanisms. If the case can be made for measurement of re-use then this pressure has the potential to lead to effective communication as well as just dumping of the research record.

Conclusion

Above all other things the major trend I see is the breakage of the direct link between publication and peer review. Formal publication in the print based world required a filtering mechanism to be financially viable. The web removes that requirement, but not the requirement of quality marking and control. The ArXiv, PLoS ONE and other experiments with simplifying peer review processes, Institutional Repositories, and other data repositories, the expanding use of DOIs, and the explosion of freely available research content and commentary on the web are all signs of a move towards lower barriers in publishing a much more diverse range of research outputs.

None of this removes the need for quality assurance. Indeed it is precisely this lowering of barriers that has brought such a strong focus on the weaknesses of our current review processes. We need to take the best of both the branding and the practice of these processes and adapt them or we will lose both the confidence of our own community and the wider public. Close examination of the strengths and weaknesses and serious evidence gathering is required to adapt and evolve the current systems for the future. Transparency, even radical transparency of review processes may well be something that is no longer a choice for us to make. But if we move in this direction now, seriously and with real intent, then we may as a research community be able to retain control.

The status quo is not an option unless we choose to abandon the web entirely as a place for research communication and leave it for the fringe elements and the loons. This to me is a deeply retrograde step. Rather, we should take our standards and our discourse, and the best quality control we can bring to bear out into the wider world. Science benefits from a diversity of views and backgrounds. That is the whole point of peer review. The members of the Invisible College knew that they might mislead themselves and took the then radical approach of seeking out dissenting and critical views. We need to acknowledge our weaknesses, celebrate our strengths and above all state clearly where we are unsure. It might be bad politics, but it’s good science.

Enhanced by Zemanta