Posts

Posted on

IP Contributions to Scientific Papers by Publishers: An open letter to Rep Maloney and Issa

Dear Representatives Maloney and Issa,

I am writing to commend your strong commitment to the recognition of intellectual property contributions to research communication. As we move to a modern knowledge economy, supported by the technical capacity of the internet, it is crucial that we have clarity on the ownership of intellectual property arising from the federal investment in research. For the knowledge economy to work effectively it is crucial that all players receive fair recompense for the contribution of intellectual property that they make and the services that they provide.

As a researcher I like to base my work on solid data, so I thought it might interest you to have some quantitation of the level of contribution of IP that publishers make to the substance of scientific papers. In this, I have focussed on the final submitted version of papers after peer review as this is the version around which the discussion of mandates for deposition in repositories revolve. This also has the advantage of separating the typesetting and copyright in layout, clearly the property of the publishers from the intellectual substance of the research.

Contribution of IP to the final (post peer review) submitted versions of papers

Methodology: I examined the final submitted version (i.e. the version accepted for publication) of the ten most recent research papers on which I was an author along with the referee and editorial comments received from the publisher. For each paper I examined the text of the final submitted version and the diagrams and figures.  As the only IP of significance in this case is copyright the specific contributions that were searched for were text or elements of figures contributed by the publisher that satisfied the requirements for obtaining copyright. Figures that were re-used from other publications (where the copyright had been transferred to the other publisher and permission been obtained to republish) were not included as these were considered “old IP” that did not relate to new IP embodied in the specific paper under consideration. The text and figures were searched for specific creative contributions from the publisher and these were quantified for each paper.

Results: The contribution of IP by publishers to the final submitted versions of these ten papers, after peer review had been completed, was zero. Zip. Nada. Zilch. Not one single word, line, or graphical element was contributed by the publisher or the editor acting as their agent. A small number of single words, or forms of expression, were found that were contributed by external peer reviewers. However as these peer reviewers do not sign over copyright to the publisher and are not paid this contribution cannot be considered work for hire and any copyright resides with the original reviewers.

Limitations: This is a small and arguably biased study based on the publications I have to hand. I recommend that other researchers examine their own oeuvre and publish similar analyses so that effects of discipline, age, and venue of publication can be examined. Following such analysis I ask that researchers provide the data via twitter using the hashtag #publisheripcontrib where I will aggregate it and republish.

Data availability: I regret that the original submissions can not be provided as the copyright in these articles was transferred after acceptance for publication to the publishers. I can not provide the editorial reports as these contain material from the publishers for which I do not have re-distribution rights.

The IP argument is sterile and unproductive. We need to discuss services.

The analysis above at its core shows how unhelpful framing this argument around IP is. The fact that publishers do not contribute IP is really not relevant. Publishers do contribute services, the provision of infrastructure, the management of the peer review process, dissemination and indexing, that are crucial for the current system of research dissemination via peer reviewed papers. Without these services papers would not be published and it is therefore clear that these services have to be paid for. What we should be discussing is how best to pay for those services, how to create a sustainable market place in which they can be offered, and what level of service the federal government expects in exchange for the services it is buying.

There is a problem with this. We currently pay for these services in a convoluted fashion which is the result of historical developments. Rather than pay up front for publication services, we currently give away the intellectual property in our papers in exchange for publication. The U.S. federal and state governments then pay for these publication services indirectly by funding libraries to hire access back to our own work. This model made sense when the papers were physically on paper; distribution, aggregation, and printing were major components of the cost. In that world a demand side business model worked well and was appropriate.

In the current world the costs of dissemination and provision of access are as near to zero as makes no difference. The major costs are in the peer review process and preparing the paper in a version that can be made accessible online. That is, we have moved from a world where the incremental cost of dissemination of each copy was dominant, to a world where the first copy costs are dominant and the incremental costs of dissemination after those first copy costs are negligible. Thus we must be clear that we are paying for the important costs of the services required to generate that first web accessible copy, and not that we are supporting unnecessary incremental costs. A functioning market requires, as discussed above, that we have clarity on what is being paid for.

In a service based model the whole issue of IP simply goes away. It is clear that the service we would wish to pay for is one in which we generate a research communication product which provides appropriate levels of quality assurance and is as widely accessible and available for any form of use as possible. This ensures that the outputs of the most recent research are available to other researchers, to members of the public, to patients, to doctors, to entrepreneurs and technical innovators, and not least to elected representatives to support informed policy making and legislation. In a service based world there is no logic in artificially reducing access because we pay for the service of publication and the full first copy costs are covered by the purchase of that service.

Thus when we abandon the limited and sterile argument about intellectual property and move to a discussion around service provision we can move from an argument where no-one can win to a framework in which all players are suitably recompensed for their efforts and contributions, whether or not those contributions generate IP in the legal sense, and at the same time we can optimise the potential for the public investment in research to be fully exploited.

HR3699 prohibits federal agencies from supporting publishers to move to a transparent service based model

The most effective means of moving to a service based business model would be for U.S. federal agencies as the major funders of global research to work with publishers to assure them that money will be available for the support of publication services for federally funded researchers. This will require some money to be put aside. The UK’s Wellcome Trust estimates that they expect to spend approximately 1.5% of total research funding on publication services. This is a significant sum, but not an overly large proportion of the whole. It should also be remembered that governments, federal and state, are already paying these costs indirectly through overheads charges and direct support to research institutions via educational and regional grants. While there will be additional centralised expenditure over the transitional period in the longer term this is at worst a zero-sum game. Publishers are currently viable, indeed highly profitable. In the first instance service prices can be set so that the same total sum of money flows to them.

The challenge is the transitional period. The best way to manage this would be for federal agencies to be able to guarantee to publishers that their funded researchers would be moving to the new system over a defined time frame. The most straight forward way to do this would be for the agencies to have a published program over a number of years through which the publication of research outputs via the purchase of appropriate services would be made mandatory. This could also provide confidence to the publishers by defining the service level agreements that the federal agencies would require, and guarantee a predictable income stream over the course of the transition.

This would require agencies working with publishers and their research communities to define the timeframes, guarantees, and service level agreements that would be put in place. It would require mandates from the federal agencies as the main guarantor of that process. The Research Works Acts prohibits any such process. In doing so it actively prevents publishers from moving towards business models that are appropriate for today’s world. It will stifle innovation and new entrants to the market by creating uncertainty and continuing the current obfuscation of first copy costs with dissemination costs. In doing so it will damage the very publishers that support it by legislatively sustaining an out of date business model that is no longer fit for purpose.

Like General Motors, or perhaps more analogously, Lehman Brothers, the incumbent publishers are trapped in a business model that can not be sustained in the long term. The problem for publishers is that their business model is predicated on charging for the dissemination and access costs that are disappearing and not explicitly charging for the costs that really matter. Hiding the cost of one thing in a charge for another is never a good long term business strategy. HR3699 will simply prop them up for a little longer, ultimately leading to a bigger crash when it comes. The alternative is a managed transition to a better set of business models which can simultaneously provide a better return on investment for the taxpayer.

We recognise the importance of the services that scholarly publishers provide. We want to pay publishers for the services they provide because we want those services to continue to be available and to improve over time. Help us to help them make that change. Drop the Research Works Act.

Yours sincerely

Cameron Neylon

Enhanced by Zemanta
Posted on

Update on publishers and SOPA: Time for scholarly publishers to disavow the AAP

Canute and his courtiers
Image via Wikipedia

In my last post on scholarly publishers that support the US Congress SOPA bill I ended up making a series of edits. It was pointed out to me that the Macmillan listed as a supporter is not the Macmillan that is the parent group of Nature Publishing Group but a separate U.S. subsidiary of the same ultimate holding company, Holtzbrinck. As I dug further it became clear that while only a small number of scholarly publishers were explicitly and publicly supporting SOPA, many of them are members of the Association of American Publishers, which is listed publicly as a supporter.

This is a little different to directly supporting the act. The AAP is a membership organisation that represents its members (including Nature Publishing Group, Oxford University Press, Wiley Blackwell and a number of other familiar names, see the full list at the bottom) to – amongst others – the U.S. government. Not all of its positions would necessarily be held by all its members. However, neither have any of those members come out and publicly stated that they disagree with the AAP position. In another domain Kaspersky software quit the Business Software Alliance over the BSA’s support of SOPA, even after the BSA withdrew its support.

I was willing to give AAP members some benefit of the doubt, hoping that some of them might come out publicly against SOPA. But if that was the hope then the AAP have just stepped over the line. In a spectacularly disingenuous press release the AAP claims significant credit for a new act just submitted to the U.S. Congress. This, in a repeat of some previous efforts, would block any efforts on the part of U.S. federal agencies to enact open access policies, even to the extent of blocking them from continuing to run the spectacularly successful PubMedCentral. That this comes days before the deadline for a request for information on the development of appropriate and balanced policies that would support access to the published results of U.S. taxpayer-funded research is a calculated political act, an abrogation of any principled stance, and clear signal of a lack of any interest in a productive discussion on how to move scholarly communications forward into a networked future.

I was willing to give AAP members some space. Not any more. The time has come to decide whether you want to be part of the future of research communication or whether you want to legislate to try and stop that future happening. You can be part of that future or you can be washed into the past. You can look forward or you can be part of a political movement working to rip off the taxpayers and charitable donors of the world. Remember that the profits alone of Elsevier and Springer (though I should be cutting Springer a little slack as they’re not on the AAP list – the one on the list is a different Springer) could fund the publication of every paper in the world in PLoS ONE. Remember that the cost of putting a SAGE article on reserve for a decent sized class or of putting a Taylor and Francis monograph on reserve for a more modest sized one at one university is more than it would cost to publish them in most BioMedCentral journals and make them available to all.

Ultimately this legislation is irrelevant – the artificial level of current costs of publication and the myriad of additional charges that publishers make for this, that, and the other (Colour charges? Seriously?) will ultimately be destroyed. The current inefficiencies and inflated markups cannot be sustained. The best legislation can do is protect them for a little longer, at the cost of damaging the competitiveness of the U.S. as a major player in global research. With PLoS ONE rapidly becoming a significant proportion of the world’s literature on its own and Nature and Science soon to be facing serious competition at the top end from an OA journal backed by three of the most prestigious funders in the world, we are moving rapidly towards a world where publishing in a subscription journal will be foolhardy at best and suicidal for researchers in many fields. This act is ultimately a pathetic rearguard action and a sign of abject failure.

But for me it is also a sign that the rhetoric of being supportive of a gradual managed change to our existing systems, a plausible argument for such organisations to make, is dead for those signed up to the AAP. Publishers have a choice – lobby and legislate to preserve the inefficient, costly, and largely ineffective status quo – or play a positive part in developing the future.

I don’t expect much; to be honest I expect deafening silence as most publishers continue to hope that most researchers will be too buried in their work to notice what is going on around them. But I will continue to hope that some members of that list, the organisations that really believe that their core mission is to support the most effective research communication – not that those are just a bunch of pretty words that get pulled out from time to time – will disavow the AAP position and commit to a positive and open discussion about how we can take the best from the current system and make it work with the best we can with the technology available. A positive discussion about managed change that enables us to get where we want to go and helps to make sure that we reap the benefits when we get there.

This bill is self-defeating as legislation but as a political act it may be effective in the short term. It could hold back the tide for a while. But publishers that support it will ultimately get wiped out as the world moves on and they spend so time pushing back the tide that they miss the opportunity to catch up. Publishers who move against the bill have a role to play in the future and are the ones with enough insight to see the way the world is moving. And those publishers who sit on the sidelines? They don’t have the institutional capability to take the strategic decisions required to survive. Choose.

Update: An interesting parallel post from John Dupuis and a trenchant expose (we expect nothing less) from Michael Eisen. Jon Eisen calls for people at the institutions and organisations with links to AAP to get on the phone and ask for them to resign from AAP. Lots of links appearing at this Google+ post from Peter Suber.

Related articles
Enhanced by Zemanta
The List of AAP Members from http://www.publishers.org/members/psp/
Posted on

The stupidity of SOPA in Scholarly Publishing

An unskippable anti-piracy film included on mo...
Image via Wikipedia

Edit and update – I’ve been told that the Macmillan supporting SOPA is the Macmillan US and not the holding company of Nature Publishing Group. NPG are however explicitly listed as members of the Association of American Publishers who are listed as supporters. The AAP list includes American Chemical Society, American Institute of Physics along with a lot of smaller society publishers. The Springer listed is apparently not the Springer that owns BioMedCentral.

It was Michael Kuhn who pointed out to me over the holiday break that both Elsevier and Macmillan (parent company of Nature Publishing Group) were listed as supporters of the Stop Online Piracy Act. If you don’t know about SOPA and why it is one of the most politically and legislatively incompetent actions of recent years then start here and look around from there. This has now been more widely picked up, Heather Morrison points out that as well as Macmillan and Elsevier that a range of other scholarly publishers as being in support and at BoingBoing Maggie Koerth-Baker suggests a boycott similar to that being targeted at other supporters.

Here I wanted to point out how utterly and stupendously stupid SOPA is in the academic communication space. Nature, every Elsevier journal, and every other academic communication medium, are full of copyright violations. The couple of paragraphs of methods text or introduction that keeps being used, that chunk of supplementary information that has appeared in a number of such places, that figure that “everyone in the field uses” but no one has any idea who drew it, as well as those figures that the authors forgot that they’d signed over the copyright to some other publisher – or didn’t understand enough about copyright to realise that they had. And that’s before we get to plagiarism issues. Or the fact that legal position over the signing of copyright agreements by authors is fraught to say the least.

Now of course reputable publishers have in place mechanisms by which authors sign off that they’ve done the right things so the journals are ok right? No. That’s the whole point of SOPA (and its partner in the US Senate, PIPA). It gives copyright holders or interested parties the right to take down an entire site based on it being the medium by which copyright violations are transmitted. So if someone, purely as a thought experiment you understand, crowd-sourced the identification of copyright violations in papers published by supporters of SOPA, then they could legitimately take down journal websites, like Science Direct and Nature.com. That’s right, just find the plagiarised papers, raise them as a copyright violation, and you can have the journal website shut down.

This is of course, just an example of why SOPA is entirely the wrong approach to dealing with online piracy. But with supposedly technically savvy organisations lined up to support it, they should be aware of what it might cost them. A fortune in responding to take down requests, a fortune in checking over every piece of every paper? Is that figure “sufficiently different”? Enjoy. Or perhaps time for a re-think about copyright in scholarly works?

Related articles
Enhanced by Zemanta
Posted on

An Open Letter to David Willetts: A bold step towards opening British research

English: Open Access logo, converted into svg,...
Image via Wikipedia

On the 8th December David Willetts, the Minister of State for Universities and Science, and announced new UK government strategies to develop innovation and research to support growth. The whole document is available online and you can see more analysis at the links at the bottom of the post.  A key aspect for Open Access advocates was the section that discussed a wholesale move by the UK to an author pays system to freely accessible research literature with SCOAP3 raised as a possible model. The report refers not to Open Access, but to freely accessible content. I think this is missing a massive opportunity for Britain to take a serious lead in defining the future direction of scholarly communication. That’s the case I attempt to lay out in this open letter. This post should be read in the context of my usual disclaimer.

Minister of State for Universities and Science

Department of Business Innovation and Skills

Dear Mr Willetts,

I am writing in the first instance to congratulate you on your stance on developing routes to a freely accessible research outputs. I cannot say I am a great fan of many current government positions and I might have wished for greater protection of the UK science budget but in times of resource constraint for research I believe your focus on ensuring the efficiency of access to and exploitation of research outputs in its widest sense is the right one.

The position you have articulated offers a real opportunity for the UK to take a lead in this area. But along with the opportunities there are risks, and those risks could entrench existing inefficiencies of our scholarly communication system. They could also reduce the value for money that the public purse, and it will be the public purse one way or another, gets for its investment. In our current circumstances this would be unfortunate. I would therefore ask you to consider the following as the implementation pathway for this policy is developed.

Firstly, the research community will be buying a service. This is a significant change from the current system where the community buys a product, the published journal. The purchasing exercise should be seen in this light and best practice in service procurement applied.

Secondly the nature of this service must be made clear. The service that is being provided must provide for any and all downstream uses, including commercial use, text mining, indeed any use that might developed at some point in the future. We are paying for this service and we must dictate its terms. Incumbent publishers will say in response that they need to retain commercial rights, or text mining rights, to ensure their viability, as indeed they have done in response to the Hargreaves Review.

This, not to put to fine a point on it, is hogwash. PLoS and BioMedCentral, both operate financially viable operations in which no downstream rights beyond that of appropriate attribution are retained by the publishers and where the author charges are lower in price then many of the notionally equivalent, but actually far more limited, offerings of more traditional publishers. High quality scholarly communication can be supported by reasonable author charges without any need for publishers to retain rights beyond those protected by their trademarks. An effective market place could therefore be expected to bring the average costs of this form of scholarly communications down.

The reason for supporting a system that demands that any downstream use of the communication be enabled is that we need innovation and development within the publishing systems well as innovation and development as a result of its content. Our scholarship is currently being held back by a morass of retained rights that prevent the development of research projects, of new technology startups and potentially new industries. The government consultation document of 14 December on the Hargreaves report explicitly notes that enabling downstream uses of content, and scholarly content in particular, can support new economic activity. It can also support new scholarly activity. The exploitation of our research outputs requires new approaches to indexing, mining, and parsing the literature. The shame of our current system is that much of this is possible today. The technology exists but is prevented from being exploited at scale by the logistical impossibility of clearing the required rights. These new approaches will require money and it is entirely appropriate, indeed desirable, that some of this work therefore occurs in the private sector. Experimentation will require both freedom to act as well as freedom to develop new business models. Our content and its accessibility and its reusability must support this.

Finally I ask you to look beyond the traditional scholarly publishing industry to the range of experimentation that is occurring globally in academic spaces, non-profits, and commercial endeavours. The potential leaps in functionality as well as the potential cost reductions are enormous. We need to work to encourage this experimentation and develop a diverse and vibrant market which both provides the quality assurance and stability that we are used to while encouraging technical experimentation and the improvement of business models. What we don’t need is a five or ten year deal that cements in existing players, systems, and practices.

Your government’s philosophy is based around the effectiveness of markets. The recent history of major government procurement exercises is not a glorious one. This is one we should work to get right. We should take our time to do so and ensure a deal that delivers on its promise. The vision of a Britain that is lead by innovation and development supported by a vibrant and globally leading research community is, I believe, the right one. Please ensure that this innovation isn’t cut off at the knees by agreeing terms that prevent our research communication tools being re-used to improve the effectiveness of that communication. And please ensure that the process of procuring these services is one that supports innovation and development in scholarly communications itself.

Yours truly,

Cameron Neylon

 

 

 

Related articles
Enhanced by Zemanta
Posted on

Open Access for the other 85%

A Landsat image of Cape Town overlaid on SRTM ...
Image via Wikipedia

One of the things you notice as a visitor from the UK in South Africa is how clean the toilets are. In restaurants, at the University, in public places. Sometimes a bit worn down but always clean. And then you start to notice how clear and clean the pavements are and your first response, well at least my first response, is that this is a sign of things going right. One element of the whole is working well. But of course one of the main reasons for this is that labour is cheap and plentiful.

And suddenly it seemed less positive. The temptation is to offer advice, experience of first world development programs, but it became clear very quickly how little of the context I was aware. The problems on the ground are often not the obvious ones – the solutions rarely easy visible from the privileged perspective of the global north. The thing I learnt was to offer advice that was tactical, not strategic, routes towards the desired goal, not the direction of travel.

So Heather Morrison’s post on the use of Creative Commons for open access, arriving as it did in the middle of my visit to Cape Town, troubled me.  Those who read here will know I am a strong partisan of the OA = CC-BY view and indeed tend to the view that we should just place things in the public domain. So my first response was a rejection. But there is an argument in there that non-commercial and share-alike terms are appropriate for the developing world, because they can protect access to the results of text mining of relevant research. These arguments are always worth taking apart because they help to illuminate the practicalities of how we take scholarly communication and make it valuable to people. It helps to pull apart the issues and raise important use cases, and the effective use of research to aid development is a key use case. Heather is also, as someone who has thought deeply about these issues, a person I will always disagree with with caution.

But here I do have to disagree. And I disagree on two levels. I gave a presentation at UCT where I talked in part about some of our open science work and a question was asked “have you thought about how accessible this is in rural Africa”. My lame answer was that we have thought about it and worried about it but not actually done anything. But the better answer for me was that it is far better for the people on the ground who really know the infrastructure, and the need, to decide what is required and do the appropriate format conversion, the printing and distribution than for us in the UK or the US to presume to know what it is of most value. This, after all is the principle of open access, allowing others to re-use as appropriate for their context precisely because we don’t know what those uses may be.

And NC terms will break this in the developing world. Right where that research is needed the communications infrastructure is patchy and in many cases non-existent. Getting that research to people, whether the raw communication, or processed or text mined material can require paper, and trucks to transport. It may require translation in format or in language or in form. All of these things cost money and in choosing NC terms we would be condemning those who could use this material to relying on charity, or which is in some cases worse, governments. A service industry that charges someone, somewhere, for production, translation, and transport is ruled out as a possible business model. Exploitation is a value-laden term, particularly in the context of Africa but in its pure and ideal sense it is neutral. We want to see research exploited for good but in choosing NC terms we are dictating the way that exploitation can occur. The risks of exploitation in the bad sense are also there. But I don’t think licences are the way to deal with it. By using legal instruments we take those key choices out of the hands of those best placed to make them.

And this is the more insidious issue from my perspective. The thing I learned in a week in Cape Town was that the last thing scholars in the developing world need is for us to make decisions for them. What we need to do, as a community, is to ensure that people can make the widest possible range of choices. Don’t get me wrong, using CC-BY has some risks in that regard, and ccZero perhaps some more, but if we act to preserve the principle of giving people the space to make their own decisisons based on local knowledge and local needs, then that is the biggest contribution we can make.

It’s not just licensing. One of the biggest holes in the entire fabric of the OA movement, is the lack of a principled and rational stance on author payment charges. Just patting people on the head and saying “oh don’t worry we won’t make you pay” is not just patronising, it is damaging to our credibility, and damaging to our own progress. One of the other key lessons I learnt from a week listening to people in Cape Town is how far ahead of the traditional centres of scholarship they are on some issues. I spend a lot of time in the UK and the US trying to convince people that there is an issue; that we need to look at how our research matters to the wider community. In South Africa the needs are clear, and scholars want to make a difference. The question is how, and how to tell when it is working.  They are damn good at this. We could learn a lot from these people about how to balance the pursuit of prestige and “research with impact” that seems to be such a struggle for us.

We need to think of the global scholarly community, not as made up of “us” and “those who are catching up”, but as made up of different groups with different priorities and different needs, and crucially with different experiences and value to bring to a global endeavour. The creation of a “special OA” for the developing world runs the risk of perpetuating a view in which 85% of the world will always be catching up. Yes we need to try and build systems that ensure access to all scholarship, primary and derived. How to do that is an important debate, and one I hope will be at the centre of the Berlin10 conference to be held in Stellenbosch in South Africa next year. To not use that meeting to address the issues and challenges of business models and safeguards that help preserve access and optimise impact for the developing world would be a terrible loss.

OA is about enabling people, enabling business, and enabling development. There is a global community of scholars who get that, who want to be part of the wider community, and have their own skills and expertise to bring. They also want to share and contribute to the resource needs of scholarly communication in an appropriate and equitable way. We need to enable them to do that and get out of their way. After all, we might learn something.

Enhanced by Zemanta
Posted on

Good practice in research coding: What are the targets and how do we get there…?

EN{code}D Exhibition, The Building Centre, Sto...
Image by olliepalmer.com via Flickr

The software code that is written to support and manage research sits at a critical intersection of our developing practice of shared, reproducible, and re-useble research in the 21st century. Code is amongst the easiest things to usefully share, being both made up of easily transferable bits and bytes but also critically carrying its context with it in a way that digital data doesn’t do. Code at its best is highly reproducible: it comes with the tools to determine what is required to run it (make files, documentation of dependencies) and when run should (ideally) generate the same results from the same data. Where there is a risk that it might not, good code will provide tests of one sort or another than you can run to make sure that things are ok before proceeding. Testing, along with good documentation is what ensures that code is re-usable, that others can take it and efficiently build on it to create new tools, and new research.

The outside perspective, as I have written before, is that software does all of this better than experimental research. In practice the truth is that there are frameworks that make it possible for software to do a very good job on these things, but that in reality doing a good job takes work; work that is generally not done. Most software for research is not shared, is not well documented, generates results that are not easily reproducible, and does not support re-use and repurposing through testing and documentation. Indeed much like most experimental research. So how do we realise the potential of software to act as an exemplar for the rest of our research practice?

Nick Barnes of the Climate Code Foundation developed the Science Code Manifesto, a statement of how things ought to be (I was very happy to contribute and be a founding signatory) and while for many this may not go far enough (it doesn’t explicitly require open source licensing) it is intended as a practical set of steps that might be adopted by communities today. This has already garnered hundreds of endorsers and I’d encourage you to sign up if you want to show your support. The Science Code Manifesto builds on work over many years of Victoria Stodden in identifying the key issues and bringing them to wider awareness with both researchers and funders as well as the work of John Cook, Jon Claerbout, and Patrick Vanderwalle at ReproducibleResearch.net.

If the manifesto and the others work are actions that aim (broadly) to set out the principles and to understand where we need to go then Open Research Computation is intended as a practical step embedded in today’s practice. Researchers need the credit provided by conventional papers, so if we can link papers in a journal that garners significant prestige, with high standards in the design and application of the software that is described we can link the existing incentives to our desired practice. This is a high wire act. How far do we push those standards out in front of where most of the community is. We explicitly want ORC to be a high profile journal featuring high quality software, for acceptance to be a mark of quality that the community will respect. At the same time we can’t ask for the impossible. If we set standards so high that no-one can meet them then we won’t have any papers. And with no papers we can’t start the process of changing practice. Equally, allow too much in and we won’t create a journal with a buzz about it. That quality mark has to be respected as meaning something by the community.

I’ll be blunt. We haven’t had the number of submissions I’d hoped for. Lots of support, lots of enquiries, but relatively few of them turning into actual submissions. The submissions we do have I’m very happy with. When we launched the call for submissions I took a pretty hard line on the issue of testing. I said that, as a default, we’d expect 100% test coverage. In retrospect that sent a message that many people felt they couldn’t deliver on. Now what I meant by that was that when testing fell below that standard (as it would in almost all cases) there would need to be an explanation of what the strategy for testing was, how it was tackled, and how it could support people re-using the code. The language in the author submission guidelines has been softened a bit to try and make that clearer.

What I’ve been doing in practice is asking reviewers and editors to comment on how the testing framework provided can support others re-using the code. Are the tests provided adequate to help someone getting started on the process of taking the code, making sure they’ve got it working, and then as they build on it, giving them confidence they haven’t broken anything. For me this is the critical question, does the testing and documentation make the code re-usable by others, either directly in its current form, or as they build on it. Along the way we’ve been asking whether submissions provide documentation and testing consistent with best practice. But that always raises the question of what best practice is. Am I asking the right questions? And were should we ultimately set that bar?

Changing practice is tough, getting the balance right is hard. But the key question for me is how do we set that balance right? And how do we turn the aims of ORC, to act as a lever to change the way that research is done, into practice?

 

Related articles
Enhanced by Zemanta
Posted on

Reflections on research data management: RDM is on the up and up but data driven policy development seems a long way off.

Data Represented in an Interactive 3-D Form
Image by Idaho National Laboratory via Flickr

I wrote this post for the Digital Curation Centre blog following the Research Data Management Forum meeting run in Warwick a few weeks back. If you feel moved to comment I’d ask you to do it over there.

The Research Data Management movement is moving on apace. Tools are working and adoption is growing. Policy development is starting to back up the use of those tools and there are some big ambitious goals set out for the next few years. But has the RDM movement taken the vision of data intensive research to its heart? Does the collection, sharing, and analysis of data about research data management meet our own standards? And is policy development based on and assessed against that data? Can we be credible if it is not?

Watching the discussion on research data management over the past few years has been an exciting experience. The tools, that have been possible for some years, now show real promise as the somewhat raw and ready products of initial development are used and tested.

Practice is gradually changing, if unevenly across different disciplines, but there is a growing awareness of data and that it might be considered important. And all of this is being driven increasingly by the development of policies on data availability, data management, and data archiving that stress the importance of data as a core output of public research.

The vision of the potential of a data rich research environment is what is driving this change. It is not important whether individual researchers, or even whole community, gets how fundamental a change the capacity to share and re-use data really is. The change is driven by two forces fundamentally external to the community.

The first is political, the top down view from government that publicly funded research needs to gain from the benefits they see in data rich commerce. A handful of people really understand how data works at these scales but these people have the ear of government.

The second force is one of competition. In the short term adopting new practices, developing new ways of doing research, is a risk. In the longer term, those who adopt more effective and efficient approaches will simply out compete those who do not or can not. This is already starting happening in those disciplines already rich in shared data and the signs are there that other disciplines are approaching a tipping point.

Data intensive research enables new types of questions to be asked, and it allows us to answer questions that were previously difficult or impossible to get reliable answers on. Questions about weak effects, small correlations, and complex interactions. The kind of questions that bedevil strategic decision-making and evidence based policy.

So naturally you’d expect that the policy development in this area, being driven by people excited by the vision of data intensive research, would have deeply embedded data gathering, model building, and analysis of how research data is being collected, made available, and re-used.

I don’t mean opinion surveys, or dipstick tests, or case studies. These are important but they’re not the way data intensive research works. They don’t scale, they don’t integrate, and they can’t provide the insight into the weak effects in complex systems that are needed to support decision making about policy.

Data intensive research is about tracking everything, logging every interaction, going through download logs, finding every mention of a specific thing wherever on the web it might be.

It’s about capturing large amounts of weakly structured data and figuring out how to structure it in a way that supports answering the question of interest. And its about letting the data guide you the answers it suggests, rather than looking within it for what we “know” should be in there.

What I don’t see when I look at RDM policy development is the detailed analysis of download logs, the usage data, the click-throughs on website. Where are the analyses of IP ranges of users, automated reporting systems, and above all, when new policy directions are set where is the guidance on data collection and assessment of performance against those policies?

Without this, the RDM community is arguably doing exactly the same things that we complain about in researcher communities. Not taking a data driven view of what we are doing.

I know this is hard. I know it involves changing systems, testing things in new ways, collecting data in ways we are not used. Even imposing disciplinary approaches that are well outside the comfort zone of those involved.

I also know there are pockets of excellent practice and significant efforts to gather and integrate information. But they are pockets. And these are exactly the things that funders and RDM professionals and institutions are asking of researchers. They are the right things to be asking for, and we’re making real progress towards realizing the vision of what is possible with data intensive research.

But just imagine if we could support policy development with that same level of information. At a pragmatic and political level it makes a strong statement when we “eat our own dogfood”. And there is no better way to understand which systems and approaches are working and not working than by using them ourselves.

Enhanced by Zemanta
Posted on

Designing for the phase change: Local communities and shared infrastructure

Pink crystal.
Image via Wikipedia

Michael Nielsen‘s talk at Science Online was a real eye opener for many of us who have been advocating for change in research practice. He framed the whole challenge of change as an example of a well known problem, that of collective action. How do societies manage big changes when those changes often represent a disadvantage to many individuals, at least in the short term. We can all see the global advantages of change but individually acting on them doesn’t make sense.

Michael placed this in the context of other changes, that of countries changing which side of the road they drive on, or the development of trade unions, that have been studied in some depth by political economists and similar academic disciplines. The message of these studies is that change usually occurs in two phases. First local communities adopt practice (or at least adopt a view that they want things changed in the case of which side of the road they drive on) and then these communities discover each other and “agglomerate”, or in the language of physical chemistry there are points of nucleation which grow to some critical level and then the whole system undergoes a phase change, crystallising into a new form.

These two phases are driven by different sets of motivations and incentives. At a small scale processes are community driven, people know each other, and those interactions can drive and support local actions, expectations, and peer pressure. At a large scale the incentives need to be different and global. Often top down policy changes (as in the side of the road) play a significant role here, but equally market effects and competition can also fall into place in a way that drives adoption of new tools or changes in behaviour. Think about the way new research techniques get adopted: first they are used by small communities, single labs, with perhaps a slow rate of spread to other groups. For a long time it’s hard for the new approach to get traction, but suddenly at some point either enough people are using it that its just the way things are done, or conversely those who are using it are moving head so fast that everyone else has to pile in just to keep up. It took nearly a decade for PCR for instance to gain widespread acceptance as a technique in molecular biology but when it did it went from being something people were a little unsure of to being the only way to get things done very rapidly.

So what does this tell us about advocating for, or designing for, change. Michael’s main point was that narrow scope is a feature, not a bug, when you are in that first phase. Working with small scale use cases, within communities is the way to get started. Build for those communities and they will become your best advocates, but don’t try to push the rate of growth, let it happen at the right rate (whatever that might be – and I don’t really know how to tell to be honest). But we also need to build in the grounding for the second phase.

The way these changes generally occur is through an accidental process of accretion and agglomeration. The phase change crystallises out around those pockets of new practice. But, to stretch the physical chemistry analogy, doesn’t necessarily crystallise in the form one would design for. But we have an advantage, if we design in advance to enable that crystallisation then we can prepare communities and prepare tooling for when it happens and we can design in the features that will get use closer to the optimum we are looking for.

What does this mean in practice? It means that when we develop tools and approaches it is more important for our community to have standards than it is for there to be an effort on any particular tool or approach. The language we use, that will be adopted by communities we are working with, should be consistent, so that when those communities meet they can communicate. The technical infrastructure we use should be shared, and we need interoperable standards to ensure that those connections can be made. Again, interchange and interoperability are more important than any single effort, any single project.

If we really believe in the value of change then we need to get these things together before we push them too hard into the diverse set of research communities where we want them to take root. We really need to get interoperability, standards, and language sorted out before the hammer of policy comes down and forces us into some sort of local minimum. In fact, it sounds rather like we have a collective action problem of our own. So what are we going to do about that?

Related articles
Enhanced by Zemanta
Posted on

Building the perfect data repository…or the one that might get used

A UV-vis readout of bis(triphenylphosphine) ni...
Image via Wikipedia

While there has been a lot of talk about data repositories and data publication there remains a real lack of good tools that are truly attractive to research scientists and also provide a route to more general and effective data sharing. Peter Murray-Rust has recently discussed the deficiencies of the traditional institutional repository as a research data repository in some depth [1, 2, 3, 4].

Data publication approaches and tools are appearing including Dryad, Figshare, BuzzData and more traditional venues such as GigaScience from BioMedCentral but these are all formal mechanisms that involve significant additional work alongside an active decision to “publish the data”. The research repository of choice remains a haphazard file store and the data sharing mechanism of choice remains email. How do we bridge this gap?

One of the problems with many efforts in this space is how they are conceived and sold as the user. “Making it easy to put your data on the web” and “helping others to find your data” solve problems that most researchers don’t think they have. Most researchers don’t want to share at all, preferring to retain as much of an advantage through secrecy as possible. Those who do see a value in sharing are for the most part highly skeptical that the vast majority of research data can be used outside the lab in which it was generated. The small remainder who see a value in wider research data sharing are painfully aware of how much work it is to make that data useful.

A successful data repository system will start by solving a different problem, a problem that all researchers recognize they have, and will then nudge the users into doing the additional work of recording (or allowing the capture of) the metadata that could make that data useful to other researchers. Finally it will quietly encourage them to make the data accessible to other researchers. Both the nudge and the encouragement will arise by offering back to the user immediate benefits in the form of automated processing, derived data products, or other more incentives.

But first the problem to solve. The problem that researchers recognize they have, in many cases prompted by funder data sharing plan requirements, is to properly store and back up their data. A further problem most PIs realize they have is access to the data of their group in a useful form. So they initial sales pitch for a data repository is going to be local and secure backup and sharing within the group. This has to be dead simple and ideally automated.

Such a repository will capture as much data as possible at source as it is generated. Just grabbing the file and storing it with the minimal contextual data, of who (which directory was it saved in), when (aquired from the initial file datestamp), and what (where has it come from), backing it up and exposing it to the research group (or subsets of it) via a simple web service. It will almost certainly involve some form of Dropbox-style system which synchronises a users own data across their own devices. Here is an immediate benefit. I don’t have to get the pen drive out of my pocket if I’m confident my new data will be on my laptop when I get back to it.

It will allow for simple configuration on each instrument that sets up a target directory and registers a filetype so that the system can recognize what instrument, or computational process, a file came from (the what). The who can be more complex, but a combination of designated directories (where a user has their own directory of data on a specific instrument), login info, and where required low irritant requests or claiming systems can be built. The when is easy. The sell here is in two parts, directory synching across computers means less mucking around with USB keys. And the backup makes everyone feel better, the researcher in the lab, the PI, and the funder.

So far so good, and there are in fact examples of systems like this that exist in one form or another or are being developed including DataStage within the DataFlow project from David Shotton’s group at Oxford, the Clarion project (PMR group at Cambridge) and Smart Research Frameworks (that I have an involvement with) led by Jeremy Frey at Southampton. I’m sure there are dozens of other systems or locally developed tools that do similar things and these are a good starting position.

The question is how do you take systems like this and push it to the next level. How do capture, or encourage the user to provide, enough metadata to actually make the stored data more widely useful? Particularly when they don’t have any real interest in sharing or data publication? I think that there is a significant potential in offering downstream processing of the data.

If This Then That (IFTTT) is a startup that has got quite a bit of attention over the past few weeks as it has come into public beta. The concept is very simple. For a defined set of services there are specific triggers (posting a tweet, favouriting a YouTube video) that can be used to set off another action at another service (send an email, bookmark the URL of the tweet or the video). What if we could offer data processing steps to the user? If the processing steps happen automatically but require a bit more metadata will that provide the incentive to get that data in?

This concept may sound a lot like the functionality provided by workflow engines but there is a difference. Workflow systems are generally very difficult for the general user to setup. This is mostly because they solve a general problem, that of putting any object into any suitable process. IFTTT offers something much simpler, a small set of common actions on common objects, that solves the 80/20 problem. Workflows are hard because they can do anything with any object. And that flexibility comes at a price because it is difficult to know whether that csv file is from a UV-Vis instrument, a small-angle x-ray experiment, or a simulated data set.

But locally, within a research group those there is a more limited set of data objects. With a local (or localized) repository it is possible to imagine plugins that do common single steps on common files. And because the configuration is local there is much less metadata required. But in turn that configuration provides metadata. If a particular filetype from a directory is configured for automated calculation of A280 for protein concentrations then we know that those data files are UV-Vis spectra. What is more, once we know that we can offer an automated protein concentration calculator. This will only work if the system knows what protein you are measuring, an incentive to identify the sample when you do the measurement.

The architecture of such a system would be relatively straightforward. A web service provides the cloud based backup and configuration for captured data files and users. Clients that sit on group users’ personal computers as well as on instruments grab their configuration information from the central repository. They might simply monitor specified directories, or they might pop up with a defined set of questions to capture additional metadata. Users register the instruments that they want to “follow” and when a new data file is generated with their name on it, it is also synchronized back to their registered devices.

The web service provides a plugin architecture where appropriate plugins for the group can be added from some sort of marketplace or online. Plugins that process data to generate additional metadata (e.g. by parsing a log file) can add that to the record of the data file. Those that generate new data will need to be configured as to where that data should go and who should have it synchronised. The plugins will also generate a record of what they did, providing an audit and provenance trail. Finally plugins can provide notifcation back to the users via email, the webservice, or a desktop client, of queued processes that need more information to proceed. The user can mute these but equally the encouragement is there to provide a little more info.

Finally the question of sharing and publication. For individual data file sharing sites like Figshare, plugins might enable straightforward direct submission of files submitted into a specific directory. For collections of data, such as those supported by Dryad, there will need to be to group files together but again this could be as simple as creating a directory. Even if files are copied and pasted or removed from their “proper directories” the system stands a reasonable chance of recognizing files it has already seen and inferring their provenance.

By making it easy to share and easier to satisfy data sharing requests by pushing data to a public space (while still retaining some ability to see how it is being used and by who) the substrate that is required to build better plugins, more functionality, and above all better discovery tools will be provided, and in turn those tools will start to develop. As the tools and functionality develop then the value gained by sharing will rise creating a virtuous circle, encouraging both good data management practice, good metadata provision, and good sharing.

This path starts with things we can build today, and in some cases already exist. It becomes more speculative as it goes forward. There are issues with file synching and security. Things will get messy. The plugin architecture is nothing more than hand waving at the moment and success will require a whole ecosystem of repositories and tools for operating on them. But it is a way forward that looks plausible to me. One that solves the problems researchers have today and guides them towards a tomorrow where best practice is a bit closer to common practice.

 

Enhanced by Zemanta
Posted on

Incentives: Definitely a case of rolling your own

Lakhovsky: The Convesation; oil on panel (Бесе...
Image via Wikipedia

Science Online London ran late last week and into the weekend and I was very pleased to be asked to run a panel, broadly speaking focused on evaluation and incentives. Now I had thought that the panel went pretty well but I’d be fibbing if I said I wasn’t a bit disappointed. Not disappointed with the panel members or what they said. Yes it was a bit more general than I had hoped and there were things that I wished we’d covered but the substance was good from my perspective. My disappointment was with the response from the audience, really on two slightly different points.

The first was the lack of response to what I thought were some of the most exciting things I’ve heard in a long time from major stakeholders. I’ll come back to that later. But a bigger disappointment was that people didn’t seem to connect the dots to their own needs and experiences.

Science Online, both in London and North Carolina forms, has for me always been a meeting where the conversation proceeds at a more sophisticated level than the usual. So I pitched the plan of the session at where I thought the level should be. Yes we needed to talk about the challenges and surface the usual problems, non-traditional research outputs and online outputs in particular don’t get the kind of credit that papers do, institutions struggle to give credit for work that doesn’t fit in a pigeonhole, funders seem to reward only the conventional and traditional, and people outside the ivory tower struggle to get either recognition or funding. These are known challenges, the question is how to tackle them.

The step beyond this is the hard one. It is easy to say that incentives need to change. But incentives don’t drop from heaven. Incentives are created within communities and they become meaningful when they are linked to the interests of stakeholders with resources. So the discussion wasn’t really about impact, or funding, or to show that nothing can be done by amateurs. The discussion was about the needs of institutions and funders and how they can be served by what is being generated by the online community. It was also about the constraints they face in acting. But fundamentally you had major players on the stage saying “this is the kind of thing we need to get the ball rolling”.

Make no mistake, this is tough. Everyone is constrained and resources are tight but at the centre of the discussion were the key pointers to how to cut through the knot. The head of strategy at a major research university stated that universities want to play a more diverse role, want to create more diverse scholarly outputs, and want to engage with the wider community in new ways. That smart institutions will be looking to diversify. The head of evaluation at a major UK funder said that funders really want to know about non-traditional outputs and how they were having a wider impact. That these outputs are amongst the best things they can talk about to government. That they will be crucial to make the case to sustain science funding.

Those statements are amongst the most direct and exciting I have heard in some years of advocacy in this space. The opportunity is there, if you’re willing to put the effort in to communicate and to shape what you are doing to match to match their needs. As Michael Nielsen said in his morning keynote this is a collective action problem. That means finding what unites the needs of those doing with the needs of those with resources. It means compromise, and it means focusing on the achievable, but the point of the discussion was to identify what might be achievable.

So mostly I was disappointed that the excitement I felt wasn’t mirrored in the audience. The discussion about incentives has to move on. Saying that “institutions should do X” or “funders should do Y” gets us nowhere. Understanding what we can do together with funders and institutions and other communities to take the online agenda forward and understanding what the constraints are is where we need to go. The discussion showed that both institutions and funders know that they need what the community of online scientists can do. They don’t know how to go about it, and they don’t even know very much what we are doing, but they want to know. And when they do know they can advise and help and they can bring resources to bear. Maybe not all the resources you would like, and maybe not for all the things you would like, but resources nonetheless.

With a lot of things it is easy to get too immersed in the detail of these issues and to forget that people are looking in from the outside without the same context. I guess the fact that I pulled out what might have seemed to the audience to be just asides as the main message is indicative of that. But I really want to get that message out because I think it  is critical if the community of online scientists wants to be the mainstream. And I think it should be.

The bottom line is that smart funders and smart institutions value what is going on online. They want to support it, they want to be seen to support it, but they’re not always sure how to go about it and how to judge its quality. But they want to know more. That’s where you come in and that’s why the session was relevant. Lars Fischer had it absolutely right: “I think the biggest and most consequential incentive for scientists is (informal) recognition by peers.” You know, we know, who is doing the good stuff and what is valuable. Take that conversation to the funders and the institutions, explain to them what’s good and why, and tell the story of what the value is. Put it in your CV, demand that promotion panels take account of it, whichever side of the table you are on. Show that you make an impact in language that they understand. They want to know. They may not always be able to act – funding is an issue – but they want to and they need your help. In many ways they need our help more than we need theirs. And if that isn’t an incentive then I don’t know what is.

Related articles
Enhanced by Zemanta