Home » Blog, Featured

Good practice in research coding: What are the targets and how do we get there…?

23 November 2011 2 Comments
EN{code}D Exhibition, The Building Centre, Sto...

Image by olliepalmer.com via Flickr

The software code that is written to support and manage research sits at a critical intersection of our developing practice of shared, reproducible, and re-useble research in the 21st century. Code is amongst the easiest things to usefully share, being both made up of easily transferable bits and bytes but also critically carrying its context with it in a way that digital data doesn’t do. Code at its best is highly reproducible: it comes with the tools to determine what is required to run it (make files, documentation of dependencies) and when run should (ideally) generate the same results from the same data. Where there is a risk that it might not, good code will provide tests of one sort or another than you can run to make sure that things are ok before proceeding. Testing, along with good documentation is what ensures that code is re-usable, that others can take it and efficiently build on it to create new tools, and new research.

The outside perspective, as I have written before, is that software does all of this better than experimental research. In practice the truth is that there are frameworks that make it possible for software to do a very good job on these things, but that in reality doing a good job takes work; work that is generally not done. Most software for research is not shared, is not well documented, generates results that are not easily reproducible, and does not support re-use and repurposing through testing and documentation. Indeed much like most experimental research. So how do we realise the potential of software to act as an exemplar for the rest of our research practice?

Nick Barnes of the Climate Code Foundation developed the Science Code Manifesto, a statement of how things ought to be (I was very happy to contribute and be a founding signatory) and while for many this may not go far enough (it doesn’t explicitly require open source licensing) it is intended as a practical set of steps that might be adopted by communities today. This has already garnered hundreds of endorsers and I’d encourage you to sign up if you want to show your support. The Science Code Manifesto builds on work over many years of Victoria Stodden in identifying the key issues and bringing them to wider awareness with both researchers and funders as well as the work of John Cook, Jon Claerbout, and Patrick Vanderwalle at ReproducibleResearch.net.

If the manifesto and the others work are actions that aim (broadly) to set out the principles and to understand where we need to go then Open Research Computation is intended as a practical step embedded in today’s practice. Researchers need the credit provided by conventional papers, so if we can link papers in a journal that garners significant prestige, with high standards in the design and application of the software that is described we can link the existing incentives to our desired practice. This is a high wire act. How far do we push those standards out in front of where most of the community is. We explicitly want ORC to be a high profile journal featuring high quality software, for acceptance to be a mark of quality that the community will respect. At the same time we can’t ask for the impossible. If we set standards so high that no-one can meet them then we won’t have any papers. And with no papers we can’t start the process of changing practice. Equally, allow too much in and we won’t create a journal with a buzz about it. That quality mark has to be respected as meaning something by the community.

I’ll be blunt. We haven’t had the number of submissions I’d hoped for. Lots of support, lots of enquiries, but relatively few of them turning into actual submissions. The submissions we do have I’m very happy with. When we launched the call for submissions I took a pretty hard line on the issue of testing. I said that, as a default, we’d expect 100% test coverage. In retrospect that sent a message that many people felt they couldn’t deliver on. Now what I meant by that was that when testing fell below that standard (as it would in almost all cases) there would need to be an explanation of what the strategy for testing was, how it was tackled, and how it could support people re-using the code. The language in the author submission guidelines has been softened a bit to try and make that clearer.

What I’ve been doing in practice is asking reviewers and editors to comment on how the testing framework provided can support others re-using the code. Are the tests provided adequate to help someone getting started on the process of taking the code, making sure they’ve got it working, and then as they build on it, giving them confidence they haven’t broken anything. For me this is the critical question, does the testing and documentation make the code re-usable by others, either directly in its current form, or as they build on it. Along the way we’ve been asking whether submissions provide documentation and testing consistent with best practice. But that always raises the question of what best practice is. Am I asking the right questions? And were should we ultimately set that bar?

Changing practice is tough, getting the balance right is hard. But the key question for me is how do we set that balance right? And how do we turn the aims of ORC, to act as a lever to change the way that research is done, into practice?


Enhanced by Zemanta


  • Andrew Dalke said:

    I couldn’t find a way to comment on the Science Code Manifesto site, or ask questions. I trust it’s okay to ask about it here.

    When the Manifesto made its way to the Computational Chemistry List, I had a couple of practical concerns with it.

    1) How much should it cost to get access to the source code? The Free Software Foundation, which has different goals than scientific software, asserts that it is an essential freedom that you be able to charge as much as you wish. Am I free to publish a paper but charge $1 million for access to the GPLv3 licensed software? Is the curator similarly free to do so?

    2) What is the curator’s obligation with respect to copyright violations in the submitted software? I think of the specific case of a chemistry toolkit available under LGPLv2, using components also distributed under LGPLv2, but the main package did not include the credits as required by the subpackage license. Is it acceptable for the curator to modify the package? If the paper author is unable or unwilling to change it, then does the curator withdraw only the software, or is the paper withdrawn as well?

    I am also curious to know your opinion on something which has been bugging me. Suppose I am asked to do a peer-review of a paper, which includes GPL software in the supporting materials. I of course get access to the software in order to review it. Under the terms of the GPL, this is a distribution event so I am now free to redistribute the software. While that’s inappropriate as part of the peer-review process, it’s not illegal. Compare that to redistributing the paper, which would be a violation of copyright law.

    Is this a valid worry, or are the social conventions on not distributing peer-review materials strong enough?

  • Greg Wilson said:

    The commercial product I’m working on has 92% code coverage — but that’s only if you consider the back-end code that we wrote.  I don’t know what the coverage is of the libraries we’re using (should I worry?), nor of the Javascript that makes the front end sing and dance (it’s at least as complicated as the back-end code).  I’m not particularly worried by this — it’s higher coverage than most commercial projects I’ve worked on, and high enough that I’m confident there are no trivial-to-stumble-over bugs in our code.  I don’t think I’d expect any more from scientific codes…