science blogging – Science in the Open

September 8, 2011September 8, 2011

Incentives: Definitely a case of rolling your own

Lakhovsky: The Convesation; oil on panel (Ð‘ÐµÑÐµ... — Image via Wikipedia

Science Online London ran late last week and into the weekend and I was very pleased to be asked to run a panel, broadly speaking focused on evaluation and incentives. Now I had thought that the panel went pretty well but Iâ€™d be fibbing if I said I wasnâ€™t a bit disappointed. Not disappointed with the panel members or what they said. Yes it was a bit more general than I had hoped and there were things that I wished weâ€™d covered but the substance was good from my perspective. My disappointment was with the response from the audience, really on two slightly different points.

The first was the lack of response to what I thought were some of the most exciting things Iâ€™ve heard in a long time from major stakeholders. Iâ€™ll come back to that later. But a bigger disappointment was that people didnâ€™t seem to connect the dots to their own needs and experiences.

Science Online, both in London and North Carolina forms, has for me always been a meeting where the conversation proceeds at a more sophisticated level than the usual. So I pitched the plan of the session at where I thought the level should be. Yes we needed to talk about the challenges and surface the usual problems, non-traditional research outputs and online outputs in particular donâ€™t get the kind of credit that papers do, institutions struggle to give credit for work that doesnâ€™t fit in a pigeonhole, funders seem to reward only the conventional and traditional, and people outside the ivory tower struggle to get either recognition or funding. These are known challenges, the question is how to tackle them.

The step beyond this is the hard one. It is easy to say that incentives need to change. But incentives donâ€™t drop from heaven. Incentives are created within communities and they become meaningful when they are linked to the interests of stakeholders with resources. So the discussion wasnâ€™t really about impact, or funding, or to show that nothing can be done by amateurs. The discussion was about the needs of institutions and funders and how they can be served by what is being generated by the online community. It was also about the constraints they face in acting. But fundamentally you had major players on the stage saying â€œthis is the kind of thing we need to get the ball rollingâ€.

Make no mistake, this is tough. Everyone is constrained and resources are tight but at the centre of the discussion were the key pointers to how to cut through the knot. The head of strategy at a major research university stated that universities want to play a more diverse role, want to create more diverse scholarly outputs, and want to engage with the wider community in new ways. That smart institutions will be looking to diversify. The head of evaluation at a major UK funder said that funders really want to know about non-traditional outputs and how they were having a wider impact. That these outputs are amongst the best things they can talk about to government. That they will be crucial to make the case to sustain science funding.

Those statements are amongst the most direct and exciting I have heard in some years of advocacy in this space. The opportunity is there, if youâ€™re willing to put the effort in to communicate and to shape what you are doing to match to match their needs. As Michael Nielsen said in his morning keynote this is a collective action problem. That means finding what unites the needs of those doing with the needs of those with resources. It means compromise, and it means focusing on the achievable, but the point of the discussion was to identify what might be achievable.

So mostly I was disappointed that the excitement I felt wasnâ€™t mirrored in the audience. The discussion about incentives has to move on. Saying that â€œinstitutions should do Xâ€ or â€œfunders should do Yâ€ gets us nowhere. Understanding what we can do together with funders and institutions and other communities to take the online agenda forward and understanding what the constraints are is where we need to go. The discussion showed that both institutions and funders know that they need what the community of online scientists can do. They donâ€™t know how to go about it, and they donâ€™t even know very much what we are doing, but they want to know. And when they do know they can advise and help and they can bring resources to bear. Maybe not all the resources you would like, and maybe not for all the things you would like, but resources nonetheless.

With a lot of things it is easy to get too immersed in the detail of these issues and to forget that people are looking in from the outside without the same context. I guess the fact that I pulled out what might have seemed to the audience to be just asides as the main message is indicative of that. But I really want to get that message out because I think it Â is critical if the community of online scientists wants to be the mainstream. And I think it should be.

The bottom line is that smart funders and smart institutions value what is going on online. They want to support it, they want to be seen to support it, but theyâ€™re not always sure how to go about it and how to judge its quality. But they want to know more. Thatâ€™s where you come in and thatâ€™s why the session was relevant. Lars Fischer had it absolutely right: â€œI think the biggest and most consequential incentive for scientists is (informal) recognition by peers.â€ You know, we know, who is doing the good stuff and what is valuable. Take that conversation to the funders and the institutions, explain to them whatâ€™s good and why, and tell the story of what the value is. Put it in your CV, demand that promotion panels take account of it, whichever side of the table you are on. Show that you make an impact in language that they understand. They want to know. They may not always be able to act – funding is an issue – but they want to and they need your help. In many ways they need our help more than we need theirs. And if that isnâ€™t an incentive then I donâ€™t know what is.

Submission to the Royal Society Enquiry (cameronneylon.net)
Recognising, Appreciating, Measuring and Evaluating the Impact of Open Science (ukwebfocus.wordpress.com)
A new sustainability model: Major funders to support OA journal (cameronneylon.net)
(S)low impact research and the importance of open in maximising re-use (cameronneylon.net)
Some questions for British research policy (softmachines.org)
Proceed with caution when setting up financial incentives for general practice doctors (eurekalert.org)

July 30, 2009December 30, 2009

Telling stories…

On Tuesday I was able to sit in on a conversation that is regularly held within the Computer Science department at University of Toronto that focuses broadly on what can computer science bring as a discipline and a skill set to the sciences more generally. The conversation is lead by Steve Easterbrook so there is a focus on climate science but we also roamed much more widely than that.

A key question that was raised, one which many of us have been struggling with for some time, is how to describe and publish descriptions of the progress of research projects in a way that provides a route in for non-specialists. Blogs provide a great way to do this, either as a generic journal with more or less detail, or as an overlay over a more detailed open notebook. Jean-Claude Bradley’s UsefulChem blog is a great example of the latter, and the blogs of Rosis Redfield’s group an example of the former.

The conversation was interesting for me in that it pinned down the idea and necessity of creating a narrative. This contrasts with the kind of (largely incomprehensible) detail found in a notebook which is usually fragmented and often distributed. One of the things that researchers are quite poor at in my experience is actually recording the why of an experiment; the question of how it fits into the wider context. Again blogs are a great format for doing this but where is the motivation? Writing that narrative in any form is hard work, a classic example of work that “takes me away from the bench” so how can it be justified?

One reason is that it raises the profile of the research, always an important issue in today’s research environment. But this is more important to some people than to others. Another very valid reason is to take personal notes, to create a personal narrative of what you are doing and have done that you can return to and use as an index to your own work. In a later discussion with Alicia Grubb she mentioned that her supervisors insisted on her blogging about literature that she collected. Equally taking notes in a bookmarking service could provide the same functionality. But understanding the context in which you bookmarked something is valuable. Brent Mombourquette, an undergraduate student also demonstrated a nice Firefox plugin that he had developed which captures and displays browsing history as a directed graph which is an interesting tool to think about in this context. I’ll write more about some of the fabulous student demos I saw later.

For me though, the biggest benefit of making your research accessible, is that it provides an entry route for new people to come in and help. The story of Galaxy Zoo shows how by placing a question within an understandable narrative you can enable people to come in and help out. No-one is going to come in from the outside and comment on my lab notebook unless they are already a specialist with a specific question. I’ve often thought I should start another blog to discuss more generally what is happening in my “real” research. Maybe this is the time to do that.

But you can also take this one step further. There was a debate in the comments on the RealClimate blog a few months ago about making the details of data and analysis publicly available. One real and valid concern was that denialists would dig into the detail and mis-represent problems or mistakes to advance their own agenda. Dealing with this kind of thing takes up valuable time, time the average researcher, particularly if they are committed to taking time to engage with a wider community, doesn’t have. My question was whether you could configure that public release of data and process in such a way that even those who are working against you are helping you. If people are searching your code for bugs then surely there must be a way of taking advantage of that?

The argument that releasing data costs you time sounds compelling when it comes from a researcher. But equally the same argument sounds dangerous when it is made by, for example, a government. As Steve said tongue in cheek, perhaps channeling some recently removed political leaders, “clearly we’re not going to release this data because it would take us time to deal with public complaints, and that will costs taxpayer. In fact, we’re not even going to run the consultation because that would cost money. It’s much cheaper and more effective use of your tax dollars if you just trust us to do the right thing.” The situations clearly aren’t exact parallels and resources for communication are much more limited in a research setting but it would be interesting to think about parallel cases in different domains, such as government and research, and how the domain effects the credibility of the argument. If you believe in the value of sunshine as disinfectant for government data then you need a strong case to argue the same doesn’t apply to research data.

But if you decide that you want to make that narrative public, or even better the narrative along with the underlying data, it does take work to make it comprehensible. As I’ve discovered recently such posts don’t translate easily into papers so making the argument that you can re-use the text doesn’t really work, at least for me. To make this worthwhile you either have to be required to do it: JISC in the UK basically requires that all funded projects have blogs; or you have to believe and work towards the benefits it can bring you. In a sense this is actually just recapturing the idea of the research notebook that many historical scientists kept, and which make such rich pickings for modern historians of science. Somewhere along the line we lost that. There are lots of tools around that can help you create that narrative, from clickstreams, records, environmental capture, but these remain only an aide memoire. Making the story is something that will probably remain a purely human activity. It is something that we seem highly evolved to do and it remains the most effective means of human to human communication. The computers can help, and they can provide the detail to dig down into if desired, but the story itself will remain ours alone for a while yet I think.

January 15, 2009December 30, 2009

Very final countdown to Science Online 09

I should be putting something together for the actual sessions I am notionally involved in helping running but this being a very interactive meeting perhaps it is better to leave things to very last minute. Currently I am at a hotel at LAX awaiting an early flight tomorrow morning. Daily temperatures in the LA area have been running around 25-30 C for the past few days but we’ve been threatened with the potential for well below zero in Chapel Hill. Nonetheless the programme and the people will more than make up for it I have no doubt. I got to participate in a bit of the meeting last year via streaming video and that was pretty good but a little limited – not least because I couldn’t really afford to stay up all night unlike some people who were far more dedicated.

This year I am involved in three sessions (one on Blog Networks, one on Open Notebook Science, and one on Social Networks for Scientists – yes those three are back to back…) and we will be aiming to be video casting, live blogging, posting slides, images, and comments; the whole deal. If you’ve got opinions then leave them at the various wiki pages (via the programme) or bring them along to the sessions. We are definitely looking for lively discussion. Two of these are being organised with the inimitable Deepak Singh who I am very much looking forward to finally meeting in person – along with many others I feel I know quite well but have never met – and others I have met and look forward to catching up with including Jean-Claude who has instigated the Open Notebook session.

With luck I will get to the dinner tomorrow night so hope to see some people there. Otherwise I hope to see many in person or online over the weekend. Thanks for Bora and Anton and David for superb organisation (and not a little pestering to make sure I decided to come!)

Related articles