Science Online London ran late last week and into the weekend and I was very pleased to be asked to run a panel, broadly speaking focused on evaluation and incentives. Now I had thought that the panel went pretty well but I’d be fibbing if I said I wasn’t a bit disappointed. Not disappointed with the panel members or what they said. Yes it was a bit more general than I had hoped and there were things that I wished we’d covered but the substance was good from my perspective. My disappointment was with the response from the audience, really on two slightly different points.
The first was the lack of response to what I thought were some of the most exciting things I’ve heard in a long time from major stakeholders. I’ll come back to that later. But a bigger disappointment was that people didn’t seem to connect the dots to their own needs and experiences.
Science Online, both in London and North Carolina forms, has for me always been a meeting where the conversation proceeds at a more sophisticated level than the usual. So I pitched the plan of the session at where I thought the level should be. Yes we needed to talk about the challenges and surface the usual problems, non-traditional research outputs and online outputs in particular don’t get the kind of credit that papers do, institutions struggle to give credit for work that doesn’t fit in a pigeonhole, funders seem to reward only the conventional and traditional, and people outside the ivory tower struggle to get either recognition or funding. These are known challenges, the question is how to tackle them.
The step beyond this is the hard one. It is easy to say that incentives need to change. But incentives don’t drop from heaven. Incentives are created within communities and they become meaningful when they are linked to the interests of stakeholders with resources. So the discussion wasn’t really about impact, or funding, or to show that nothing can be done by amateurs. The discussion was about the needs of institutions and funders and how they can be served by what is being generated by the online community. It was also about the constraints they face in acting. But fundamentally you had major players on the stage saying “this is the kind of thing we need to get the ball rollingâ€.
Make no mistake, this is tough. Everyone is constrained and resources are tight but at the centre of the discussion were the key pointers to how to cut through the knot. The head of strategy at a major research university stated that universities want to play a more diverse role, want to create more diverse scholarly outputs, and want to engage with the wider community in new ways. That smart institutions will be looking to diversify. The head of evaluation at a major UK funder said that funders really want to know about non-traditional outputs and how they were having a wider impact. That these outputs are amongst the best things they can talk about to government. That they will be crucial to make the case to sustain science funding.
Those statements are amongst the most direct and exciting I have heard in some years of advocacy in this space. The opportunity is there, if you’re willing to put the effort in to communicate and to shape what you are doing to match to match their needs. As Michael Nielsen said in his morning keynote this is a collective action problem. That means finding what unites the needs of those doing with the needs of those with resources. It means compromise, and it means focusing on the achievable, but the point of the discussion was to identify what might be achievable.
So mostly I was disappointed that the excitement I felt wasn’t mirrored in the audience. The discussion about incentives has to move on. Saying that “institutions should do X†or “funders should do Y†gets us nowhere. Understanding what we can do together with funders and institutions and other communities to take the online agenda forward and understanding what the constraints are is where we need to go. The discussion showed that both institutions and funders know that they need what the community of online scientists can do. They don’t know how to go about it, and they don’t even know very much what we are doing, but they want to know. And when they do know they can advise and help and they can bring resources to bear. Maybe not all the resources you would like, and maybe not for all the things you would like, but resources nonetheless.
With a lot of things it is easy to get too immersed in the detail of these issues and to forget that people are looking in from the outside without the same context. I guess the fact that I pulled out what might have seemed to the audience to be just asides as the main message is indicative of that. But I really want to get that message out because I think it  is critical if the community of online scientists wants to be the mainstream. And I think it should be.
The bottom line is that smart funders and smart institutions value what is going on online. They want to support it, they want to be seen to support it, but they’re not always sure how to go about it and how to judge its quality. But they want to know more. That’s where you come in and that’s why the session was relevant. Lars Fischer had it absolutely right: “I think the biggest and most consequential incentive for scientists is (informal) recognition by peers.†You know, we know, who is doing the good stuff and what is valuable. Take that conversation to the funders and the institutions, explain to them what’s good and why, and tell the story of what the value is. Put it in your CV, demand that promotion panels take account of it, whichever side of the table you are on. Show that you make an impact in language that they understand. They want to know. They may not always be able to act – funding is an issue – but they want to and they need your help. In many ways they need our help more than we need theirs. And if that isn’t an incentive then I don’t know what is.
Related articles
- Submission to the Royal Society Enquiry (cameronneylon.net)
- Recognising, Appreciating, Measuring and Evaluating the Impact of Open Science (ukwebfocus.wordpress.com)
- A new sustainability model: Major funders to support OA journal (cameronneylon.net)
- (S)low impact research and the importance of open in maximising re-use (cameronneylon.net)
- Some questions for British research policy (softmachines.org)
- Proceed with caution when setting up financial incentives for general practice doctors (eurekalert.org)
I also blogged on this: http://www.nature.com/scitable/blog/bioscience-elearning/solo11_science_online_london_2011_188651
I won’t re-hash my views here other than to say that I thought the panel did not work, it did not address the main question. For me the comments by Mari talked about ‘outputs, outcomes, and impact’ and yet these were never defined. How are these to be measured?
Hi Nick
Thanks for the comment and I’ll put a link in. I agree that we had to skip over some definitional stuff but we could have spent the whole hour on terminology and that didn’t seem worthwhile. I have spent days discussing what outputs, outcomes, and impact mean. I definitely take on board that those concepts didn’t get across though and I’ll accept the blame for not setting that out better.
I will note however that I took that absolute opposite meaning from Mari’s remarks (and from discussions I know I’m on more or less the right track) that the RCs are desperately interested in all sorts of other kinds of outputs. They don’t know what, where, how good these are and they want to know. That’s exactly my point. The audience had the opportunity to talk about what those are, how they might be measured but no-one seemed to want to take that opportunity. Believe me there is interest at the highest level in getting better informed on this if for no other reason than they make great case studies for documents sent to ministers…
Couldn’t be at solo, and don’t know if this was covered. But just to state the obvious: that participation in open science can lead to a traditional peer-reviewed publication, which represents a static report of a completed piece of work, and a useful summary of a project once it has reached a milestone. Authorship on this kind of output is thus an incentive to participate.
This does not answer other questions, such as how do we value research outputs that are not suitable for publication. But it does account for a fair amount of research that is done – we would expect valid research to be traditionally “publishable” even though it has been carried out openly. So from the point of view of the RCs, one could then say that a) there can be traditional outputs, but that b) perhaps (*all other things being equal*) one could treat research arising from an open process as preferable for a number of reasons we needn’t go through again here. i.e. publications arising from open research are particularly worthy of note.
Couldn’t be at solo, and don’t know if this was covered. But just to state the obvious: that participation in open science can lead to a traditional peer-reviewed publication, which represents a static report of a completed piece of work, and a useful summary of a project once it has reached a milestone. Authorship on this kind of output is thus an incentive to participate.
This does not answer other questions, such as how do we value research outputs that are not suitable for publication. But it does account for a fair amount of research that is done – we would expect valid research to be traditionally “publishable” even though it has been carried out openly. So from the point of view of the RCs, one could then say that a) there can be traditional outputs, but that b) perhaps (*all other things being equal*) one could treat research arising from an open process as preferable for a number of reasons we needn’t go through again here. i.e. publications arising from open research are particularly worthy of note.
I’m slowly catching up with the internet – sorry for the delayed response. I think it wasn’t the topic of the session that made some people uninterested, but the scheduling.Â
It would have been a great breakout session, attracting the people who wanted and needed this discussion at this level, but it didn’t quite work as an overall thing for everyone, as not all attendees had been thinking about incentives a lot (that day, or ever). Especially people who had gone to all of the science communication-type breakouts, this just felt very much like something that would have been part of some of the *other* tracks.Â
I *have* thought about incentives before – in fact I do so a lot – and even I had a hard time getting into the session simply because I had spent the day filling my head with thoughts about *other* topics. I’m sure that if I’d gone to a few other breakout sessions, the panel would have flowed naturally from that, but now it just felt like I was joining in the middle of something. Maybe if it had been scheduled right after Michael’s talk, at the start of the day, it would have been more on people’s minds (and then the arsenic story, which was far more generally accesible, could have been the closing thing.)
But of course, I know scheduling is challenging in and of itself, and I don’t blame the organisers for this, nor you for moderating. It just felt a bit out of place for people who had just come out of breakout sessions about storytelling and colour perception and whatnot. The variety of attendees and topics at science online, while amazing, comes with its own little problems.
Hi Eva
That’s an interesting thought and I hadn’t really considered it from that perspective. I can see how it would have been a big snap back from those sessions. I was in other more technical sessions for most of the afternoon (and the policy Question TIme session) so I was coming at it from a different set of experiences at that point. I think the same was probably true of most of the panellists (I think we were nearly all in either the Open Research Reports and/or Microsoft Academic Search sessions for instance).