Home » Blog, Featured

A collaborative proposal on research metrics

24 September 2010 3,151 views 41 Comments
Measuring time
Image by aussiegall via Flickr

tldr: Proposed project to connect metrics builders with those who can most effectively use them to change practice. Interested? Get involved! Proposal doc is here and free to edit.

When we talk about open research practice, more efficient research communication, wider diversity of publication we always come up against the same problem. What’s in it for the jobbing scientist? This is so prevalent that it has been reformulated as “Singh’s Law” (by analogy with Godwin’s law) that any discussion of research practice will inevitably end when someone brings up career advancement or tenure. The question is what do we actually do about this?

The obvious answer is to make these things matter. Research funders have the most power here in that they have the power to influence behaviour through how they distribute resources. If the funder says something is important then the research community will jump to it. The problem of course it that in practice funders have to take their community with them. Radical and rapid change is not usually possible. A step in the right direction would be to provide funders and researchers with effective means of measuring and comparing themselves and their outputs. In particular means of measuring performance in previously funded activities.

There are many current policy initiatives on trying to make these kinds of judgements. There are many technical groups building and discussing different types of metrics. Recently there have also been calls to ensure that the data that underlies these metrics is made available. But there is relatively little connection between these activities. There is an opportunity to connect technical expertise and data with the needs of funders, researchers, and perhaps even the mainstream media and government.

An opportunity has arisen for some funding to support a project here. My proposal is to bring a relevant group of stakeholders together; funders, technologists, scientists, adminstrators, media, publishers, and aggregators, to identify needs and then to actually build some things. Essentially the idea is a BarCamp style day and a bit meeting followed by a two day hackfest. Following on from this the project would fund some full time effort to take the most promising ideas forward.

I’m looking for interested parties. This will be somewhat UK centric just because of logistics and funding but the suggestion has already been made that following up with a similar North American or European project could be interesting. The proposal is available to view and edit as a GoogleDoc. Feel free to add your name, contact me directly, or suggest the names of others (probably better to me directly). I have a long list of people to contact directly as well but feel free to save me the effort.

Ed. Note: This proposal started as a question on Friendfeed where I’ve already got a lot of help and ideas. Hopefully soon I will write another post about collaborative and crowdsourced grant writing and how it has changed since the last time I tried this some years back.

Enhanced by Zemanta

1 Star2 Stars3 Stars4 Stars5 Stars (No Ratings Yet)
Loading ... Loading ...

  • http://friendfeed.com/kubke Kubke

    Wonder whether Andrew Treolar might have insight to contribute http://andrew.treloar.net/

    This comment was originally posted on FriendFeed

  • M J Weller

    Hi Cameron
    definitely count me in. I’m working on digital scholarship at the Open University and ‘making this stuff count’ is the biggest problem. I’m going to be looking at some form of metrics this year so would definitely be interested.
    Martin

  • http://ouseful.info Anonymous

    Happy to help hack stuff together if i can…

  • http://cameronneylon.net Cameron Neylon

    Very happy to have you both on board. Starting to get some traction now which is positive. Of course. I still need to actually get the money as well…

  • http://friendfeed.com/cameronneylon Cameron Neylon

    AT certainly has experience of the problem and also seems to be collecting a lot of data…

    This comment was originally posted on FriendFeed

  • http://friendfeed.com/kubke Kubke

    He was strong about making data sets a primary citable object at the data matters meeting where I met him, to make the ‘use’ of the data as trackable as any publication

    This comment was originally posted on FriendFeed

  • http://friendfeed.com/cameronneylon Cameron Neylon

    Yep, I think that’s a key theme. Need to make sure have Dryad and Datacite involvement with this.

    This comment was originally posted on FriendFeed

  • http://friendfeed.com/kubke Kubke

    He suggested giving the ‘data’ a DOI and tap onto the search/link/tracking that already exists for paper publications as something which could provide an initial viable solution

    This comment was originally posted on FriendFeed

  • http://friendfeed.com/cameronneylon Cameron Neylon

    Essentially this is what the DataCite project are doing. I think Dryad is connected with them as well. I worry about using dois for this because I would rather URLs [ducks in case geoff bilder arrives to pummel me (-; ] but the reality is that the scientific mindset seems to be doi = real so I think it’s going to be the way we move forward in practice.

    This comment was originally posted on FriendFeed

  • http://friendfeed.com/dsalo Dorothea

    dois have urls. what’s the problem, exactly?

    This comment was originally posted on FriendFeed

  • http://friendfeed.com/cameronneylon Cameron Neylon

    The redirection mainly. Essentially it breaks a whole set of fun things that you can do with URLs. None of which to be fair are very commonly done. Also in principle they could be enabled through the doi redirect. Essentially its a minor quibble that it doesn’t add anything technically and breaks some stuff. But the reasons for dois are fundamentally social not technical.

    This comment was originally posted on FriendFeed

  • http://friendfeed.com/dsalo Dorothea

    mmm, yes and no. There’s actually some quite keen stuff that can be done with the handle system that DOIs are based on. Nobody does any of it, unfortunately. I do think there’s one basic technosocial thing that makes DOIs/handles necessary: domains are constructed/named as branding tools, and brands change. That brands change should not be an excuse to 404 important stuff.

    This comment was originally posted on FriendFeed

  • http://friendfeed.com/dsalo Dorothea

    mmm, yes and no. There’s actually some quite keen stuff that can be done with the handle system that DOIs are based on. Nobody does any of it, unfortunately.

    This comment was originally posted on FriendFeed

  • http://friendfeed.com/mfenner Martin Fenner

    I’m a big DOI fan. Just yesterday I ran into a few duplicate papers in the Mendeley library. One reason is that there are several URLs associated with a paper: PubMed, Journal HTML page, Journal PDF, institutional repository URL,… Not to mention that many URLs break over time.

    This comment was originally posted on FriendFeed

  • http://friendfeed.com/mfenner Martin Fenner

    And DOIs would really help to automatically find science blog posts associated with a paper. The Journal of Neuroscience editorial about supplementary information in August has no DOI. Almost impossible to find all blog posts (there are many) that talk about this editorial.

    This comment was originally posted on FriendFeed

  • http://friendfeed.com/cameronneylon Cameron Neylon

    All true but I guess my problem is that fundamentally dois are a special system so that generic web tools (and yes particularly semantic web tools) will break over them and the info won’t get incorporated into the wider web. e.g. Martin’s problem could be solved by an owl:sameAs which would mean that it would be transparent to other web systems. But with dois someone has to build something special to link that in. I think my other objection is that it perpetuates the idea that nothing is real unless its a journal article. And real = journal article = doi

    This comment was originally posted on FriendFeed

  • http://friendfeed.com/cameronneylon Cameron Neylon

    None of which changes the reality. DOIs are dominant and are here to stay as far as I can tell so I just need to get over it and move on…

    This comment was originally posted on FriendFeed

  • http://friendfeed.com/dsalo Dorothea

    I think OAI-ORE might solve some of your problems, Cameron.

    This comment was originally posted on FriendFeed

  • http://friendfeed.com/joergkurtwegner joergkurtwegner

    Just added a comment and we need to do more than offering incentives for "re-use".

    This comment was originally posted on FriendFeed

  • http://friendfeed.com/billhooker Bill Hooker

    While I am a big fan of finer-grained measures of contribution (e.g. peer review), I am not convinced that the current proposal should be stretched to cover such measures. It seems to me that what is wanted here is a hard-nosed, outcome-focused project designed to answer hard questions about ROI. For that purpose, a close focus on re-use is probably optimal.

    This comment was originally posted on FriendFeed

  • http://friendfeed.com/researchremix Heather Piwowar

    Nice proposal. It currently focuses strongly on funder needs. Useful, perhaps necessary, but it means delayed and indirect rewards to the jobbing scientist. Would it make sense to work in a more direct focus on "jobbing scientist needs" too, with possible hacks of automated reuse-metric CVs and reuse-metricful tenure packages?

    This comment was originally posted on FriendFeed

  • http://friendfeed.com/cameronneylon Cameron Neylon

    Joerg – I commented on your comment back at the proposal. Two things really. My view is that the "measuring re-use" idea covers things like review. A good review needs to be "cited" in some way and that re-use measured and rewarded. I’m not quite sure where you’re going with the micropayments thing. A kind of micropayments has been tried by EPSRC in the UK where they pay £50 or something into departmental accounts for each on time grant review. At best this is an irrelevance as the amounts are too small to be useful and at worst an insult. My time costs out at around £200 an hour. Doing a proper review is somewhere betwen £500 and £1500. The problem to me is that micropayments just doesn’t cut it, these are macropayments. Now on the other hand if we got a small amount each time the review was used then I’d be intersted, but then we’re back to measuring re-use again aren’t we? Am I missing something here?

    This comment was originally posted on FriendFeed

  • http://friendfeed.com/cameronneylon Cameron Neylon

    Heather, good point. Although the response I’ve got suggests that people are interested enough in the possibility of getting credit for a more diverse range of things that thats enough at the moment. But yes, the CV is a very good place to realise some benefits. Need to talk to someone inside VIVO I think to see if there is some low hanging fruit there.

    This comment was originally posted on FriendFeed

  • http://friendfeed.com/joergkurtwegner joergkurtwegner

    @Bill , @Cameron – I understand your views and the ROI question is important. As long as we agree that there is some measure, recognition, and reward scheme I agree. So, a re-use needs to be tracked otherwise, we need to prevent ab-use of re-use.

    This comment was originally posted on FriendFeed

  • http://friendfeed.com/cameronneylon Cameron Neylon

    So if I’m reading you right your core point is that we shouldn’t assume that simply by measuring something that the rewards will flow from that. That it is important to consider explicit reward schemes, which might include payment or aggregated micropayment?

    This comment was originally posted on FriendFeed

  • http://friendfeed.com/cameronneylon Cameron Neylon

    Actually it strikes me that both Joerg and Heather are pointing in the same direction here. Keeping up the issue of direct benefits.

    This comment was originally posted on FriendFeed

  • Pingback: Quick Links | A Blog Around The Clock

  • Sergey Parinov

    It is interesitng that similar ideas cross one’s mind at the same time in different places. E.g. we in Russia are implementing, I think, similar approach. See an article in English on this topic “The electronic library: using technology to measure and support Open Science” at http://socionet.ru/publication.xml?h=repec:rus:mqijxk:25

    What do you think about our views on research metrics?

  • Pingback: Science in the Open » Blog Archive » Metrics and Money