Image via Wikipedia
Once again a range of conversations in different places have collided in my feed reader. Over on Nature Networks, Martin Fenner posted on Researcher ID which lead to a discussion about attribution and in particular Martin’s comment that there was a need to be able to link to comments and the necessity of timestamps. Then DrugMonkey posted a thoughtful blog about the issue of funding body staff introducing ideas from unsuccessful grant proposals they have handled to projects which they have a responsibility in guiding.
It stuck me that at the core of all these issues is a problem of recognising a contribuion that is not in for the form of a peer reviewed paper. We assume that having a good idea taken from an unsuccessful proposal and used by someone else is a bad thing. But as long as there is clear attribution, and the rules on how the information is transferred is clear this has a clear potential to increase value for money for the funder, get the results quicker, and get more credit for the unsuccessful applicant. The key is appropiate recognition of the idea. This could be as an authorship on resulting papers but this may not be appropriate in some cases. In such cases the best route would be to cite and recognise the original document in which the idea was proposed; the grant proposal.
If we could protect the right of the original proposer to be identified with their idea then we take a number of steps forward. First we recognise the idea itself as a valid contribution to the scientific effort and start to break down the complete reliance on published papers for assigning academic credit. How do I show I’ve got good ideas? Well lots of people cite my proposals, even if they are not successful. The discussion in the comments following DrugMonkey’s post is quite instructive (slighlty edited by me).
Physio Prof: If you are implying that a PO would take an experiment or hypothesis directly out of an R01 and tell a U01 awardee to perform or test it, respectively, I would be careful about making this kind of claim of grossly unethical behavior without adverting any supporting facts.
DrugMonkey: It is not clear to me that this is “grossly unethical behavior”. That’s part of my point here. Is this “unethical”? To whom and why? The taxpayer is benefited best by the experiments being done quicker and better. Is she not? So even if the ICs do have rules preventing the best possible route to the data because of “ownership” it is not entirely clear to me that they should…
PhysioProf: It’s clear to me that it is [unethical …]. It is wrong to the extent that it involves confidential information being used for purposes other than the purpose pursuant to which the confidential information was provided by its originator.
I think this is conflating two issues. One is the question of ‘what the information was provided for’, i..e. what are the expectations and rules under which the proposal was submitted. The other is how can the funding body make the best and most effective use of that information to fulfil its mandate.
I would submit that it would be possible for any funding body to require, as a condition of submission, that the information be available for re-use by that funding body in whatever way delivers the most effective science. It would not be unethical for a funder to impose such conditions. What would but unethical would be for such information to be used without proper attribution and credit being given. The problem with giving such credit is that the proposal is not a citeable document. Because these are confidential documents that have no public existence there is no way to give credit for good ideas that didn’t quite make the grade. The inability of the academic credit system to recognise this kind of contribution arises directly from the fact that these documents cannot be cited. Now how this could be done I don’t know. Noam Harel has proposed a database for unsuccessful applications and has taken this forward. I don’t think the world is ready yet for a database where people make their unsuccesful grant applications publically available but it raises the key question of what constitutes a citeable document. Should the existence and some index of unsuccessful proposals be a part of the public record. Even if they are not public in their entirety an ability to point at a citation might encourage appropriate re-use of ideas.
The discussion with Martin Fenner arose where he commented that it would be useful to be able to cite the comments on a blog post. On some blogs you can, on others you can but its not obvious how, and on many you can’t. If a comment on a blog, or on an online paper, kicks of a set of ideas or new experiments, surely that comment needs to be cited and to be citeable. Many journals remain uncomfortable about citing web pages so the debate over what consistutes a citeable document needs to be had. A DOI for every comment and document would be great but prohibitively expensive. WebCite provides a mechanism for improving the permanence of web pages but is intended primarily for citation in the traditional literature than for general citation of weblinks. A very encouraging aspect of this is the move by the PLoS Journals to make the comments on published papers citeable at least in the first instance through a URI.
As an aside it is no accident that this arose in a discussion about identity either. Identity is a key aspect of attribution and maintaing a consistent identity is critical for building up credit. The problem of identity, however, seems to be one that is being gradually solved with the wider takeup of OpenID. The recent advance of being able to use your own domain to host an OpenID to me means that as long as adoption increases on its current trajectory the problem of maintaining a consistent identity is large solved.
The majority of arguments against Open Science relate to a lack of attribution. Whether they are getting scooped or having your ideas stolen the central objections appear to me to be about people note getting proper credit for their work. One key to proper attribution is giving people something to cite. Without that it is very difficult. This again is a strong argument for open notebooks, open documents, and good practice in maintaining them. If you’re experiment, or your blog post, has a solid URI then someone can cite it (or at least try to). It is technically relatively straightforward to aggregate and count those citations and therefore assign credit where it is due. Noam Harel has been discussing a new type of metric for academic credit. At some level, I think this issue goes away as long as you can track people’s contribution properly. That doesn’t meant that we shouldn’t discuss these issues but at the centreof the problem lies the issue of who a person is, and what all of their contributions are. Different contributions have different values but if we can’t pull them all together then we can’t even make a start.
None of this solves the problem of people claiming they had the idea/did the experiment independently, and in many cases this may in fact be true. But if the argument is once of precedence then this problem goes away. The person with the idea gets the credit for having it first. The person who did the experiment and published gets the credit for doing that first. If the information flows properly then the claim that the experiment was done with no knowledge of the idea becomes increasingly implausible. Nonetheless there is a big outstanding issue. How can we create legally enforceable licences that encourage re-use of data but require proper attribution. Strong moral pressure for proper attribution will take us some of the way but the constantly re-iterated fears of many scientists that they are being ripped off is a good indicator that stronger protections are needed.
I agree we need stronger modes of attribution – but we need to couple this with ‘academic reward’. If my unsuccessful grant is used for another grant – the second person gets the money. And while I might not mind as I get credit and see my idea come to fruition, my university gets no money. And when they run out, they sack me.
The other issue is recognition for collaborative contribution. My experience of this arises from a colleague who insists that, as of late last year, I only have one publication. At least, to him, only one which counts – ie with me as first and undisputed author. This ignores all contributions to any other work for which I might not have had the leading/authorship role. I found the whole thing quite repellant – but so long as such attitudes still exist, we still are fighting an uphill battle :(
I agree we need stronger modes of attribution – but we need to couple this with ‘academic reward’. If my unsuccessful grant is used for another grant – the second person gets the money. And while I might not mind as I get credit and see my idea come to fruition, my university gets no money. And when they run out, they sack me.
The other issue is recognition for collaborative contribution. My experience of this arises from a colleague who insists that, as of late last year, I only have one publication. At least, to him, only one which counts – ie with me as first and undisputed author. This ignores all contributions to any other work for which I might not have had the leading/authorship role. I found the whole thing quite repellant – but so long as such attitudes still exist, we still are fighting an uphill battle :(
Yes that’s fair. Its not enough to get credit, there needs to be reward as well. And departments need money. I think this ties in with the area of paying for results rather than upfront for proposals. And then notionally the money flows back down the chains of attribution. That’s pretty fanciful in practise I know but you can see conceptually how it might work.
And yes, the notion of collaboration, especially the concept that the contribution might be in bringing the right people together. Its not the way research is currently structured. I got asked the same question at one stage in a different form ‘what is the signature of your group?’. Again we need to change the range of things that get credited. After all if the RAE is going to based on metrics all we need to do is change the metric…
Yes that’s fair. Its not enough to get credit, there needs to be reward as well. And departments need money. I think this ties in with the area of paying for results rather than upfront for proposals. And then notionally the money flows back down the chains of attribution. That’s pretty fanciful in practise I know but you can see conceptually how it might work.
And yes, the notion of collaboration, especially the concept that the contribution might be in bringing the right people together. Its not the way research is currently structured. I got asked the same question at one stage in a different form ‘what is the signature of your group?’. Again we need to change the range of things that get credited. After all if the RAE is going to based on metrics all we need to do is change the metric…
Another eloquent argument Cam. Attribution for contributions of all sorts are necessary – collaborating on experiments, helping design the research plan in the first place, replicating important new results from another lab, teaching, reviewing grants/papers, etc. Thanks for referring to my preliminary, much less articulate attempts to come up with an objective/comprehensive rating system.
One point of clarification regarding the hypothetical Centralized Grant Proposal Repository – it is NOT meant to be an archive for ‘unsuccessful’ grants – it is meant to be an exchange between scientists requesting funding and agencies granting funding. It is essentially analogous to ‘universal’ applications for college, professional schools, or residencies – it would just save an enormous amount of time and paper. And it would NOT have to be ‘public’ (at least initially, to allay concerns of all the close-minded scientists out there). I think Google or some analogous organization would be perfect to host such a repository…
Another eloquent argument Cam. Attribution for contributions of all sorts are necessary – collaborating on experiments, helping design the research plan in the first place, replicating important new results from another lab, teaching, reviewing grants/papers, etc. Thanks for referring to my preliminary, much less articulate attempts to come up with an objective/comprehensive rating system.
One point of clarification regarding the hypothetical Centralized Grant Proposal Repository – it is NOT meant to be an archive for ‘unsuccessful’ grants – it is meant to be an exchange between scientists requesting funding and agencies granting funding. It is essentially analogous to ‘universal’ applications for college, professional schools, or residencies – it would just save an enormous amount of time and paper. And it would NOT have to be ‘public’ (at least initially, to allay concerns of all the close-minded scientists out there). I think Google or some analogous organization would be perfect to host such a repository…
When Nature Precedings accepted proposals it was a nice way to give your proposal a DOI and proper author list.
Still I don’t think citing a blog post or wiki page will be a problem in papers if it is truly the most relevant document to cite.
When Nature Precedings accepted proposals it was a nice way to give your proposal a DOI and proper author list.
Still I don’t think citing a blog post or wiki page will be a problem in papers if it is truly the most relevant document to cite.
Noam – yes I wasn’t entirely clear there. I have had a quick look at the site (not in depth yet!) and I saw some of the comments and discussion re:public and how it works.
My point was that making a (unsuccessful) grant application citeable requires it to be somewhere, even if it is not fully visible. I see your efforts as a first step along this road but I am unsure what the next step should (or can) be. JC’s point about Precedings is a good one, but unfortunately they don’t seem to want to take proposals at the moment. Equally I can publish my proposals (and Rosie Redfield does this regularly) but that isn’t solving the generic problem.
WRT rating systems my feeling is that you need different systems for different purposes and any comprehensive system will be flawed for some uses. The interesting question I guess is whether the benefits of having a single system outweigh the problems associated with any particular single system?
Jean-Claude – been talking to a few editorial staff recently and they still seem to be uncomfortable (but not dismissive) with citing web pages. Certainly some referees are. The central concerns, as always, are stability, so WebCite is certainly at the very least a part answer to that. But we will certainly be doing it extensively in our next paper so we’ll see how that works out.
Noam – yes I wasn’t entirely clear there. I have had a quick look at the site (not in depth yet!) and I saw some of the comments and discussion re:public and how it works.
My point was that making a (unsuccessful) grant application citeable requires it to be somewhere, even if it is not fully visible. I see your efforts as a first step along this road but I am unsure what the next step should (or can) be. JC’s point about Precedings is a good one, but unfortunately they don’t seem to want to take proposals at the moment. Equally I can publish my proposals (and Rosie Redfield does this regularly) but that isn’t solving the generic problem.
WRT rating systems my feeling is that you need different systems for different purposes and any comprehensive system will be flawed for some uses. The interesting question I guess is whether the benefits of having a single system outweigh the problems associated with any particular single system?
Jean-Claude – been talking to a few editorial staff recently and they still seem to be uncomfortable (but not dismissive) with citing web pages. Certainly some referees are. The central concerns, as always, are stability, so WebCite is certainly at the very least a part answer to that. But we will certainly be doing it extensively in our next paper so we’ll see how that works out.
Just wanted to alert you to a very well-reported article (and accompanying editorial) about a recent scandal in structural chemistry in which a prominent PI (Homme Hellinga) accused his student of fraud after another lab spent significant time and money showing that two papers’ conclusions were wrong.
(Link: http://www.nature.com/news/2008/080514/full/453275a.html)
Both the article and the editorial mention that the independent PI who refuted Hellinga’s findings did so at “no discernable benefit to his own career”.
This is a perfect example of the perverse incentives that encourage labs to publish work first, often with data that is shoddy at best, fraudulent at worst.
One way to improve these perverse incentives is to give more credit to those who reproduce or refute others’ findings. Though there will never be a perfect formula, we should strive/collaborate to come up with the best system possible to gauge an individual’s overall contribution to science.
Just wanted to alert you to a very well-reported article (and accompanying editorial) about a recent scandal in structural chemistry in which a prominent PI (Homme Hellinga) accused his student of fraud after another lab spent significant time and money showing that two papers’ conclusions were wrong.
(Link: http://www.nature.com/news/2008/080514/full/453275a.html)
Both the article and the editorial mention that the independent PI who refuted Hellinga’s findings did so at “no discernable benefit to his own career”.
This is a perfect example of the perverse incentives that encourage labs to publish work first, often with data that is shoddy at best, fraudulent at worst.
One way to improve these perverse incentives is to give more credit to those who reproduce or refute others’ findings. Though there will never be a perfect formula, we should strive/collaborate to come up with the best system possible to gauge an individual’s overall contribution to science.