Fantasy Science Funding: How do we get peer review of grant proposals to scale?
This post is both a follow up to last week’s post on the cost’s of peer review and a response to Duncan Hull‘s post of nine or so months ago proposing a game of “Fantasy Science Funding“. The game requires you to describe how you would distribute the funding of the BBSRC if you were a benign (or not so benign) dictator. The post and the discussion should be read bearing in mind my standard disclaimer.
Peer review is in crisis. Anyone who tells you otherwise either has their head in the sand or is trying to sell you something. Volumes are increasing, quality of review is decreasing. The willingness of scientists to take on refereeing is increasingly the major problem for those who commission it. This is a problem for peer reviewed publication but the problems for the reviewing of funding applications are far worse.
For grant review, the problems that are already evident in scholarly publishing, fundamentally the increasing volume, are exacerbated by the fact that success rates for grans are falling and that successful grants are increasingly in the hands of a smaller number of people in a smaller number of places. Regardless of whether you agree with this process of concentrating grant funding this creates a very significant perception problem. If the perception of your referees is that they have no chance of getting funding, why on earth should they referee
Is this really happening? Well in the UK chemistry community last year there was an outcry when two EPSRC grant rounds in a row had success rates of 10% or lower. Bear in mind this was the success rate of grants that made it to panel, i.e. it is an upper bound, assuming there weren’t any grants removed at an earlier stage. As you can imagine there was significant hand wringing and a lot of jumping up and down but what struck me was two statements I heard made. The first, was from someone who had sat on one of the panels, was that it “raises the question of whether it is worth our time to attend panel meetings”. The second was the suggestion that the chemistry community could threaten to unilaterally withdraw from EPSRC peer review. These sentiments are now being repeated on UK mailing lists in response to EPSRC’s most recent changes to grant submission guidelines. Whether serious or not, credible or not, this shows that the compact of community contribution to the review process is perilously close to breaking down.
The research council response to this is to attempt to reduce the number of grant proposals, generally by threatening to block those who have a record of serial rejection. This will fail. With success rates as low as they are, and with successful grants concentrated in the hands of the few, most academics are serial failures. The only way departments can increase income is by increasing the volume and quality of grant applications. With little effective control over quality the focus will necessarily be on increasing volume. The only way research councils will control this is either by making applications a direct cost to departments, or by reducing the need of academics to apply.
The cost of refereeing is enormous and largely hidden. But it pales into insignificance compared to the cost of applying for grants. Low success rates make the application process an immense waste of departmental resources. The approximate average cost of running a UK academic for a year is £100,000. If you assume that each academic writes one grant per year and that this takes around two weeks full time work that amounts to ~£4k per academic per year. If there are 100,000 academics in the UK this is £400M, which with a 20% success rate means that £320M is lost in the UK each year. Let’s say that £100M is a reasonable ballpark figure.
In more direct terms this means that academics who are selected for their ability to do research, are being taken away from what they are good at to play a game which they will on average lose four times out of five. It would be a much more effective use of government funding to have those people actually doing research.
So this is a game of Fantasy Funding, how would I spend the money? Well, rather than discuss my biases about what science is important, which are probably not very interesting, it is perhaps more useful to think about how the system might be changed to reduce these wastages. And there is a simple, if somewhat radical, way of doing this.
Cut the budget in two and distribute half of it directly to academics on a pro-rata basis.
By letting researcher’s focus on getting on with research you will reduce their need for funding and reduce the burden. By setting the bar naturally higher for funding research you still maintain the perception that everyone is in with a chance and reduce the risk of referee drop out due to dis-enchantment with the process. More importantly you enable innovative research by allowing it to keep ticking over and in particular you enable a new type of peer review.
If you look at the amounts of money involved, say a few hundred million pounds for BBSRC, and divide that up amongst all bioscience academics, you end up with figures of a £5-20K per academic per year. Not enough to hire a postdoc, just about enough to run a PhD student (at least at UK rates). But what if you put that together with the money from a few other academics? If you can convince your peers that you have an interesting and fun idea then they can pool funds together. Or perhaps share a technician between two groups so that you don’t lose the entire group memory every time a student leaves? Effective collaboration will lead to a win on all sides.
If these arguments sound familiar it is because they are not so different to the notion of 20% time, best known as a Google policy of having all staff spend some time on personal projects. By supporting low level innovation and enabling small scale judging of ideas and pooling of resources it is possible to enable bottom up innovation of precisely the kind that is stifled by top down peer review.
No doubt there would be many unintended consequences, and probably a lot of wastage, but in amongst that I wouldn’t bet against the occassional brilliant innovation which is virtually impossible in the current climate.
What is clear is that doing nothing is not an option. Look at that EPSRC statement again. People with a long term success rate below 25% will be blocked…I just checked my success rate over the past ten years (about 15% by numbers of grants, 70% by value but that is dominated by one large grant). The current success rate at chemistry panel is around 15%. And that is skewed towards a limited number of people and places.
The system of peer review relies absolutely on the communities agreement to contribute and some level of faith in the outcome. It relies absolutely on trust. That trust is perilously close to a breakdown.
Well, I saw the statement today and was, probably best put, stunned. Especially the part where they ban you from applying for anything for the year – even as Co-Investigator, which to me sounds like it would be an excellent way to mentor (given as, when I have been co-investigator, it means writing the grant anyway :) The language was also outright hostile and I was surprised that the announcement was allowed to be issued in this way.
Aside from that, as someone who has several projects that run off meagre funds, £5-20K, non-project specific, would be fantastic! I was already engaging my fantasies about the chemicals and computers I could buy with impunity, the unbudgeted conferences I and my students could attend, the collaborative visits that I wouldn’t have to pay for out of my own pocket because it wasn’t related to the currently funded project! The collaborative potential to act as instant seed corn without having to pass through local funding committees. The minimisation of hassle! Being able to get a summer student when good ones appear without having to spend hours writing grants for £2K. *drool* And the collaborative solutions sounded even better (although better obviously the larger the department)! No doubt the Universities would find some way to swallow it in space charging or instrument fees …
Well, I saw the statement today and was, probably best put, stunned. Especially the part where they ban you from applying for anything for the year – even as Co-Investigator, which to me sounds like it would be an excellent way to mentor (given as, when I have been co-investigator, it means writing the grant anyway :) The language was also outright hostile and I was surprised that the announcement was allowed to be issued in this way.
Aside from that, as someone who has several projects that run off meagre funds, £5-20K, non-project specific, would be fantastic! I was already engaging my fantasies about the chemicals and computers I could buy with impunity, the unbudgeted conferences I and my students could attend, the collaborative visits that I wouldn’t have to pay for out of my own pocket because it wasn’t related to the currently funded project! The collaborative potential to act as instant seed corn without having to pass through local funding committees. The minimisation of hassle! Being able to get a summer student when good ones appear without having to spend hours writing grants for £2K. *drool* And the collaborative solutions sounded even better (although better obviously the larger the department)! No doubt the Universities would find some way to swallow it in space charging or instrument fees …
A fascinating post. I was particularly interested in the following excerpt:
There has recently been a proposal for academics to refuse to peer review the new (and entirely flawed) two page impact summary statement that EPSRC is imposing from April 21 2009. See Scientists call for a revolt against grant rule they claim will end blue skies research . (I should point out that I’m one of the signatories of the letter). There’s also an excellent blog post by David Colquhoun (UCL) ( How to get good science: again ) which describes the background to our call for a “modest revolt” and provides some links to follow-up comments (including a post from the head of RCUK’s Strategy Unit, Steven Hill).
I’ve certainly left EPSRC panel meetings in the past thinking that, in the majority of cases, we should have simply thrown the proposals up in the air and ranked them on the basis of where they fell on the floor. Indeed, David Delpy (Chief Executive of EPSRC) admitted in an Open Meeting in Nottingham last year that the peer review process is a “lottery”. We all know this but it was, errmm, “interesting” to hear it stated so bluntly by an EPSRC representative.
As you say, the peer review system is very close to breakdown.
In response to Anna: EPSRC specifically state that the year-long ban is on the basis of grants submitted only as PI (not Co-I). I’ll agree, however, that the mentoring idea is poorly thought out and offensive. In my experience, the majority of proposals fail not because they are poorly written or of poor quality but due to minor quibbles from referees and the “sociology” of the panel (where one vocal individual can make a big difference to the progress of a grant – everything rests on who is speaking to a given proposal). I was lucky – and I mean lucky! – to be awarded an EPSRC fellowship last year. Six weeks before the award of the fellowship grant, a responsive mode proposal I submitted came third from bottom of the ranked list. Was there a vast difference in quality of those two proposals? Of course not. As David Delpy says, it’s a lottery.
Best wishes,
Philip
A fascinating post. I was particularly interested in the following excerpt:
There has recently been a proposal for academics to refuse to peer review the new (and entirely flawed) two page impact summary statement that EPSRC is imposing from April 21 2009. See Scientists call for a revolt against grant rule they claim will end blue skies research . (I should point out that I’m one of the signatories of the letter). There’s also an excellent blog post by David Colquhoun (UCL) ( How to get good science: again ) which describes the background to our call for a “modest revolt” and provides some links to follow-up comments (including a post from the head of RCUK’s Strategy Unit, Steven Hill).
I’ve certainly left EPSRC panel meetings in the past thinking that, in the majority of cases, we should have simply thrown the proposals up in the air and ranked them on the basis of where they fell on the floor. Indeed, David Delpy (Chief Executive of EPSRC) admitted in an Open Meeting in Nottingham last year that the peer review process is a “lottery”. We all know this but it was, errmm, “interesting” to hear it stated so bluntly by an EPSRC representative.
As you say, the peer review system is very close to breakdown.
In response to Anna: EPSRC specifically state that the year-long ban is on the basis of grants submitted only as PI (not Co-I). I’ll agree, however, that the mentoring idea is poorly thought out and offensive. In my experience, the majority of proposals fail not because they are poorly written or of poor quality but due to minor quibbles from referees and the “sociology” of the panel (where one vocal individual can make a big difference to the progress of a grant – everything rests on who is speaking to a given proposal). I was lucky – and I mean lucky! – to be awarded an EPSRC fellowship last year. Six weeks before the award of the fellowship grant, a responsive mode proposal I submitted came third from bottom of the ranked list. Was there a vast difference in quality of those two proposals? Of course not. As David Delpy says, it’s a lottery.
Best wishes,
Philip
Hi Philip,
Just to clarify, I was referring to the fact that you can’t subsequently apply as CoI during the ban (rather than being taken down while in CoI status). Thanks also for the UCL blog post link :)
To be fair, from the select committee interview last year, I had the impression that it was not just a lottery, but a deliberately weighted lottery – we certainly have had RCUK representatives openly telling us not only about funding localisation, but giving the impression that this was a good thing.
Hi Philip,
Just to clarify, I was referring to the fact that you can’t subsequently apply as CoI during the ban (rather than being taken down while in CoI status). Thanks also for the UCL blog post link :)
To be fair, from the select committee interview last year, I had the impression that it was not just a lottery, but a deliberately weighted lottery – we certainly have had RCUK representatives openly telling us not only about funding localisation, but giving the impression that this was a good thing.
Philip, thanks for the comments and the pointer to both the THES piece and David’s blog post. I should have mentioned the THES thing. I guess the reason I didn’t is because in that matter I disagree in some ways with the principle of your objection, of not the practice.
What do I mean by that? Well primarily that I think we should be focussing our attention on return on investment and economic impact. If we can’t make a case on these grounds then perhaps the money should go into schools or hospitals instead. However I think that we can make a very strong economic impact as well as ROI case for blue skies research, indeed it may well be the case that the EI and ROI are higher for blue skies than applied research.
Two reasons for this. One is that the vast majority of the economic impact of research is through training and attracting a skilled workforce to the UK. This is much greater than any spin out activity, and I beleive that government is starting to see this as a major component of what we do. That and the occasional break throughs have a much bigger EI than the small and irregular contribution of applied projects.
On the ROI side (science output per pound spent) then I think it is very clear that less commercial research produces more accessible science outputs more quickly (because they are in fact published) and that they also provide people who are trained to be more flexible and adataptable in research settings.
All that said, two pages of nonsense to add to a proposal is a waste of everyone’s time. It will in most cases be unmitigated rubbish and will not be seriously refereed, even if people do agree to look at it. The proposals as they stand are a complete joke. So, given my strong disagreement with the rest of the changes I am happy to support the disagreement with this part of them.
My only caveat is that I think scientists should engage and educate themselves on the arguments around EI and ROI because at the end of the day we do have to justify our existence. But this is a broader issue than cen be dealt with by a box in a form.
Philip, thanks for the comments and the pointer to both the THES piece and David’s blog post. I should have mentioned the THES thing. I guess the reason I didn’t is because in that matter I disagree in some ways with the principle of your objection, of not the practice.
What do I mean by that? Well primarily that I think we should be focussing our attention on return on investment and economic impact. If we can’t make a case on these grounds then perhaps the money should go into schools or hospitals instead. However I think that we can make a very strong economic impact as well as ROI case for blue skies research, indeed it may well be the case that the EI and ROI are higher for blue skies than applied research.
Two reasons for this. One is that the vast majority of the economic impact of research is through training and attracting a skilled workforce to the UK. This is much greater than any spin out activity, and I beleive that government is starting to see this as a major component of what we do. That and the occasional break throughs have a much bigger EI than the small and irregular contribution of applied projects.
On the ROI side (science output per pound spent) then I think it is very clear that less commercial research produces more accessible science outputs more quickly (because they are in fact published) and that they also provide people who are trained to be more flexible and adataptable in research settings.
All that said, two pages of nonsense to add to a proposal is a waste of everyone’s time. It will in most cases be unmitigated rubbish and will not be seriously refereed, even if people do agree to look at it. The proposals as they stand are a complete joke. So, given my strong disagreement with the rest of the changes I am happy to support the disagreement with this part of them.
My only caveat is that I think scientists should engage and educate themselves on the arguments around EI and ROI because at the end of the day we do have to justify our existence. But this is a broader issue than cen be dealt with by a box in a form.
Cameron,
Thanks for responding. You might be surprised to hear that I am actually in broad agreement with your views. It’s good to see your well-argued position on this extremely important issue – it’s a great shame that the upper echelons of RCUK and DIUS can’t manage to muster comparably strong arguments.
The key points are as follows:
(i) Yes, the “human capital” academia produces is absolutely key to the UK economy and far outstrips any spin-out activity in terms of contribution to GDP. A number of commentators have previously (and regularly) pointed this out. I also discussed this in some detail at the Prometheus blog a few weeks ago. But, as I also mention there, how are we supposed to quantify/predict the economic impact of that “human capital”??
(ii) As you say, the economic impact summary element of the proposal will be “unmitigated rubbish”. You will hear many in the academic community say something along the lines of “Well, it’s only another two pages of bullshit we’ll have to produce. That won’t be too difficult”. Any academic with any experience of grant writing will be able to churn out the vacuous nonsense that RCUK will expect. For the reasons given by Glazer at the Prometheus blog, however, this is fundamentally dishonest and is an abuse of public funding.
(iii) Your final point: I entirely agree with you that publicly-funded academics must engage with the return on investment/economic impact arguments. The difficulty, of course, is that DIUS and RCUK continually fail to work those arguments through for themselves! As is argued in Nature Nanotech. 3 60 (2008) (free version of the paper here ), it’s very important to consider the fundamental economic rationale for state funding of science (i.e. public good -> market failure -> state investment). I’ll not rehearse the arguments here – they’re covered at length in the comments at the Prometheus blog linked to above.
—-Anna—-
Sorry, misinterpreted your comment. Yes, EPSRC is indeed going to prohibit those who are on the “naughty list” (or, as colleague in Nottingham has called it, the “Grants Gulag”) from applying as PI or Co-I for a year.
John Denham has recently argued that, regardless of the outcome of the recent RAE (which identified “pockets of excellence” right across the academic sector), there must be a focus on the elite research groups. RCUK does what DIUS tells it to do, either implictly (“we need to win as much money in the next CSR”) or, increasingly, explicitly. So, if John Denham thinks that funding localisation is a good thing, so too will Philip Esler, David Delpy et al….
Best wishes,
Philip
Cameron,
Thanks for responding. You might be surprised to hear that I am actually in broad agreement with your views. It’s good to see your well-argued position on this extremely important issue – it’s a great shame that the upper echelons of RCUK and DIUS can’t manage to muster comparably strong arguments.
The key points are as follows:
(i) Yes, the “human capital” academia produces is absolutely key to the UK economy and far outstrips any spin-out activity in terms of contribution to GDP. A number of commentators have previously (and regularly) pointed this out. I also discussed this in some detail at the Prometheus blog a few weeks ago. But, as I also mention there, how are we supposed to quantify/predict the economic impact of that “human capital”??
(ii) As you say, the economic impact summary element of the proposal will be “unmitigated rubbish”. You will hear many in the academic community say something along the lines of “Well, it’s only another two pages of bullshit we’ll have to produce. That won’t be too difficult”. Any academic with any experience of grant writing will be able to churn out the vacuous nonsense that RCUK will expect. For the reasons given by Glazer at the Prometheus blog, however, this is fundamentally dishonest and is an abuse of public funding.
(iii) Your final point: I entirely agree with you that publicly-funded academics must engage with the return on investment/economic impact arguments. The difficulty, of course, is that DIUS and RCUK continually fail to work those arguments through for themselves! As is argued in Nature Nanotech. 3 60 (2008) (free version of the paper here ), it’s very important to consider the fundamental economic rationale for state funding of science (i.e. public good -> market failure -> state investment). I’ll not rehearse the arguments here – they’re covered at length in the comments at the Prometheus blog linked to above.
—-Anna—-
Sorry, misinterpreted your comment. Yes, EPSRC is indeed going to prohibit those who are on the “naughty list” (or, as colleague in Nottingham has called it, the “Grants Gulag”) from applying as PI or Co-I for a year.
John Denham has recently argued that, regardless of the outcome of the recent RAE (which identified “pockets of excellence” right across the academic sector), there must be a focus on the elite research groups. RCUK does what DIUS tells it to do, either implictly (“we need to win as much money in the next CSR”) or, increasingly, explicitly. So, if John Denham thinks that funding localisation is a good thing, so too will Philip Esler, David Delpy et al….
Best wishes,
Philip
Hi Cameron, there is an interesting followup to your piece above in Times Higher Education this week: See Peer review teeters as experts struggle with burden of work
Hi Cameron, there is an interesting followup to your piece above in Times Higher Education this week: See Peer review teeters as experts struggle with burden of work
License
I am also found at...
Tags
Recent posts
Recent Posts
Most Commented