Open Access week is a fitting time to be finalising a project on Open Data. About two years ago I started working with the Canadian development funder, the International Development Research Center, to look at the implementation of Open Data policy. This week the final report for that project is being published.

Everyone, it seems agrees that opening up research data is a good thing, at least in the abstract. While there are lots of good reasons for not making data open in specific cases, its hard to make a case against the idea that, in general, it would be good for more of the outputs of research to be more available. The traditional way to achieve this, at least for funded research projects is through funder policies requiring data management and data sharing.

But while there’s a steadily increasing number of funders requiring data management plans and data sharing there is less work asking whether they work. There’s a lot less work asking what might be an even harder question, what does it actually mean for them to work? For the past two years I’ve been working with the IDRC, a Canadian funder of development research to probe that question. In particular I’ve been looking at how policy, and support of policy can support the kind of cultural change that leads to good data management and data sharing being the default for researchers.

The reason the project focused this way was due to a striking early finding. My first task was to interview a set of international experts on research data management and sharing for their views on what worked and what did not. What emerged was a sharp divide between those who thought that Data Management Plans were the best way to get researchers thinking about RDM, and those who were worried that the requirement for DMPs was leading to a tick box “compliance culture”. There was a concern that this might actually be damaging the process of culture change. I wrote about this concern in the review written for the project, published in RIO Journal (where all the formal documentary outputs for the project can be found).

The main part of the project was focussed on seven case studies of IDRC-funded projects. We worked with the projects to examine the data that they were generating, and then to develop a Data Management Plan. Then we followed the challenges and issues that arose as the projects implemented those plans. IDRC funds development research done in developing countries. We had projects based in Vietnam, Niger, South Africa, Columbia Egypt and Brazil working on projects in areas from copyright to climate change, health issues to sexual harassment, ecology to economics.

When we looked at the projects we found any of the challenges you might expect for data sharing in places away traditional centers of research prestige. Lack of good network connectivity, limited capacity and resources. We saw serious issues arising out of the English-centric nature of not just research communication, but systems and infrastructures. And we saw that the funding of infrastructures was inconsistent, periodic and unstable.

But what we also saw were challenges that arose at the funder level. These were not restricted to IDRC, and not limited to development funding or even data sharing, but general issues of policy design and implementation for research funders. Funders work with many projects, and those projects have different contexts and issues. Often the relationship between funder and project is mediated through a single person, the Program Officer. This single person then has to be the conduit for all policy communication, implementation, and monitoring.

Its hardly news that the hard working people within funders have a lot on their plate. What I certainly hadn’t seen before was how this limited progress towards culture change amongst researchers. Within our pilot project we had a group of highly engaged program officers that were learning about data management as they went. New policies required them to become instant experts able to advise, and where necessary push, funded projects towards new practice. There are a lot of policy initiatives within funders at the moment, and the load means that not all of them get the attention they need.

This turns out to be a problem because of the message it can send. Culture change really involves change within both researchers, and the funder. Each on their own can change practice, but embedding that practice so it becomes a culture, of “that’s just how things are done” requires mutual reinforcement. It is not enough to say something is important, you have to show you care about it. Many policies say nice things, but without the backup, and the infrastructure to support it, the message that can come across is “we talk about this, but we don’t really care”.

I had a very engaged and sympathetic group of program officers and projects, but even for this group, data wasn’t always the highest priority. We had to plan to regularly reinforce, and provide support for, the message that data was important over the course of the pilot project. Across a whole funder the risk of balls being dropped is very high. This raises issues for policy design and implementation. In the final report, just published at RIO the following recommendations are made.

1. Policies serve at least two distinct functions. They serve as a) signals that a specific issue, in this case improvements in data management and sharing, are an issue that the policy maker takes seriously and b) as a means of creating interventions in the behaviour of those subject to the policy. These two aspects of policy intent can be in tension with each other. Policies in general have been designed with the intent of requiring and driving behaviour change. The aspect of signalling, and its role in supporting culture change has been taken less seriously. In the initial review it was noted that the aspirational signalling aspects of some policies were at odds with the content of their interventions.
2. The signalling function of a policy is important for culture change. Researchers and funder staff take note of the direction of travel. Articulation of aspirations, particularly when they align with existing sympathies and narratives is important. The details of policy design seem less important and can be antagonistic to cultural change where the aspirations are not matched by capacities provided by the policy maker.
3. Internal audiences are at least as important as external. Policy needs to be designed with a view as to how it can effect the desired cultural change within a funder as much as among researcher communities. How does the policy articulate the importance of its aspirational goals, does it give permission for individuals to act, how does it help create a strong culture that aligns with the policy goals? Perhaps most importantly does it provide the necessary internal levers to ensure that those charged with implementing policy can access the necessary resources, infrastructure and expertise?
4. The primary failure mode for policy is overreach. The most common issue with policy implementation is where demands are made that cannot be met or evaluated. Generally the concern is whether researchers have the capacities, resources or expertise to deliver on policy requirements. However the risks can be substantially higher where the shortfall is at the funder. The core of culture change is a sense that the aspirations underpinning the change are both important and shared amongst stakeholders. Where a grantee falls short, allowances can be made in a way that reinforces the importance of the direction of travel to the grantee. However when a funder fails to provide the support and infrastructures – including Program Officer time, skills and expertise – necessary for policy implementation, and most particularly when implementation is not evaluated, the message is sent that the issue is not in fact important. This in turn leads to instrumental behaviour and in turn a lack of, or even negative, cultural change.
5. The interventions required by policy must be properly resourced, continuous, and self-reinforcing. The second aspect of policy remains important. Interventions at the level of requirements and practice have a role to play, both as opportunities to engage – and therefore transmit culture – and as a means of driving best practice and therefore achieving successes for grantees from their behaviour change. For this goal to be achieved the interventions need to be embedded in an ongoing and reinforced narrative that engages both researchers and funder staff.

This being Open Access week it seems worth thinking about how we design policy, not just within funders but more generally so as to effect culture change, not just change behavior. What might we do to better design both policies and implementation to build a greater sense of a collective direction of travel, and how do we ensure that travel is monitored in a way that is sustainable and builds momentum rather than becoming a tick-box exercise?