Category: Featured

Posted on

PolEcon of OA Publishing I: What is it publishers do anyway?

English: Peer review system Polski: System rec...
Peer review system…in Polish (Photo credit: Wikipedia)

A note on changes: I’m going to vary my usual practice in this series and post things in a rawer form with the intention of incorporating feedback and comments over time. In the longer term I will aim to post the series in a “completed” form in one way or another as a resource. If there is interest then it might be possible to turn it into a book.

There is no statement more calculated to make a publisher’s blood boil than “Publishers? They just organise peer review” or perhaps “…there’s nothing publishers do that couldn’t be done cheaper and easier by academics”. By the same token there is little that annoys publishing reform activists, or even most academics, more than seeing a huge list of the supposed “services” offered by publishers, most of which seem unfamiliar at best and totally unnecessary, or even counter productive at worst.

Much of the disagreement over what scholarly publishing should cost therefore turns on a lack of understanding on both sides. Authors are unaware of much of what publishing actually involves in practice, and in particular how the need for safeguards is changing. Publishers, steeped in the world of how things have been done tend to be unaware of just how ridiculous the process looks from the outside and in defending the whole process are slow, or in some cases actively antagonistic, to opening up a conversation with authors about which steps are really necessary or wanted, and whether or not anything can be easily taken away (or equally added to the mix).

And both sides fail to really understand the risks and costs that the other sees in change.

So what is it that publisher do in fact do? And why? Publishers manage some sort of system for submitting manuscripts, manage the peer review process, and then convert the manuscript into “the published version”. This last part of the process, which involves the generation of versions in specific formats, getting DOIs registered and submitting metadata, as well as providing the web platform for actually hosting articles, is a good place to start.

Production and Publication

This process has plenty of bits that look as though they should be cheap. Getting identifiers (around $1 an article) and hosting (surely next to nothing?) look as though they should be cheap and at one level they are. The real cost in minting a DOI however is not the charge from Crossref but the cost of managing the process. Some publishers are very good at managing this (Elsevier have an excellent and efficient data pipeline for instance) while small publishers tend to struggle because they manage it manually. Hosting also has complications; the community expects high availability and rapid downloads, and this is not something that can be done on the cheap. High quality archiving to preserve access to content in the long term is also an issue.

Nonetheless this part of the process need not be the most expensive. Good systems at scale can make these final parts of the publishing process pretty cheap. The more expensive part is format conversion and preparation for publication. It is easy to say (and many people do) that surely this should be automated and highly efficient. It is easy to say because its true. It should. But “should” is a slippery word.

If documents came into this process in a standardised form it would be easy. But they don’t. And they don’t on a large scale. One of the key elements we will see over and over again in this series is that scholarly publishing often doesn’t achieve economics of scale because its inputs are messy and heterogeneous. Depending on the level of finish that a publisher wants to provide, the level of cleanup required can differ drastically. Often any specific problem is a matter of a few minutes work, but every fix is subtly different.

This is the reason for the sometimes seemingly arbitrary restrictions publishers place on image formats or resolutions, reference formats or page layouts. It is not that differences cannot be accomodated, it is that to do so involves intervention in a pipeline that was never designed, but has evolved over time. Changing some aspect is less a case of replacing one part with a more efficient newer version, and more a case of asking for a chicken to be provided with the wings of a bat.

Would it not be better to re-design the system? To make it a both properly designed and modular. Of course the answer is yes, but its not trivial, particularly at scale. Bear in mind that for a large publisher stopping the pipeline may mean they would never catch up again. More than this, such an architectural redesign requires not just changes at the end of the process but at the beginning. As we will see later in the series, new players and smaller players can have substantial advantages here, if they have the resources to design and build systems from scratch.

It’s not so bad. Things are improving, pipelines are becoming more automated and less and less of the processing of manuscripts to published articles is manual. But this isn’t yet accruing large savings because the long tail of messy problems was always where the main costs were. Much of this could be avoided if authors were happy with less finish on the final product, but where publishers have publicly tried this, for instance by reducing copy editing, not providing author proofs, or simply not investing in web properties there is usually a backlash. There’s a conversation to be had here about what is really needed and really wanted, and how much could be saved.

Managing Peer Review

The part of the process where many people agree there is value is in the management of peer review. In many cases the majority of labour contributed here is donated by researchers, particularly where the editors are volunteer academics, however there is a cost to managing the process, even if, as is the case for some services that is just managing a web property.

One of the big challenges is discussing the costs and value added in managing peer review is that researchers who engage in this conversation tend to be amongst the best editors and referees. Professional publishers on the other hand tend to focus on the (relatively small number of) contributors, who are, not to put too fine a point on it, awful. Good academic editors tend to select good referees who do good work, and when they encounter a bad referee they discount it and move on. Professional staff spend the majority of their time dealing with editors who have gone AWOL, referees who are late in responding, or who turn out to be inappropriate either in what they have written or their conflicts of interest, or increasingly who don’t even exist!

An example may be instructive. Some months ago a scandal blew up around the reviews of an article where the reviewer suggested that the paper would be improved if it had some male co-authors on it. Such a statement is inappropriate and there was justifiably an outcry. The question is who is responsible for stopping this happen? Should referees be more thoughtful, well yes, but training costs money as well and everyone makes mistakes. Should the academic editor have caught it? Again thats a reasonable conclusion but what is most interesting is that there was a strong view from the community that the publishing staff should have caught it. “That referee’s report should never have been sent back to the author…” was a common comment.

Think about that. The journal in question was PLOS ONE, publishing tens of thousands of papers a year, handling some amount more than that, with a few referees reports each. Lets say 100,000 reports a year. If someone needs to check and read every referee’s report and each one took 20 minutes on average (remember the long tail is the problem) then thats about four people working full time just reading the reports (before counting in the effort of getting problems fixed). You could train the academic editors better but with thousands of editors the training would also take about the same number of people to run it.  And this is just checking one part of one piece of the process. We haven’t begun to deal with identifying unreported conflicts of interest, managing other ethical issues, resolving disagreements etc etc etc.

Much of the irritation you see from publishers when talking about why managing peer review is more than “sending a few emails” relates to this gap in perception. The irony is that the problems are largely invisible to the broader community because publishers keep them under wraps, hidden away so that they don’t bother the community. Even in those cases where peer review histories are made public this behind the scenes work is never released. Academic Editors see a little more of it but still really only the surface, at least on larger journals and with larger publishers.

And here the irony of scale appears again. On the very smallest journals, where academics really are doing the chasing and running, there are also much fewer problems. There are fewer problems precisely because these small and close-knit communities know each other personally. In those niches the ethical and technical issues that are most likely to arise are also well understood by referees and editors. As a journal scales up this personal allegiance drops away, the problems increase, and the less likely that any given academic editor will be able to rely on their own knowledge of all the possible risks. We will return to this diseconomy of scale, and how it is or is not balanced by economies of scale that can be achieved again and again in this series.

A key question here is who is bearing the risk of something going wrong. In some cases editors are closely associated with a journal and bear some or much of that risk personally. But this is rare. And as we’ll see, with that personal interaction any slip ups are much more likely to be forgiven. Generally the risk lies primarily with the brand of the journal, and that appears as a financial risk to the publisher. Damage to the brand leads to weaker submissions and that is a substantial danger to viability. That risk is mitigated by including more checks throughout, those checks require staff, and those staff cost money. When publishers talk about “supporting the community’s review process” what they are often thinking is “preventing academics from tripping up and making a mess”.

Submission systems and validation

Probably the least visible part of what publishers do is the set of processes that occur before formal peer review even starts. The systems through which authors submit articles are visible and usually a major target for complaints. These are large complex systems that have built up over time to manage very heterogeneous processes, particularly at large publishers. They are mostly pretty awful. Hated in equal measure by both authors and publisher stuff.

The few examples of systems that users actually like are purpose built by small publishers. Many people have said it should be easy to build a submission system. And its true. It is easy to build a system to manage one specific work flow and one specific journal, particularly a small one. But building something that is flexible(ish) and works reliably at a scale of tens or hundreds of thousands of articles is quite a different issue. Small start ups like PeerJ and Pensoft have the ability to innovate and build systems from scratch that authors enjoy using. Elsevier by contrast has spent years, and reportedly tens of millions of dollars, trying to build a new system. PLOS has invested heavily in taking a new approach to the problem. Both are still in practice using different variants of the Editorial Manager system developed by Aries (Elsevier run their own version, PLOS pays for the service).

These systems are the biggest blocker to innovation in the industry. To solve the problems in production requires new ways of thinking about documents at submission, which in turn requires totally new systems for validation and management. The reasons why this acts as a technical block will be discussed in the next post.

What I want to focus on here is the generally hidden process that occurs between article submission and the start of the formal peer review process. Again, this is much more than “just sending an email” to an editor. There are often layers upon layers of technical and ethical checks. Are specific reporting requirements triggered? Has a clinical trial been registered? Does the article mention reagents or cell lines that have ethical implications? Is it research at all? Do the authors exist? Different journals apply different levels of scrutiny here. Some, like Acta Crystallographica E do a full set of technical checks on the data supporting the articles, others seem to not run any checks at all. But it can be very hard to tell what level any given journal works to.

It is very rare for this process to surface, but one public example is furnished by John Bohannan’s “Open Access Sting” article. As part of this exercise he submitted an “obviously incorrect” article to a range of journals, including PLOS ONE. The article included mention of human cell lines, and at PLOS this triggered an examination of whether appropriate ethical approval had been gained for those lines. Usually this would remain entirely hidden, but because Bohannan published the emails (something a publisher would never do as it would be breaking confidence with the author) we can see the back and forth that occurred.

Much of what the emails contain is automatic but that which isn’t, the to-and-fro over ethics approval of the cell lines is probably both unfamiliar to many of those who don’t think publishers do much but also surprisingly common. Image manipulation, dodgy statistical methods, and sometimes deeper problems are often not obvious to referees or editors. And when something goes wrong it is generally the publisher, or the brand that gets the blame.

Managing a long tail of issues

The theme that brings many of these elements together is that the idea that there is a long tail of complex issues, many of which only become obvious when looking across tens of thousands of articles. Any one of these issues can damange a journal badly, and consistent issues will close it down. Many of these could be caught by better standardised reporting, but researchers resist the imposition of reporting requirements like ARRIVE and CONSORT as an unnecsessary burden. Many might be caught by improved technical validation systems, provided researchers provided data and metadata in standardised forms. If every author submitted all their images and text in the same version of the same software file format using the same template (that they hadn’t played with to try and get a bit of extra space in the margins) then much could be done. Even in non-standardised forms progress could be made but it would require large scale re-engineering of submission platforms, a challenge to be discussed in the next section.

Or these checks and balances could be abandoned as unnecessary. Or the responsibility placed entirely on academic editors and referees. This might well work at a small scale for niche journals but it simply doesn’t work as things scale up, as noted above, something that will be a recurring theme.

There are many other things I have missed or forgotten: marketing, dissemination, payments handling, as well as the general overheads of managing an organisation. But most of these are just that, the regular overheads of running organisations as they scale up. The argument for these depends on the argument for need anything more than a basic operation in the first place. The point that most people miss from the outside is this consistent issue of managing the long tail: keeping the servers up 99.99%, dealing with those images or textual issue that don’t fit the standard pipeline, catching problems with the reviewing process before things get out of hand, and checking that the basics are right, that ethics, reporting requirements, data availability have been done properly across a vast range of submissions.

However the converse is also true, for those communities that don’t need this level of support, costs could be much lower. There are communities where the authors know each other, know the issues intimately and where there isn’t the need or desire for the same level of finish. Publishers would do well to look closer at these and offer them back the cost savings that can be realised. At the very least having a conversation about what these services actually are and explaining why they get done would be a good start.

But on the other side authors and communities need to engage in these discussions as communities. Communities as a whole need to decide what level of service they want, and to take responsibility for the level of checking and validation and consistency they want. There is nothing wrong with slapping the results of “print to PDF” up on the web after a review process managed by email. But it comes with both a lot of work and a fair amount of risk. And while a few loud voices are often happy to go back to basics, often the rest of the community is less keen to walk away from what they see as the professional look that they are used to.

That is not to say that much of this could be handled better with better technology and there has been a lack of investment and attention in the right parts of the article life cycle. Not surprisingly publishers tend to focus technology development in parts of the process visible to either authors or readers and neglect these less visble elements. And as we will see in future parts of the series, solving those problems requires interventions right the way through the system. Something that is a serious challenge.

Conclusion

It’s easy to say that much of what publishers do is hidden and that researcher are unaware of a lot of it. It is only when things go wrong, that scandals break, that the curtain gets pulled back. But unless we open up an honest conversation about what is wanted, what is needed, and what is currently being missed we’re also unlikely to solve the increasingly visible problems of fraud, peer review cartels and ethical breaches. In many ways, what has not been visible because they were problems at a manageable scale, are growing to the point of being unmanageable. And we can tackle them collectively or continue to shout at each other.

 

Posted on

The Political Economics of Open Access Publishing – A series

Victory Press of Type used by SFPP
Victory Press of Type used by SFPP (Photo credit: Wikipedia)

One of the odd things about scholarly publishing is how little any particular group of stakeholders seems to understand the perspective of others. It is easy to start with researchers ourselves, who are for the most part embarrassingly ignorant of what publishing actually involves. But those who have spent a career in publishing are equally ignorant (and usually dismissive to boot) of researchers’ perspectives. Each in turn fail to understand what libraries are or how librarians think. Indeed the naive view that libraries and librarians are homogenous is a big part of the problem. Librarians in turn often fail to understand the pressures researchers are under, and are often equally ignorant of what happens in a professional publishing operation. And of course everyone hates the intermediaries.

That this is a political problem in a world of decreasing research resources is obvious. What is less obvious is the way that these silos have prevented key information and insights from travelling to the places where they might be used. Divisions that emerged a decade ago now prevent the very collaborations that are needed, not even to build new systems, but to bring together the right people to realise that they could be built.

I’m increasingly feeling that the old debates (what’s a reasonable cost, green vs gold, hybrid vs pure) are sterile and misleading. That we are missing fundamental economic and political issues in funding and managing a global scholarly communications ecosystem by looking at the wrong things. And that there are deep and damaging misunderstandings about what has happened, is happening, and what could happen in the future.

Of course, I live in my own silo. I can, I think, legitimately claim to have seen more silos than the average; in jobs, organisations and also disciplines. So it seems worth setting down that perspective. What I’ve realised, particularly over the past few months is that these views have crept up on me, and that there are quite a few things to be worked through, so this is not a post, it is a series, maybe eventually something bigger. Here I want to set out some headings, as a form of commitment to writing these things down. And to continuing to work through these things in public.

I won’t claim that this is all thought through, nor that I’ve got (even the majority of) it right. What I do hope is that in getting things down there will be enough here to be provocative and useful, and to help us collectively solve, and not just continue to paper over, the real challenges we face.

So herewith a set of ideas that I think are important to work through. More than happy to take requests on priorities, although the order seems roughly to make sense in my head.

  1. What is it publishers do anyway?
  2. What’s the technical problem in reforming scholarly publishing
  3. The marginal costs of article publishing: Critiquing the Standard Analytics Paper and follow up
  4. What are the assets of a journal?
  5. A journal is a club: New Working Paper
  6. Economies of scale
  7. The costs (and savings) of community (self) management
  8. Luxury brands, platform brands and emerging markets (or why Björn might be right about pricing)
  9. Constructing authority: Prestige, impact factors and why brand is not going away
  10. Submission shaping, not selection, is the key to a successful publishing operation
  11. Challenges to the APC model I: The myth of “the cost per article”
  12. Challenges to the APC model II: Fixed and variable costs in scholarly publishing
  13. Alternative funding models and the risks of a regulated market
  14. If this is a service industry why hasn’t it been unbundled already (or where is the Uber of scholarly publishing?)
  15. Shared infrastructure platforms supporting community validation: Quality at scale. How can it be delivered and what skills and services are needed?
  16. Breaking the deadlock: Where are the points where effective change can be started?
Posted on

Who’s in the Club? New frames for understanding knowledge sharing

English: Venn diagram (coloured)
Venn diagram (Photo credit: Wikipedia)

The following is a version of the text I spoke from at the STEPS 2015 Conference, Resource Politics, at a session on Open Science organised by Valleria Arza, where I spoke along with Ross Mounce and Cindy Regalado. This version is modified slightly in response to comments from the audience.

There aren’t too many privileged categories I don’t fall into. White, male, middle class, middle aged, home owner. Perhaps the only claim I could make in the UK context is not having a connection with Oxbridge. The only language I speak is English and I’ve never lived in a non-english speaking country, never lived outside of Australia or England in fact. What do I have to say about developing countries? Or transitional, or emerging or peripheral…all problematic terms rooted in a developed world, western, narrative.

I hope it is more than just a hand-wringing liberal response. I suspect all of us do, genuinely base our work on a conviction that we can make a difference for good. In this context we collectively  believe that the critical tradition of scholarship developed in Western Europe can bring benefits to disadvantaged groups. And after all the resources to actually take action are in our hands or within our gift to influence. Something must be done. We can do something. Therefore we must do it.

Obviously this is an old critique and one that has shaped  decisions about how to act. We seek to move beyond charity and paternalistic intervention to offering frameworks and taking consultative approaches. To requiring a deeper understanding of context. In my own work I’ve tried to focus on offering ideas about processes and implementation, not on what should be done. But my ideas are of course still trapped within the frameworks I work within.

Central to those frameworks for me is “Open”. I’m not going to seek to explain in detail what I mean by “open”. Nor am I going to provide a critical analysis of the issues it raises in a development context, or its dependence on western liberal democratic, or neoliberal, or even libertarian values. Others are better placed to do that. What I do want to propose is that “Open” in the sense that I mean it is a culture, and it is a culture deeply rooted in its particular (north) western historical context.

If you accept that Open is a culture then our traditional thinking would be that it is the product of a particular community. Or rather communities. Again the history is complex but we can identify a complex of groups, clubs if you like, that have grown up with their own motivations and agendas but have sufficient alignment that we can collect them together under the label of “Open”. “Open Source”, “Open Data”, “Open Access”, “Open Science”, but also perhaps “Open Government”, transparency and others. Often these groups are identified with a charismatic individual.

John Hartley and Jason Potts in their book Cultural Science, propose a shift in our usual way of thinking about these groups and their cultures, that is both subtle and to my mind radical. We would usually think of individuals coming together to form groups in a common interest (often framed as an political-economic analysis of the way the collective resources of the group combine to achieve action). The individuals in the group and their values combine to define the culture of the group.

Hartley and Potts invert this. Their claim is that it is culture that creates groups. This inversion, whether you take it at face value as a real description of causation, or simply as a useful way to reframe the analysis has an important consequence. It focuses the unit of analysis onto the group rather than the individual. Rather than asking how individual behaviour lead to the consequences of groups interacting, we ask how cultures do or do not align, reinforce or cancel out.

In the session at the Resource Politics Conference on Tuesday on Assessment of Assessments we heard how governance reflects the structural moves allowed within an institution and how the framing of a problem reflects (or creates) these structures. Martin Mahony spoke of certain framings as “colonising spaces” which I would in turn appropriate as an example of how adaptive cultural elements can be spread through their re-creation or co-creation by groups.

In any case take your pick. New model of how culture, groups, society and sociotechnical institutions  that they co-create actually work and evolve, or a different framing that lets us tackle interesting problems from a new perspective. Either way, is it helpful? And what does it have to do with “development” or “sustainability” or “open” for that matter?

I think its useful (and relevant) because it lets us take a new view on the status of knowledge created in different contexts and it provokes some new ways of asking what we should do with the resources that we have.

First it gives us a license to say that some forms of knowledge are simply incompatible, growing as they do out of different cultures. But crucially it requires us to accept that in both directions – forms of knowledge from other cultures that are inaccessible to us, but also that our knowledge is accessible to others. It also suggests that some forms of compatibility may be defined through absence, exclusion or antagonism.

An anecdote: A few years ago I was working on a program focussing on Scholarly Communication in Sub-Saharan Africa. One striking finding was the way the communities of scholars, in disciplines that traditionally don’t communicate across groups, were actively engaging on social media platforms. Researchers from Russia, Iran, Iraq, Pakistan, India, Madagascar, Zimbabwe, Brazil, and Chile were all discussing the details of the problems they were facing in synthetic chemistry, a discipline that in my world is almost legendary for its obsessive individualism and lack of sharing. They shared the language of chemistry, and of English as the lingua franca, but they shared that with the absent centre of this geographical circle. Their shared culture was one of exclusion from the North Western centre of their discipline.

And yet, our culture, that of western scholarship was still dominant. I was struck yesterday in the “Assessment of Assessment” session focussing as it did on questions of transparency, engagement, and above all framing, that the framing of the session itself was not interrogated. Why, in an area focussed on building an inclusive consensus, is the mode of communication one of individual experts at the centre (as I am here on the podium) with questions to be asked, when allowed, from the periphery (you in the audience)?

Worse than that the system these chemists were using, ResearchGate is a western commercial infrastructures built from the classic Silicon Valley mindset, seeking to monetise, to in many ways to colonise, the interactions of these scholars who define themselves precisely through opposition to much of that culture. Is it possible to build a system that would help this group communicate within the scope of their culture but that doesn’t impose assumptions of our western culture? What infrastructures might be built that would achieve this and how would they be designed?

For Hartley and Potts the group, this level of analysis is one defined by shared culture. And of which cultures support groups which support the dynamic co- and re-creation of that culture. So another way of approaching this is to view them through the lens of the economics of clubs. What makes a club viable and sustainable? What goods does it use to achieve this? This group economics focus is interesting to me because it challenges many of our assumptions about the politics and economics of “Open”.

Rather than adopt a language of nationalisation of private goods: you journal publisher must give up your private property (articles) and transfer them to the public, you researcher must share your data with the world; we ask a different question – what is the club giving up and what are they gaining in return? Knowledge in this model is not a public good, but rather a club good – there is always some exclusion – that we are seeking to make more public through sharing. The political/economic (or advocacy) challenge is how to create an environment that tips the balance for clubs towards knowledge sharing.

These two questions – how can we support peripheral communities to co-create their own cultures without imposing ours and how we might change the economics of knowledge systems to favour investment in sharing – lead for me to an interesting suggestion and a paradox. What enabling infrastructures can be built and how can we make them as neutral and inclusive as possible while simultaneously embracing that anything built with western resources will be framed by our own cultures?

My stance on this is a re-statement of the concept from Zittrain, Benkler, Shirky and others that networks at scale can deliver new kinds of value. That the infrastructures we seek to build can tip the balance towards club investment in sharing if they provide mechanisms for clubs to gain access to networks. This is an architectural principle, that we can take a step up (or down if you prefer) identifying the common aspects of functionality required. It is also not new.

The new step is to adopt a principle of cultural engagement in governance, a means of – in the language of this conference – aligning the institutions that provide infrastructures and their governance and structures with the maximum possible number (and not power, not centrality) of cultures. The the criteria we use is one of maximising the number of productive interactions between cultures through the platforms we provide.

And this is what brings us back to Open, to what for me is the core of the philosophy, value system, or culture of Open Practice. Not that sharing outwards to the public is the target in itself but that it is through sharing that we create new opportunities for interaction and it is being open to contributions, to productive interactions that in my old world view creates value, but in this new framing promotes the “clashes of culture” that create new knowledge.

Posted on

Researcher as victim. Researcher as predator.

English: Illustration of a leopard and cheetah
English: Illustration of a leopard and cheetah (Photo credit: Wikipedia)

Researchers for the most part are pretty smart people. At the very least they’ve managed to play the games required of undergraduate and post graduate students, and out-competed a substantial proportion of other vying for the same places. Senior academics have survived running the gauntlet of getting published, and getting funded, at least enough to stay in the race.

It has been observed that when smart people do dumb things it is worth looking closer. The dumb thing is usually being done for a smart reason. Indeed we might go one step further and suggest that where a system is populated largely by smart people the proportion of dumb things they are doing could be a good diagnostic of how good the system is at doing what it is meant to do, as opposed to what it is measured to do. On this basis we might wonder about the health of many universities.

We also know there are a few issues with our systems of scholarly communications. And some of these involve unscrupulous and deceitful players out to reap a commercial gain. The story of so called “predatory” journals is a familiar one, which usually focusses on “publishers” claiming to offer open access services, but more recently an article in the Guardian took a similar stance on traditional academic monograph publishers.

In both cases the researcher is presented as a hapless victim, “hoodwinked” as the headline states into parting with money (either directly in the form of APCs or indirectly through their libraries). But really? I’ve no intent to excuse the behaviour of these publishers, but they are simply serving a demand. A demand created by researchers under immense pressure to demonstrate their productivity. Researchers who know how to play the game.

What is a line on a CV worth? Does it make that grant a little more likely? Does it get you past the magic threshold to get on the applicant short list? Is there a shortcut? Researchers are experts at behaviour optimisation and seeing how systems work. I simply don’t buy the “hapless victim” stance and a lot of the hand wringing is disingenuous at best. On a harsh economic analysis this is perfectly rational behaviour. Smart people doing dumb things for smart reasons.

The expansion of journal lists, the increasing costs to libraries, and the ever expanding list of journals that would take just about anything were never perceived as a problem by researchers when they didn’t see the bills. Suddenly as the business model shifts and the researcher sees the costs the arms are going up. The ever dropping circulation (and ever rising prices) of monographs was never really seen as a problem until the library budgets for monographs started to disappear as the serials crisis started to bight.

The symptoms aren’t restricted to dodgy publishing practices of course. Peer review cartels and fake reviewers result from the same impulse, the need to get more stuff published. Paper mills, fake journals, secondary imprints that will take any book proposal, predatory OA and bottom feeding subscription journals are all expressions of the same set of problems. And the terrifying thing is that responsible publishers are doing a pretty good job of catching a lot of it. The scale of the problem is much, much greater than is obvious from the handful of scandals and a few tens of retractions.

At times it is tempting to suggest that it is not publishers that are predatory, but researchers. But of course the truth is that we are all complicit, from publishers and authors producing content that no-one reads, through to administrators counting things that they know don’t matter, and funders and governments pointing to productivity, not to mention secondary publishers increasing the scope of they indices knowing that this leads to ever increasing inflation of the metrics that makes the whole system go round.

We are all complicit. Everyone is playing the game, but that doesn’t mean that all the players have the same freedom to change it. Commercial suppliers are only responding to demand. Governments and funders can only respond to the quality assessments of the research community. It is only the research community itself that can change the rules. And only a subset of that.

Emerging researchers don’t have the power to buck the system. It is senior researchers, and in particular those who mediate the interface between the sources of funding and the community, the institutional leaders, Vice-Chancellors, Presidents, Deans and Heads of Department. If institutional leaders chose to change the game, the world would shift tomorrow.

Scott Edmunds perhaps summed it up best at the FORCE2015 meeting in Oxford:

It is no longer the case that people are gaming the system, the system has become a game. It’s time to say Game Over.

If we cast ourselves as mere victims we’ll never change the rules. The whole narrative is an excuse for doing nothing.

This post was prompted in large part by tweets from Martin Coward and Nancy Sims

Posted on

What exactly is infrastructure? Seeing the leopard’s spots

This is a photo of a black leopard from the Ou...
Black leopard from Out of Africa Wildlife Park in Arizona (Wikipedia)

Cite as: What exactly is infrastructure? Seeing the leopard’s spots. Geoffrey Bilder, Jennifer Lin, Cameron Neylon. figshare. http://dx.doi.org/10.6084/m9.figshare.1520432

We ducked a fundamental question raised by our proposal for infrastructure principles: “what exactly counts as infrastructure?” This question matters. If our claim is that infrastructures should operate according to a set of principles, we need to be able to identify the thing of which we speak. Call the leopard, a leopard. Of course this is not a straightforward question and part of the reason for leaving it in untouched in the introductory post. We believe that any definition must entail a much broader discussion from the community. But we wanted to kick this off with a discussion of an important part of the infrastructure puzzle that we think is often missed.

In our conversations with scholars and others in the research ecosystem, people frequently speak of “infrastructures” when what they mean are services on top of deeper infrastructures. Or indeed infrastructures that sit on deeper infrastructures. Most people think of the web as an essential piece of infrastructures and it is the platform that makes much of what we are talking about possible. But the web is built on deeper layers of infrastructure: the internet, MAC addresses, IP, and TCP/IP. Things that many readers will never even have heard of because they have disappeared from view. . Similarly in academia, a researcher will point to “CERN” or “Genbank” or “Pubmed” or “The Perseus Project” when asked about critical infrastructure. The Physicists at CERN have long since taken the plumbing, electricity, roads, tracks, airports etc. that make CERN possible for granted.

All these examples involve services operating a layer above ones we have been considering. Infrastructure is not commonly seen. That lower level infrastructure has become invisible just as the underlying network protocols, which make Genbank, PubMed and the Perseus Project possible have also long since become invisible and taken for granted. To put a prosaic point on it, what is not commonly seen is also not commonly noticed. And that makes it even more important for us to focus on getting these deeper layers of infrastructure right.

If doing research entails the search for new discoveries, those involved are more inclined to focus on what is different about their research. Every sub-community within academia tends to think at a level of abstraction that is typically one layer above the truly essential – and shared – infrastructure. We hear physicists, chemists, biologists, humanists at meetings and conferences assume that the problems that they are trying to solve in online scholarly communication are specific to their particular discipline. They say “we need to uniquely identify antibodies” or “we need storage for astronomy data” or “we need to know which journals are open access” or “how many times has this article been downloaded”. Then they build the thing that they (think they) need.

This then leads to another layer of invisibility – the infrastructures that we were concerned with in the Principles for Open Scholarly Infrastructure are about what is the same across disciplines, not what is different. It is precisely the fact that these common needs are boring that means they starts to disappear from view, in some cases before they even get built. For us it is almost a law: people tend to identify infrastructure one layer too high. We want to refocus attention on the layer below, the one that is disappearing from view. It turns out that a black leopard’s spots can be seen – once they’re viewed under infrared light.

So where does that leave us? What we hear in these conversations across disciplines are not the things that are different (ie., what is special about antibodies or astronomy data or journal classifications or usage counts) but what is in common across all of these problems. This class of common problems need shared solutions. “We need identifiers” and “we need storage” and ”we need to assign metadata” and “we need to record relationships”. These infrastructures are ones that will allow us to identify objects of interest (ex: identify new kinds of research objects), store resources where more specialised storage doesn’t already exists (ex: validate a data analysis pipeline), and record metadata and relationships between resources, objects and ideas (ex: describing the relationships between funders and datasets). For example, the Directory of Open Access Journals provides identifiers for Open Access journals, claims about such journals, and relationships with other resources and objects (such as article level metadata, like Crossref DOIs and Creative Commons license URLs).

But what has generally happened in the past  is that each group re-invents the wheel for its own particular niche. Specialist resources build a whole stack of tools rather than layering on the one specific piece that they need on an existing set of infrastructures. There is an important counter-example: the ability to easily cross-reference articles and datasets as well as connect these to the people who created them. This is made possible by Crossref and Datacite DOIs with ORCID IDs. ORCID is an infrastructure that provides identifiers for people as well as metadata and claims about relationships between people and other resources (e.g., articles, funders, and institutions) which are in turn described by identifiers from other infrastructures (Crossref, FundRef, ISNI). The need to identify objects is something that we have recognised as common across the research enterprise. And the common infrastructures that have been built are amongst the most powerful that we have at our disposal. But yet most of us don’t even notice that we are using them.

Infrastructures for identification, storage, metadata and relationships enable scholarship. We need to extend the base platform of identifiers into those new spaces, beyond identification to include storage and references. If we can harness the benefits on the same scale that have arisen from the provision of identifiers like Crossref DOIs then the building of new services that are specific to given disciplines will become so much easier.  In particular, we need to address the gap in providing a way to describe relationships between objects and resources in general. This base layer may be “boring” and it may be invisible to the view of most researchers. But that’s the way it should be. That’s what makes it infrastructure.

It isn’t what is immediately visible on the surface that makes a leopard a leopard, otherwise the black leopard wouldn’t be, it is what is buried beneath.

[pdf-lite]

Posted on

Costly signalling in scholarly communications

Male Blue Peacock in Melbourne Zoo, Australia.
Male Blue Peacock in Melbourne Zoo, Australia. (Photo credit: Wikipedia)

For a long time it was difficult for evolutionary biology to make sense of a (male) peacock’s tail. Clearly it is involved in courtship but the investment in growing it, and the disdvantage of carrying it around, would seem to be a disadvantage over all. The burden of the tail might be worth it for a single male if female preferences are fixed

Fisher found a solution to this problem by noting that the genes for large tails in male peacocks would tend to be carried along with the genes for a preference for males with large tails expressed in females. In combination these two traits can cause a run away selection process which could explain the extravagant display in many animals.

Zahavi proposed another solution in which the display is a form of “costly signalling”. The ability to invest in the production of a large tail demonstrates the health or fitness of the animal. For this to work the signalling must be costly and it must be difficult to fake. Coloured plumage in the presence of stalking predators implies speed and agility, large horns (or simply size) a sufficient ability to obtain food.

Hartley and Potts in their book Cultural Science (chapter 3) apply the idea of costly signalling to question of cultural evolution. They suggest that cultures will adopt forms of costly signalling to create within-group trust and cohesion. In turn cultural norms of truth-telling and even traditions of narrative (the assumption of sympathy for the ‘white hat’, the presentation of compromises as ‘necessary’, that even bad acts reveal the underlying goodness of the hero) build community and in extremis send members of that community out to die for it in battle. This is not a facile claim about “group evolution” or how genetic evolution might drive culture but part of a program to understand how culture itself evolves.

One of the challenges of understanding peer review in the scientific community is why we do it at all. It is a part of our culture but it is very hard to demonstrate how and where it contributes value. The humanistic approach to the empirical challenge to value is to respond that it is a cultural norm that defines the scholarly community. Even if peer review achieved nothing it would have value as a means of defining a community, the community that has a cultural dedication to peer review. The “we”, the culture that valuesand engaged with peer review, is defined in terms of its different from the “they” who do not. This form of identification reinforces the analogy both with Fisher (we select those who share culture) and Zahavi (the costly signalling of engaging in peer review is part of the creation of our scholarly culture).

So perhaps another way to look at engaging with peer review is as costly signalling. The purpose of submitting work to peer review is to signal that the underlying content is “honest” in some sense. In the mating dance between researchers and funders or researchers and institutions the peer review process is intended to make the pure signalling of publication and to make it harder to fake. Taking Fisher’s view of mutual selection, authors on one side, funders and instiutions on the other, we can see, at least as analogy, a reason for the run away selection for publishing in prestigious journals. A runaway process where the signalling bares a tenous relationship with the underlying qualities being sought, in the same way as the size of the peacock’s tail has a tenous link with its health and fitness.

But as Martin Eve has argued (Open Access in the Humanities, Chapter 2), we need such signals. The labour of detailed assessment of all research for the full range of desirable qualities is unaffordable. Summaries and signals are needed. The question, perhaps, is whether this costly signalling is as honest as it could be. Is it creating a sustainable culture and community with a solid base? The apparent rise in fraud in retractions, particularly amongst those high prestige publications, suggests that this is a question that should be seriously addressed. To stretch the biological analogy, has a gene for faked tails emerged? Such fake display is not uncommon in biology.

Addressing that question means asking questions about what the underlying qualities we desire are. That’s an important question which I’ve raised elsewhere but I don’t want to go down that route here. I want to explore a different possibility. One that arises from asking whether a different form of signalling might be possible.

Communicating research in a reproducible (or replicable, or generalizable, the semantics are also an issue for another time) fashion is hard work. Many of us have argued that to enable greater reproducibility we need to provide better tools to reduce that cost. But what if the opposite were true? What if the value actually lies precise in the fact that communicating reproducibility is costly but is also potentially a more honest representation of what a community values than publication in a high profile journal.

If you buy that argument then we have a problem. The sexual selection run away is hard to break out of, at least in the case of biological evolution. At some point survivability prevents tails or horns growing so big they overbalance the animal, but by that stage a huge and unnecessary investment has been made. However in the case made by Potts and Hartley the thing that is evolving is more malleable. Perhaps, by creating a story of how the needs of funders and institutions are better served by focussing on a different form of signalling it will be possible to shift.

Of course this does happen in nature as well. When a sub-population develops a different form of display and co-selection kicks off then populations diverge, sometimes to occupy different niches, sometimes to compete, and ultimately displace the original population. It’s one way that new species form.

 

 

Posted on

Freedoms and responsibilities: Goffman, Hunt, Bohannan and Stapel

There has been much talk about both “academic freedom” as well as the responsibilities of scholars over the past few weeks. Both of these are troublesome concepts, not least because one person’s “freedom” is another’s irresponsible conduct. But particularly in the context of “academic freedom” the question of freedom to do or say what, and what responsibilities come with that is complex. And of course the freedom to speak is not the right to an expectation to be taken seriously. Any such right or authority is also tied to certain, usually unspecified responsibilities.

The question of academic freedom has been most visibly raised in the context of Tim Hunt’s reported comments at a Korean conference. As it happens I have my own story involving Hunt and misintepretation, once which might provide a useful way in to the issue.

At the closing panel of the Berlin11 meeting I spoke in a discussion panel about the progress towards open access and the future of scholarly communications. The meeting, in Berlin, was held in an old building that had been wonderfully refurbished as a conference space. In my remarks I drew an analogy with the building, the idea of taking the best of the past and repurposing it to support the future, noting that the building literally showed the scars of history, in this case the damage inflicted by allied bombing in World War II.

It was later related to me that Hunt had said that this was a very eloquent defence of journals like Nature and Science. Of course anyone who knows me will know that was absolutely not my intended meaning. What I meant, and what I said, were not congruent with what was heard. My intent was to provoke thought on what was worth keeping, not to defend the status quo. But who is responsible for the misunderstanding? What is my responsibility for greater clarity?

It may seem like a trivial misunderstanding, but it could have not been. We were in Berlin. The building might have a very dark history, certainly it is a near statistical certainty that some members of the audience had lost family members to allied bombing. My comments could have been misintepreted as saying that the building was more important than their relative’s suffering. That issue did not occur to me at the time, and looking back today I am ashamed by that. It may not have changed what I said, but it would certainly have changed the way I said it. As a sufficient authority to be asked to offer my views in the final session of an important meeting I had a responsibility to take care that my comments were authoritative but also that they were responsible.

Nobel Laureates travel a lot. They are in demand as speakers because they have authority. They have authority obviously in their area of research but also as senior members of the research community they bring a perspective as leaders who have been involved in the governance and strategic development of the research community. When that authority is assumed without sufficient care, or in areas where the person in question is not well informed, the result tends to rebound badly – Jim Watson’s comments on race, Pauling’s on vitamin C come to mind.

To those who are given great authority, whether in the form of Nobel prizes or large twitter followings, is also given great responsibility. Sometimes discharged well, sometimes not. Academic authority and academic freedom are not easy bedfellows. The right to speak one’s mind is freedom of speech. The ability to deploy one’s authority is not a right. Authority is the ability to be listened to not the ability to speak freely. And that ability comes with responsibility. Academic freedom is not the right to speak one’s mind. It is rather the responsibility to speak on issues, with the authority that arises from scholarly rigour. It is the tradition that employment should not be at risk when a scholar speaks in their area of expertise.

The most damning indictment therefore of the cries of “Academic Freedom” in the defense of Hunt is that his comments were bad science. They were spectacularly uninformed by the large quantity of literature that shows virtually the opposite of what he said (see Curt Rice’s blog for an up to date summary). Further the defence that “it was just a joke” can only be made by failing to engage with the literature that shows that not only do jokes surface real bias and real issues, but that asking a disdvantaged group to accept something as a joke normalises that disadvantage. Hilda Bastian has covered this in her excellent post.

The question of responsibility has also been raised in the furore of John Bohannan’s recent exposes, first on fly by night scholarly publishers seeking to fleece researchers, and more recently on the reporting, review and publicity around poorly run “medical” studies. In both cases questions are raised of methodology. In the Open Access sting many commentators, myself included, excoriated Bohannan for not running a proper control, in essence not running a proper scientific study. In the more recent chocolate study issues of ethical oversight and risk to participants were raised. If Bohannan was a scientist, speaking with the authority of a scholar then this would be a reasonable criticism. His own claim of the title “gonzo scientist” raises some interesting questions in this regard but fundamentally he is a journalist and writer, governed by different rules of authority and responsibility.

In the case of the OA sting those questions of authority were muddied by the publication of the piece in Science. Online the distinction between this journalistic piece and a research article is not immediately clear. To be fair, in the piece itself John does make the point that conclusions on the prevalence of poor peer review practices in subscription vs open access journals cannot be drawn from this work. Indeed his aim is a different kind of “proof”, in this case an existence proof of the problem – there are “journals” that do little to no peer review, and many of them are open access.

The problems I have with the piece, and they are many, arguably conflate my expectations of a piece of scholarly research and the responsibilities of a scholar – the need to tell us something new or useful – with the very different aims of a journalist, to expose an issue to a wider audience. Indeed the very establishment power structures moving into place to defend Hunt are the ones that I, arguably hypocritically, deployed to combat Bohannan. “The problem has been known for some time” “Quiet work was being done on it” “Leave our community to get on and sort out our problems”. But did we need an outsider to make it public enough and urgent enough to drive real action? Did Bohannan have responsibilities to the Open Access community to tell us more about the problem, to do a proper study, or as a journalist was his responsiblity to publicly expose the issue?

Alice Goffman is another researcher facing a different set of tensions over responsibility, freedom and authority. Her book On the Run gives a challenging account of inner city life amongst deprived black american youth. Published in 2014 it can be seen as a warning of the subsequent events in Ferguson and Baltimore and other places.

Goffman is an enthographer and her book started life as a scholarly monograph, but one that has gone on to have success as a mainstream non-fiction book. Ethnography involves working closely with, often living with research subjects, and the protection of the privacy of subjects is held as a very high principle. As described in this Slate article (which is my main source) this generally means obscuring locations, names, even the chronology of events to create a narrative which surfaces a deeper underlying truth about what is going on. Goffman took this responsibility particularly seriously given she observed events that could land people in jail, going so far as to destroy her notebooks so as to protect her research subjects.

But as this uncomfortable narrative became more public and transformed into a mainstream non-fiction book the responsibilities of the author (no longer a scholar?) seemed to change. General non-fiction is supposed to be “true” and Goffman’s rearrangement of facts, people and timelines breaks this expectation. What is interesting is that was in turn is used to raise charges of scholarly misconduct. The responsibility of the author to the reader is in direct conflict with the responsibility of the scholar to their subjects, yet the critic chooses to attack the scholarship. Indeed, given that the criticism and claims of misconduct are based on a forensic analysis of the text in some sense Goffman is under attack because she didn’t do a good enough job of hiding the process of discharging her scholarly responsibilities, leaving inconsistencies in the timelines and events.

Which responsibility trumps which? What does “integrity” mean in this context, or rather disparate and competing contexts, and how does a public scholar working on important and challenging problems navigate those competing issues? Where is Goffman’s academic freedom and where do her academic responsibilities lie? In restricting her space for communication to the academic world? In speaking (her) truth to power? Or is that space left for those licensed to speak through mainstream books? Is it left for Bohannan because only the outsider can make that transition?

The question of research integrity in Goffman’s case is challenging. Her destruction of notebooks certainly disturbs me as someone concerned primarily with the integrity of the research record. But I can respect the logic and to the extent that it is seen as reasonable within her disciplinary context accept that as appropriate scholarly practice.

The question of fraud in natural and social science research may seem much clearer. Diederik Stapel (I could have easily chosen Jan Hendrik Schön or many others) simply made up datasets. Here it seems there are clear lines of responsibility. The scholar is expected to add to the record, not muddy it. As we move towards digital records and data sharing these expectations are rising. Reproducible research is a target that seems plausible at least in some disciplines, although ironically we are perhaps merely returning to the level of record keeping recommended by Robert Boyle in 1660.

Does academic freedom mean the right to publish results based on made up data? Of course not. The scholar has a responsibility to report accurately when speaking in a scholarly context. It is not a crime to make up data, even in a research context. Ideas might be expressed though imagined or constructed datasets, they may even be an integral part of the research process as test sets, statistical tools or training sets. It is a “crime” to misrepresent or mis-use them. Even carelessness is treated a significant misdemeanour, leading as it does to retraction and consequent embarassment. Where does “carelesness” of the type that leads to retraction become “foolishness” that only requires mild rebuke?

But the idea of a “complete record” and “reproducibility” is a slippery one. In Goffman’s case reproducibility is impossible even in principle. Ethnographers would I imagine regard it as deeply problematic. The responsibility here is not even to report true facts, but the deeper truth – as the scholar sees it – that underlies the events they observe. Stapel may well have thought he was also telling “a truth”, just one for which the data wasn’t quite clean enough. A serious issue behind Bohannan’s chocolate expose is that p-value hacking, searching a weak dataset for “a truth” to tell, is endemic in many disciplines and that peer review as currently constructed is impotent in tackling it. Peer review assumes that authors have taken on board the responsibility to tell the truth (something Bohannan explicitly didn’t do for instance in the correspondence he had with PLOS One staff in the technical checks done before formal peer review).

Many of the technical discussions of reproducibility and data sharing founder on issues of reproducible for who? At what level? In what way? Bohannan shared his data, but you could not now reproduce his “experiment” precisely. His actions make that impossible. Goffman’s data does not exist but events in Ferguson, Baltimore and elsewhere arguably confirm her claims and narrative. Does Amgen’s failure to reproduce the vast majority of findings published on cancer biology in “top journals” mean we have a crisis?

Perhaps better to ask, what is the responsibility of authors publishing in cancer biology to their readers. To tell the truth as they see it? Obviously. To use all the tools at our disposal to prevent us fooling ourselves, to prevent us seeing what we want to see? Certainly. To provide the data? Increasingly so. To ensure the materials (cell lines, antibodies, reagents) are available to those who do want to do direct replication? Oh, that might be too much to expect. Today at least, but tomorrow? This is a discussion about responsibilies. Not technical details. Responsibilities to who, and for what, and how does that vary across disciplines. Perhaps focussing on “reproducibility” is the wrong approach.

As freedom of speech is merely right to a voice, not to a listener, academic freedom has its limits. The boundaries between scholarly speech, within a scholarly community, and wider public speech is blurring, as Goffman and Hunt have found, and as Bohannan has shown us. Context matters, whether the context of my previous writing on the merits and de-merits of Nature, the history of a building, or in the choice to make a joke of the wrong type in the wrong place. And the authority that comes from experience and responsibility in one space does not always travel well into a different context.

Does this mix of contexts and expectations mean we should simply give up? Just be quiet and retreat? That would be the easy answer. But the wrong one. Academic Freedom, or Academic Responsibility comes with the responsibility to speak. But it is a responsibility to be exercised with care. And with empathy for the different contexts that different audiences may find themselves in. Showing our working and showing our thinking. Showing the disciplinary traditions and expectations, the responsibilities that we have assumed, explicitly will help.

Above all, speaking from a position of authority (and I have chosen to use the word authority, rather than power deliberately) means assuming a higher level of reponsibility. This is perhaps best summed up in the direct advice “never punch down”. When speaking from a position of scholarly authority the limits of that authority, the limits of the experience, and the care expected in having mastery of the evidence are higher. And this is reasonable. And more and more important if scholarship is to be part of the wider world and not something that is done to it. If, after all, scholarship is about informed criticism and discussion, we all have a responsibility not just to speak, with care, but also to listen.

This piece has been very strongly shaped by a range of recent discussions, most strongly with Michael Nielsen (on John Bohannan’s work) and Michelle Brook (on diversity, power relations, integrity and the tensions between them), but also the ongoing discussion on twitter and more generally about Tim Hunt’s comments and Bohannan’s recent “sting”.

Posted on

Community Support for ORCID – Who’s next to the plate?

Geoff Bilder, Jennifer Lin, Cameron Neylon

The announcement of a $3M grant from the Helmsley Trust to ORCID is a cause for celebration. For many of us who have been involved with ORCID, whether at the centre or the edges, the road to sustainability has been a long one, but with this grant (alongside some other recent successes) the funding is in place to take the organization to where it needs to be as a viable membership organization providing critical community services.

When we wrote the Infrastructure Principles we published some weeks back, ORCID was at the centre of our thinking, both as one of the best examples of good governance practice and as an infrastructure that needs sustaining. To be frank it has been disappointing, if perhaps not surprising how long it has taken for key stakeholders to step up to the plate to support its development. Publishers get a lot of stick when it comes to demanding money, but when it comes to community initiatives it is generally publisher that put up the initial funding. This has definitely been the case with ORCID, with funders and institutions falling visibly behind, apparently assuming others will get things moving.

This is a common pattern, and not restricted to scholarly communications. Collective Action Problems are hard to solve, particularly when communities are diverse and have interests that are not entirely aligned. Core to solving collective action problems is creating trust. Our aim with the infrastructure principles was very much to raise the issue of trust and what makes a trustworthy organization to a greater prominence.

Developing trust amongst our diverse communities requires that we create trustworthy institutions. We have a choice. We can create those institutions in a way that embodies our values, the values that we attempted to articulate, or we can leave it to others. Those others will have other values, and other motives, and cannot be relied upon to align with our communities’ interests.

Google Scholar is a classic example. Everybody loves Google Scholar, and it’s a great set of tools. But it has not acted in a way that serve the communities’ broader needs. It does not have an API to allow the data to be re-used elsewhere. We cannot rely on it to continue in its current form.

Google Scholar exists fundamentally so that researchers will help Google organize the world’s research information for Google’s benefit. ORCID exists so that the research community can organize the world’s research information for our community’s benefit. One answers to its shareholders, the other to the community. And as a result needs the support of our community for its sustainability. As the saying goes, when you don’t pay for the product, you are the product.

ORCID is a pivotal case. Will our communities choose to work together to build sustainable infrastructures that serve our needs and answer to us? Or will we repeat the mistakes of the past and leave that to other players whose interests do not align with our own. If we can’t collectively bring ORCID to a place where it is sustainable, supported by the whole community then what hope is there for more specialist, but no less necessary, infrastructures?

The Helmsley Trust deserves enormous credit for stepping up to the plate with this grant funding, as do the funders (including publishers) and early joining members who have gone before. But as a community we need to do more than provide time limited grants. We need to take the collective responsibility to fund key infrastructure on an ongoing basis. And that means that other funders, institutions, alongside publishers need to play their part.

Posted on

.everyone or .science? Or both? Reflections on Martha Lane Fox’s Dimbleby Lecture

English: Martha Lane Fox
Martha Lane Fox (Photo: The Cabinet Office License: OGL v1.0)

On March 30 the BBC broadcast a 40 minute talk from Martha Lane Fox. The Richard Dimbleby Lecture is an odd beast, a peculiarly British, indeed a peculiarly BBC-ish institution. It is very much an establishment platform, celebrating a legendary broadcaster and ring marshaled by his sons, a family that as our speaker dryly noted are “an entrenched monopoly” in British broadcasting.

 

Indeed one might argue Baroness Lane Fox, adviser to two prime ministers, member of the House of Lords, is a part of that establishment. At the same time the lecture is a platform for provocation, for demanding thinking. And that platform was used very effectively to deliver a brilliant example of another very British thing, the politely impassioned call for radical (yet moderate) action.

The speech calls for the creation of a new public institution. Dubbed “Dot Everyone” such an institution would educate, engage and inform all citizens on the internet. It would act as a resource, it would show what might be possible, it would enhance diversity and it would explore and implement a more values based approach to how we operate on the web. I have quibbles, things that got skipped over or might merit more examination, but really these are more the product of the space available than the vision itself.

At the centre of that vision is a call for a new civics supported by new institutions. This chimes with me as it addresses many of the same issues that have motivated my recent thinking in the research space. The Principles for Open Infrastructures I wrote with Geoff Bilder and Jennifer Lin, could as easily have been called Principles for Institutions – we were motivated to work on them because we believe in a need for new institutions. For many years I have started talks on research assessment by posing the question “what are your values” – a question implicit in the speech as it probes the ethics of how the internet is built in practice.

I was excited by this speech. And inspired.

And yet.

One element did not sit easily with me. I emphasized the British dimension at the top of this piece. Martha Lane Fox’s pitch was to “make Britain brilliant at the internet” and was focused on the advantages for this country. By contrast the first of the Principles for Open Infrastructures is that these new institutions must transcend geography and have international reach. Is this a contradiction? Are we pushing in different directions? More particularly is there a tension between an institution “for everyone” and one having a national focus?

The speech answers this in part and I think the section is worth quoting in full:

We should be ambitious about this. We could be world leading in our thinking.

In this 800th year anniversary of Magna Carta, the document widely upheld as one of the first examples of the rule of law, why don’t we establish frameworks to help navigate the online world?

Frameworks that would become as respected and global as that rule of law, as widely adopted as the Westminster model of parliamentary democracy.

Clearly this new institution, “our new institution” as it is referred to throughout, has international ambitions. But I don’t imagine I am the only person to find something almost neo-colonial in these words. Britain has sought to export its values to the world many times, and been remarkably successful. But in the past this has also been paternalistic. The very best possible assessment of what was in many cases well intentioned imposition of British values is equivocal. Lane Fox sets up the “good” values of Britain against the lack of values that inhere in the big commercial players building the web. What is it today that make “our” values those that should inspire or lead any more than the, now questionable, values of the past?

To be clear I am absolutely not suggesting that these are issues that have escaped the speaker’s notice. Martha Lane Fox is an outstanding and effective campaigner for diversity and inclusion and the section of her talk that I have taken out of context above comes after a substantial section on the value of inclusion, focused largely on gender but with a recognition that the same issues limit the inclusion and contribution of many people on the basis of many types of difference. In truth, her views on the how and the why of what we need to change, on what those values are, are highly aligned with mine.

But that’s kind of the point.

If we are to have a new civics, enabled by the communications infrastructure that the web provides, then diversity will lie at the heart of this. Whether you take the utilitarian (not to say neo-liberal) view that inclusion and diversity drives the creation of greater value, or see it as simply a matter of justice, diversity and inclusion and acceptance of difference are central.

But at the same time the agile and flat governance models that Lane Fox advocates, to be fair in passing, for our new institution arise out the concept that “rough consensus and running code” are the way to get things done. But whose consensus matters? And how does the structural imbalance of the digital divide affect whose code gets to run first? This seems to me the central question to be resolved by this new civics. How do we use the power of web to connect communities of interest, and to provide infrastructures that allow them to act, to have agency, while at the same time ensuring inclusion.

At its best the web is an infrastructure for communities, a platform that allows people to come together. Yet communities define themselves by what they have in common, and by definition exclude those who do not share those characteristics. My implicit claim above that our institutional principles are somehow more inclusive or more general than Lane Fox’s is obviously bogus. Our focus is on the research community, and therefore just as exclusive as a focus on a single nation. There are no easy answers here.

The best answer I can give is that we need multiple competing centres. “Dot Everyone” is a call for a national institution, a national resurgence even. Alone it might be successful, but even better is for it to have competition. Martha Lane Fox’s call is ambitious, but I think it’s not enough. We need many of these institutions, all expressing their values, seeking common ground to build a conversation between communities, domains, geographies and nations.

The tension between facilitating community and diversity can be a productive one if two conditions are satisfied. First that all can find communities where they belong, and secondly that the conversation between communities is just and fair. This is a huge challenge, it will require nothing less than a new global infrastructure for an inclusive politics.

It is also probably a pipe dream, another well meaning but ultimately incomplete effort to improve the world. But if the lesson we learn from colonialism is that we should never try, then we should give up now. Better is to do our best, while constantly questioning our assumptions and testing them against other’s perspectives.

As it happens, we have some new systems that are pretty good for doing that. We just need to figure out how best to use them. And that, at core, was Martha Lane Fox’s point.

 

 

Posted on

Principles for Open Scholarly Infrastructures

Cite as “Bilder G, Lin J, Neylon C (2015) Principles for Open Scholarly Infrastructure-v1, retrieved [date], http://dx.doi.org/10.6084/m9.figshare.1314859“

UPDATE: This is the original blogpost from 2015 that introduced the Principles. You also have the option to cite or reference the Principles themselves as: Bilder G, Lin J, Neylon C (2020), The Principles of Open Scholarly Infrastructure, retrieved [date], https://doi.org/10.24343/C34W2H

infrastructure /ˈɪnfɹəˌstɹʌkt͡ʃɚ/ (noun) – the basic physical and organizational structures and facilities (e.g. buildings, roads, power supplies) needed for the operation of a society or enterprise. – New Oxford American Dictionary

Everything we have gained by opening content and data will be under threat if we allow the enclosure of scholarly infrastructures. We propose a set of principles by which Open Infrastructures to support the research community could be run and sustained. – Geoffrey Bilder, Jennifer Lin, Cameron Neylon

Over the past decade, we have made real progress to further ensure the availability of data that supports research claims. This work is far from complete. We believe that data about the research process itself deserves exactly the same level of respect and care. The scholarly community does not own or control most of this information. For example, we could have built or taken on the infrastructure to collect bibliographic data and citations but that task was left to private enterprise. Similarly, today the metadata generated in scholarly online discussions are increasingly held by private enterprises. They do not answer to any community board. They have no obligations to continue to provide services at their current rates, particularly when that rate is zero.

We do not contest the strengths of private enterprise: innovation and customer focus. There is a lot of exciting innovation in this space, much it coming from private, for profit interests, or public-private partnerships. Even publicly funded projects are under substantial pressures to show revenue opportunities. We believe we risk repeating the mistakes of the past, where a lack of community engagement lead to a lack of community control, and the locking up of community resources. In particular our view is that the underlying data that is generated by the actions of the research community should be a community resource – supporting informed decision making for the community as well as providing as base for private enterprise to provide value added services.

What should a shared infrastructure look like? Infrastructure at its best is invisible. We tend to only notice it when it fails. If successful, it is stable and sustainable. Above all, it is trusted and relied on by the broad community it serves. Trust must run strongly across each of the following areas: running the infrastructure (governance), funding it (sustainability), and preserving community ownership of it (insurance). In this spirit, we have drafted a set of design principles we think could support the creation of successful shared infrastructures.

Governance

If an infrastructure is successful and becomes critical to the community, we need to ensure it is not co-opted by particular interest groups. Similarly, we need to ensure that any organisation does not confuse serving itself with serving its stakeholders. How do we ensure that the system is run “humbly”, that it recognises it doesn’t have a right to exist beyond the support it provides for the community and that it plans accordingly? How do we ensure that the system remains responsive to the changing needs of the community?

  • Coverage across the research enterprise – it is increasingly clear that research transcends disciplines, geography, institutions and stakeholders. The infrastructure that supports it needs to do the same.
  • Stakeholder Governed – a board-governed organisation drawn from the stakeholder community builds more confidence that the organisation will take decisions driven by community consensus and consideration of different interests.
  • Non-discriminatory membership – we see the best option as an ‘opt-in’ approach with a principle of non-discrimination where any stakeholder group may express an interest and should be welcome. The process of representation in day to day governance must also be inclusive with governance that reflects the demographics of the membership.
  • Transparent operations – achieving trust in the selection of representatives to governance groups will be best achieved through transparent processes and operations in general (within the constraints of privacy laws).
  • Cannot lobby – the community, not infrastructure organizations, should collectively drive regulatory change. An infrastructure organisation’s role is to provide a base for others to work on and should depend on its community to support the creation of a legislative environment that affects it.
  • Living will – a powerful way to create trust is to publicly describe a plan addressing the condition under which an organisation would be wound down, how this would happen, and how any ongoing assets could be archived and preserved when passed to a successor organisation. Any such organisation would need to honour this same set of principles.
  • Formal incentives to fulfil mission & wind-down – infrastructures exist for a specific purpose and that purpose can be radically simplified or even rendered unnecessary by technological or social change. If it is possible the organisation (and staff) should have direct incentives to deliver on the mission and wind down.

Sustainability

Financial sustainability is a key element of creating trust. ‘Trust’ often elides multiple elements: intentions, resources and checks and balances. An organisation that is both well meaning and has the right expertise will still not be trusted if it does not have sustainable resources to execute its mission. How do we ensure that an organisation has the resources to meet its obligations?

  • Time-limited funds are used only for time-limited activities – day to day operations should be supported by day to day sustainable revenue sources. Grant dependency for funding operations makes them fragile and more easily distracted from building core infrastructure.
  • Goal to generate surplus – organisations which define sustainability based merely on recovering costs are brittle and stagnant. It is not enough to merely survive it has to be able to adapt and change. To weather economic, social and technological volatility, they need financial resources beyond immediate operating costs.
  • Goal to create contingency fund to support operations for 12 months – a high priority should be generating a contingency fund that can support a complete, orderly wind down (12 months in most cases). This fund should be separate from those allocated to covering operating risk and investment in development.
  • Mission-consistent revenue generation – potential revenue sources should be considered for consistency with the organisational mission and not run counter to the aims of the organisation. For instance…
  • Revenue based on services, not data – data related to the running of the research enterprise should be a community property. Appropriate revenue sources might include value-added services, consulting, API Service Level Agreements or membership fees.

Insurance

Even with the best possible governance structures, critical infrastructure can still be co-opted by a subset of stakeholders or simply drift away from the needs of the community. Long term trust requires the community to believe it retains control.

Here we can learn from Open Source practices. To ensure that the community can take control if necessary, the infrastructure must be ‘forkable’. The community could replicate the entire system if the organisation loses the support of stakeholders, despite all established checks and balances. Each crucial part then must be legally and technically capable of replication, including software systems and data.

Forking carries a high cost, and in practice this would always remain challenging. But the ability of the community to recreate the infrastructure will create confidence in the system. The possibility of forking prompts all players to work well together, spurring a virtuous cycle. Acts that reduce the feasibility of forking then are strong signals that concerns should be raised.

The following principles should ensure that, as a whole, the organisation in extremis is forkable:

  • Open source – All software required to run the infrastructure should be available under an open source license. This does not include other software that may be involved with running the organisation.
  • Open data (within constraints of privacy laws) – For an infrastructure to be forked it will be necessary to replicate all relevant data. The CC0 waiver is best practice in making data legally available. Privacy and data protection laws will limit the extent to which this is possible.
  • Available data (within constraints of privacy laws) – It is not enough that the data be made ‘open’ if there is not a practical way to actually obtain it. Underlying data should be made easily available via periodic data dumps.
  • Patent non-assertion – The organisation should commit to a patent non-assertion covenant. The organisation may obtain patents to protect its own operations, but not use them to prevent the community from replicating the infrastructure.

Implementation

Principles are all very well but it all boils down to how they are implemented. What would an organisation actually look like if run on these principles? Currently, the most obvious business model is a board-governed, not-for-profit membership organisation, but other models should be explored. The process by which a governing group is established and refreshed would need careful consideration and community engagement. As would appropriate revenue models and options for implementing a living will.

Many of the consequences of these principles are obvious. One which is less obvious is that the need for forkability implies centralization of control. We often reflexively argue for federation in situations like this because a single centralised point of failure is dangerous. But in our experience federation begets centralisation. The web is federated, yet a small number of companies (e.g., Google, Facebook, Amazon) control discoverability; the published literature is federated yet two organisations control the citation graph (Thomson Reuters and Elsevier via Scopus). In these cases, federation did not prevent centralisation and control. And historically, this has occurred outside of stewardship to the community. For example, Google Scholar is a widely used infrastructure service with no responsibility to the community. Its revenue model and sustainability are opaque.

Centralization can be hugely advantageous though – a single point of failure can also mean there is a single point for repair. If we tackle the question of trust head on instead of using federation as a way to avoid the question of who can be trusted, we should not need to federate for merely political reasons. We will be able to build accountable and trusted organisations that manage this centralization responsibly.

Is there any existing infrastructure organisation that satisfies our principles? ORCID probably comes the closest, which is not a surprise as our conversation and these principles had their genesis in the community concerns and discussions that led to its creation. The ORCID principles represented the first attempt to address the issue of community trust which have developed in our conversations since to include additional issues. Other instructive examples that provide direction include Wikimedia Foundation and CERN.

Ultimately the question we are trying to resolve is how do we build organizations that communities trust and rely on to deliver critical infrastructures. Too often in the past we have used technical approaches, such as federation, to combat the fear that a system can be co-opted or controlled by unaccountable parties. Instead we need to consider how the community can create accountable and trustworthy organisations. Trust is built on three pillars: good governance (and therefore good intentions), capacity and resources (sustainability), and believable insurance mechanisms for when something goes wrong. These principles are an attempt to set out how these three pillars can be consistently addressed.

The challenge of course lies in implementation. We have not addressed the question of how the community can determine when a service has become important enough to be regarded as infrastructure nor how to transition such a service to community governance. If we can answer that question the community must take the responsibility to make that decision. We therefore solicit your critique and comments on this draft list of principles. We hope to provoke discussion across the scholarly ecosystem from researchers to publishers, funders, research institutions and technology providers and will follow up with a further series of posts where we explore these principles in more detail.

The authors are writing in a personal capacity. None of the above should be taken as the view or position of any of our respective employers or other organisations.