Speculations: Abstracting Evolution

This post is aiming to get down some thoughts around how the superset of evolutionary models can be framed. It’s almost certainly work that has been done somewhere before but I’m struggling to find it so it seemed useful to lay out what I’m looking for. 

Evolutionary models are extraordinarily powerful, in part because they are extremely flexible. At their worst they are tautologies – the criticism of “survival of the fittest” as an idea is well founded even if it’s description of Darwinism is not – but at their best they provide an understanding that stretches from our capacity to engineer molecules, to an understanding of our origins and possibilities, to the design and development of various forms of artificial intelligence, the optimisation algorithms that is based on, and our understanding of the development of culture.

Within that enormous scale there seems to be quite a lot of sloppiness. Much of the popular literature on biological evolution focusses on refining down to a strict definition of what evolution is, and what is evolving, largely as a response to the political challenge of creationism and intelligent design advocates. In turn these strict definitions fail to cover the emerging complexities of how different parts of biological systems, genes, genomes, cells, organisms, communities, species, ecologies are subject to selection and differential persistence. The simplifying assumption that biological evolution is equivalent to DNA sequence evolution is both enormously powerful and obviously incomplete. The easy parallels made with genetic optimization algorithms and the apparently “biological” look of items that sometimes arise from them makes for easy analogies but common analysis is harder.

One of the first scientific insights that captured my imagination was an intuition that biological systems are set up in such a way as to be evolvable. The layering of just enough chemical diversity in the form of proteins onto a neat sequential instruction set. The way a simple linear molecule is set up to enable huge structural diversity. Most fascinating of all these systems must necessarily have evolved to be evolvable. And yet at the same time they can be quite recalcitrant to our naive attempts to apply what we think of as the same evolutionary process – randomise gene, express protein, select for function – in the lab. Like most persistent systems and many biological systems, the molecular evolvability of proteins is both resistant to small scale perturbation, but clearly based on the reconstruction of evolution over the long term, immensely flexible in the face of existential challenges.

Twenty years later I’m working within a model of culture and groups that has as one of its central claims, that it is an evolutionary model. Attempts to transplant evolutionary framings into those domains traditionally seen as belonging to the  academic humanities have generally been unsuccessful. The claim of the Cultural Science program is that this has been due to a misdiagnosis of what it is that is under selection.

In cultural science it is meaningfulness that evolves, ‘demically’. However, it is misleading to represent this as referring to ‘units of meaningfulness’, as if merely seeking to re-label ‘basic cultural unit’ with ‘basic unit of meaningfulness’. Instead, our claim is ontological and seeks to develop the idea that cultural evolution is the emergence of meaningfulness from webs of associations and relations and also by negotiation and use within a deme and between demes. Importantly, this is not a thing, or even information per se, but a structure of associations in action. It is these dynamic demic associations that evolve.

Hartley and Potts (2015), Cultural Science p126

Ironically this is not a paragraph that will very meaningful to many evolutionary biologists. One important point is that identifying the “unit of selection” is a challenge for any evolutionary theory. Here it is claimed that the unit of selection for culture is inherently complex, a dynamic network of narratives and meanings. These can be probed using traditional humanistic and social sciences techniques, discourse analysis, close and distant reading, critique of framing, ethnography and sociology. My own internal analogy is that the results of these studies are to the true “genes of culture” as the image of bands on a gel (showing my age there!) are to the real operation of DNA in a complex organism – simplifications built on techniques that frame the underlying complexity in a way that both makes it comprehensible but also reinforces the framing of the technique.

John Welch in a recent paper makes a similar point in response to calls for reform of (biological) evolutionary theory.

It is argued that a few inescapable properties of the field make it prone to criticisms of predictable kinds, whether or not the criticisms have any merit. For example, the variety of living things and the complexity of evolution make it easy to generate data that seem revolutionary (e.g. exceptions to well-established generalizations, or neglected factors in evolution), and lead to disappointment with existing explanatory frameworks (with their high levels of abstraction, and limited predictive power).

Welch (2016), http://doi.org/10.1007/s10539-016-9557-8

Biological systems are damn good at co-opting effects, systems to their own end. It is in the nature of evolved and evolvable systems to be capable of opportunistically taking advantage of whatever they can get their hands on. If the anthropomorphic language bothers you; those systems/collectives more capable of extracting benefiting opportunistically from the environment will, all other things being equal, persist more consistently than those that do not. Eugene Koonin in another very recent paper cautions against focussing too much on selection, in comparison to the role of neutral sequence change and diversification. That attention to the mechanisms of diversification is as important as to those of selection is obvious. The assumption that diversification, or rather change, and selection, are coupled in some form to the idea of replication, and particularly that there is a sequential pattern to be followed is a less obvious error, but one common particularly in cases of modelling or algorithmic design. See for instance slide 12 in this slidedeck from Knowles and Watson (2017).

In the case of demic cultural evolution sketched above it is not clear we even expect replication. Cultures may divide and split but that is not a necessary part of the model1, rather that they compete and differentially survive. We can talk about the persistence and continuity of human microbiome (and its successful or unsuccessful transmission to a new-born child) in ways that sound Lamarckian at one level. That a microbiome evolves is clear. That it has an effect on the adaptive fitness of the host is clear, and that its characteristics likely have a long term effect on the population genetics of the host species(s) is at the very least plausible. To understand the whole system we need a highly sophisticated notion of what it is that is evolving. In practice we focus on the persistence of one identifiable object (genetic markers in the host, the presence of a specific sequence – or set of sequences – within the microbiome) for any given study. This is the point Welch makes, that there are principled reasons for focussing on the evolution of a specific identifiable object.

The reductionist agenda gave us the view of genes as digital DNA sequences and from that the full power of the evolutionary synthesis. One logical endpoint of Welch’s argument is that we have to accept this is not the full story that other elements of the system are evolving. Koonin cautions us to be more open to what forms of selection and diversification matter, but to my mind falls into the same trap, the assumption that all genes are DNA sequences, and therefore that all evolution can be examined through statistical analysis of DNA sequences. Evolution may “only make sense in the light of population genetics” as Lynch tells us, but the DNA sequence is, at least in my analogy, on the trace, of a trace of what the real “gene” is.

To generalise

It seems to me that question of identification must lie at the centre of any abstract model of evolutionary framing. An evolutionary model or framing is one that helps us analyse why a specific recognizable object persists. This is necessarily circular as we likely recognise an object precisely because it is in some sense persistent. The choice to focus on a particular object is contextual, and semantic, ideally supplemented by an analysis of the extent to which other framings can be neglected. The great success of the evolutionary synthesis rests on the fact that seeing DNA sequences as genes has enormous explanatory power. It is not complete but it gets us a long way.

It’s not clear to me that replication is a necessary component of an evolutionary model. Differential persistence amongst a collection of objects seems enough to me. Change or variation within objects is not strictly necessary, but systems without it would appear to be rather boring. Both starting a model with diversity, as population genetics generally does, or starting with consistency and then incorporating change, as is the case for evolutionary optimizations, are both feasible. In those cases where there is a Platonic conception that provides the means of recognising relevant objects, they will be formally equivalent through a time transformation. The assumption of a universal common ancestor as a way of identifying cognate genes is one example. A more trivial example is whether to assign time point zero in an computational optimisation prior to or after the first introduction of variation.

If replication is part of the model it can take many forms and is not necessarily linked to variation. Variation can take many forms. Differential persistence – selection – may take many forms. Differential persistence is linked in a complex way to those things outside of the objects of interest, broadly the environment, but the environment is also a set of evolving systems. All of these may be sequential or continuous or some combination of those. Some combinations of replication, variation, differential persistence and environment will be stable, some presumably will not. The interesting metaquestion is the characteristics of such systems that lead to different forms of behaviour and interesting, or useful, dynamics of the system as a whole. How can recognizable attributes of the object be linked to survival and can lineages be constructed? If recognition and identification is at the centre of these models then an inability to reconstruct lineage will appear the same as the loss of the object in question.

At the highest level, this kind of model does become tautologous. That which persists, persists, so long as we can recognise it. It is through the characterisation of objects and the dynamics of the system that we can return to scientific models of specific systems. These are all models, as soon as we focus our attention on one set of objects we recognise we are neglecting the full complexity of the system. But it does seem productive to consider how those different models can be compared to each other, and how we can understand the dynamics of differing systems. That in turn offers the opportunity to turn that back around and ask what are we missing, if we seek to focus on a different element (the persistence of a demic narrative, or of a particular marker in a microbiome), how does that change our view?

This may in turn allow us to understand how multiple evolving systems interact. E.O. Wilson’s Consilience remains probably the largest scale attempt to frame everything in evolutionary terms. Where Wilson fails to my mind is in finding a way to tackle the complexity of interaction between the layers he talks about, as though chemistry can neatly be separated from biology, biology from psychology, psychology from sociology and on and on. Nonetheless he makes a strong point about the success of these approaches in tackling specific classes of problem, those that respond well to a reductionist approach. What is missing is a means of putting those back together that works with the complexity of interactions between what were never really layers in the first place, merely convenient categorisations for our identification of objects.

I’m sure this work has been done somewhere, but I’m slightly at a loss as to which discipline it would be in.

1. Strictly speaking some form of splitting or replication is required for the model to continue in the long term if some cultures are extinguished but that’s a consequence not a central point.

Telling a story…

This is the first pass at an introductory chapter for a book I’ve had in my head to work on for a long time. The idea is that it relates some of personal history shifting from my grounding science towards the humanities, while interleaving this with a survey of the theoretical work that develops those different perspectives. This is just a first draft written on a Sunday afternoon. Comments, as always, welcome.

“As a patient I struggle to relate to survival curves…”

This is a book about narratives, perspectives, and knowledge. It’s a personal story of re-thinking the way that I think, as well as an attempt to find a way through the various, apparently irreconcilable, ways of understanding how we can know. Others, far more qualified than I, have attempted this in the past with differing degrees of success. In some ways this is more a survey of those attempts than anything new. What is perhaps different from those previous efforts is that I’m also telling a story of how I have trodden a path of my own through these differing ways of thinking about thinking, knowing about knowledge.

The idea of telling a story, one that traces a progression from being a scientist to being some sort of humanities scholar, is that a story of my own changing perspective is a different way of trying to bridge the gap that those others tried to fill. The gap itself is referred to in many ways, and from many different perspectives. Snow’s Two Cultures is the common touch-stone but it appears in many differing versions. The gulf of incomprehension between natural sciences and engineering on one side and the humanities on the other, the modernist consensus built on facts of nature versus post-modern perspectives rooted in context, traditional versus de-colonizing versions of history.

The battles between these incommensurate world views are also celebrated, sometimes with the verve, and steadfast adherence to the “correct” version, of historical re-creation societies. Shapin and Schaeffer’s Leviathan and the Air-pump, a broadside against the core tenets of the scientific method, or perhaps rather an over-simplified version of the story we tell about it, is brandished as a badge of identity for those who wish to associate with the Edinburgh School. The Sokal hoax, a puncturing of the opaque and circuitous arguments and the lack of quality control that underpins the sociology of science, is celebrated in science circles as the triumph of the skills of the scientific viewpoint despite being a rather disingenuous and mis-represented tale that fails to engage with what the editors of Social Text were trying to achieve.

The battles are of course not limited to the academy. Is the role of journalists to “report facts” or to provide interpretation? Are media sources neutral? Can media sources be neutral? The rise of extremism, religious and racial violence, our apparent devolution into a “post-fact” world is to be laid variously at the feet of post-modernist critical theory or neo-liberal economics depending on your perspective. The question of how we come to know what we know, and how it can be tested and validated, is crucial at the same time as, indeed likely because of, the way technology is allowing us to construct a world in which we only interact with those who agree with us.

It may be crucial, but this is a problem that has not been solved despite millennia of work from great thinkers. We have a much richer understanding of how we can be wrong about what we know, from errors of logic, to the untrustworthiness of our perceptions, to the choice of foundational axioms, believed to be eternal truths, that turn out to be simply the product of history and culture. We know something about how these systems are inconsistent, where their weakest points are, but despite heroic efforts there has been little success and putting them together. We have no clear ways to show whether knowledge is reliable, let alone true. How can telling a story help?

My goal is much more modest. Rather than asking whether we can tell whether something is true, perhaps it is more useful to think about the ways in which we can test our knowledge. Can we become better at identifying whether we are wrong, and is this more productive than trying to pin down the transcendental? If both god and the devil are the in the details, might it nonetheless be easier to find the devil?

This is a question unashamedly rooted in an empirical perspective, one rooted in my formative history as a scientist. I will position the scientific method as a cultural practice that, at its best, is an effective means to prevent us from fooling ourselves. But it is for all that a practice with its limitations, and one stymied by a set of assumptions that are difficult to support; that a claim can be unambiguously communicated, that an experiment can directly test a claim, and that the connection between claim and experiment is clear and objective. When the scientific method fails internally, it is because these assumptions have not been fully tested.

I will argue that science is effective at helping us to reliable answers to well framed questions. But those questions are often limited. Sometimes by our capabilities as experimentalists, sometimes by an inconsistent or incoherent theoretical framework. Studies of reproducibility, of communication of science, of its institutions and power structures inform us how the questions we ask are biased, inappropriate or confused. The best of these studies are not pursued from a scientific perspective.
Science is terrible at probing its questions critically. By contrast, humanistic disciplines are built on critique. Indeed the inverse criticism might be made of the humanities. It is a brilliant set of tools for developing probing questions through shifting perspectives, undercutting power, asking how the same issues can be approaches from an entirely different direction. The humanities are not so good at providing answers, or at least not ones that can be tested in the same way as the results of scientific experiments. But both provide a tool set for rooting out certain – different – kinds of errors.

In telling my own story, of a journey from a scientific background to a more humanistic perspective, my aim is less to build a system to reconcile two world views than to offer some experience of attempting to apply both. In alternating chapters I will tell the story of how my perspective shifted over the course of 15 years, and seek to survey the scholars that have developed and studied those perspectives.

The epigraph that heads this chapter, half remembered from a tweet from some years back, offers one kind of perspective on that journey. I remember it striking me as pertinent as my views changed. To begin with, simply the idea that scientists needed to communicate better to those who could benefit, that a patient perspective was important, and often missing from medical studies. Later, as I saw how power structures in the academy biased research towards the diseases of wealthy Americans, I saw it as posing a deeper question: how are the priorities of research set, are we even asking the right questions?

Questioning the questions leads to a deeper concern. The system of medical research is set up with the aim of keeping us honest as we distribute scarce resources available for medical treatment. The randomized control trial, the gold standard of medical research, is set up very carefully to ensure our best chance to get a reliable answer to the question: “all other things being equal does this intervention make the patient better/live longer”. These are the source of the survival curves. And yet this question “all other things being equal…” is never the question that a physician asks when recommending a treatment. Their question is “what is the best advice I can give to this particular person under these very particular circumstances”, a question that our entire edifice of medical information is extraordinarily badly configured to answer. Are we asking the wrong questions at the level of the entire system?

But how can we tell what is best for the individual patient? At the individual level placebos effects can matter, a rare side affect can be fatal – or curative – and the question of whether the patient “is better” is a largely subjective concern. It is not difficult to find people who will swear blind that homeopathy or acupuncture “works for them” regardless of a swathe of evidence that it has no effect in randomized trials. Personal testimony is suspect, memory is unreliable, and yet we reject them in one case, but also use them as the basis for our scientific studies.

Ironically the epigraph tells the story here. On actually checking its provenance I find that it is both much more recent than I thought, far too recent to have been part of my early shifts in thinking, and not even a direct quote. My memory is unreliable, a product of the narrative I have created for myself. The story I will tell is just another narrative, and an unreliable one at that.

But unreliable need not mean un-useful. If my task is to get better at testing my knowledge, then it can serve both as metaphor, narrative thread, and reminder of the ways we can go wrong. The process of questioning, testing, and the way we can use external perspective to do that is at the core. Whether it is the half-remembered words of an otherwise un-represented group, or the perspective of a technical system like the twitter archive, it is a cycle of testing, stepping sideways and testing again that lets us understand the limits of our knowledge.

Darkness/Dream

I had a dream, which was not all a dream.
The bright sun was extinguish’d, and the stars
Did wander darkling in the eternal space,
Rayless, and pathless, and the icy earth
Swung blind and blackening in the moonless air;
Morn came and went— and came, and brought no day

I have a dream…
I have a dream that one day this Nation will rise up
And live out the true meaning of the creed,
“We hold these truths to be self evident,
That all men are created equal”

And men forgot their passions in the dread
Of this their desolation; and all hearts
Were chill’d into a selfish prayer for light:
And they did live by watchfires— and the thrones,
The palaces of crowned kings— the huts,
The habitations of all things which dwell,
Were burnt for beacons; cities were consum’d,
And men were gather’d round their blazing homes
To look once more into each other’s face

I have a dream,
That one day every valley shall be exalted,
And every hill and mountain shall be made low,
The rough places will be made plain,
And the crooked places will be made straight;
And the glory of the Lord shall be revealed
And all flesh shall see it together.

I have a dream…

A fearful hope was all the world contain’d;
Forests were set on fire— but hour by hour
They fell and faded— and the crackling trunks
Extinguish’d with a crash— and all was black.

And this will be the day —
This will be the day when all of God’s children
Will be able to sing with new meaning:

My country ’tis of thee,
Sweet land of liberty, of thee I sing.
Land where my fathers died,
Land of the Pilgrim’s pride,
From every mountainside
Let freedom ring!

The waves were dead; the tides were in their grave,
The moon, their mistress, had expir’d before;
The winds were wither’d in the stagnant air,
And the clouds perish’d;
Darkness had no need
Of aid from them—
She was the Universe.

Text: Lord Byron, Darkness; Dr Martin Luther King Jr, I have a dream, Lincoln Memorial Speech, August 1963, including texts from the U.S Declaration of Independence, Handel’s Messiah (King James Bible), My Country ’tis of Thee.

This is the text that I compiled for a piece of choral music that I never finished writing. I’d always been intrigued by the parallels between Byron’s poem Darkness and Martin Luther King Jr’s I have a Dream speech. The musical intent was to build a kind of a capella cantata on a highly disguised version of the hymn tune which is (ironically) used for both My country ’tis of thee and the British national anthem. The juxtaposition of the two texts seems apposite as 2016 draws to a close.

There are a range of issues with the appropriation of this text. Including that the selections from the King speech largely avoid the issue of race, which is central to the text. In part this is due to the copyright issues noted below. In part it is because to appropriate those texts feels more inappropriate to me. The core reason for choosing those portions is that my reading of the text is that King is (in part) challenging Anglo-American white culture with our own texts, literally calling us to “live out the true meaning” of texts which are central to our culture. His framing was one of hope, that we might choose to do that for a common good. Despair at our collective failure to do that, our “selfish prayer for light” is the context of my re-framing.

Observation on copyright: As with everything on this site I waive any copyright or other rights in the assembly and arrangement of this text with a cc0 waiver. The Byron poem Darkness is well and truly out of copyright and in the public domain. The King speech is not and the full text and audio is (c) of The Estate of Dr Martin Luther King Jr. Here I claim Fair Use and Fair Dealing on the following basis:

  1. The use is transformational, and additionally amounts to commentary/criticism
  2. The work is published and in common circulation in both authorised and unauthorised forms. The elements of text I use are quoted widely in and out of context.
  3. My use is of a very small proportion of the original material from the speech and is primarily of third party public domain material incorporated into the speech text. Less than 2% of the original speech text is used here.
  4. The use is non-commercial and does no market harm to the original speech text or audio

First steps…

Amongst all the hot takes, the disbelief and the angst the question many of us are asking is what to actually do. I don’t have answers to that question, or rather I have lots of answers and no clarity as to which to choose. An inventory of resources reveals this blog, a substantial, if not massive, social media following on Twitter and some degree of influence associated with that. And money and the network of influence that goes with being moderately affluent and middle class. Not the 1% but definitely the 10%.

This is a list of concrete actions so far and immediate next steps:

Web presence and online:

  • This site now has a LetsEncrypt certificate so https:// should work and I believe I’ve ironed out the kinks
  • I’ve turned off and removed all Google Analytics scripts
  • What I write here will become more political, what I retweet and post more so
  • TODO: Should I move off Disqus as the commenting platform? Do I need more robust hosting? Should I be using a US host at all?

Support

  • Made donations to ACLU, Planned Parenthood and the Southern Policy Law Center
  • TODO: Identify other organisations, particularly in France. Figure out and budget for regular donations
  • Joined the Guardian, subscribed to New York Times
  • TODO: Is there a conservative leaning, continental European, English language media organisation with its own reporting team? Review NYT based on its breadth of reporting and prioritisation.

As per the weekend post, none of this is to claim credit. It’s a return to the original purpose of this blog, which was to think out loud, with the recognition that it is only because I am privileged that that even makes sense. Each of the above represents a first step towards redressing the failure to do things I should have done in the past.

Some thoughts on safety pins

Like many people over the past week, and months, I’ve had some cause to reflect on what it is I do, and why. A lot of that circles around an issue that’s been troubling me for a while, how do you simultaneously acknowledge a personal and historical failure to act and credibly and coherently move to change that. How can I know when to challenge and when to shut up and listen – because its not always immediately obvious. There is perhaps a greater risk of challenging when it is in appropriate than the converse, but if the charge is to be a vocal and consistent ally then its a risk to be aware of. And this comes to a head with the question of safety pins.

For anyone who doesn’t know the story the idea of wearing a safety pin arose following the Brexit referendum in the UK as a way of signalling support for immigrants and opposition to racism. The idea was that it represented a commitment to the safety of people around us. It’s a simple and clever form of gesture politics. And it came in for a lot of criticism because it can be just that, merely a gesture. But I have been wearing a safety pin on and off since the referendum. I haven’t seen that many others. I haven’t had an occasion where the commitment it represents has been tested. But what it has done is provoke a whole bunch of conversations in the predominantly white, middle class, professional communities I’m a part of in the UK. That is one thing in its favour. A small thing, but a thing.

The idea of wearing one has suddenly resurfaced in the wake of the U.S. election. In the confusion and dismay people like me will reach for something, anything, to do. It’s a gesture, and again there has been a lot of criticism. What does it mean? Is it a nice gold ally star to assuage our guilt? Isn’t it a bit late? What message does it send to those people who’ve been dealing with abuse and discrimination for years, or rather centuries? That now we’re onside? After everything’s gone to shit and maybe just maybe it finally has consequences for us comfortable, middle class, white, professionals?

Already, also we can hear stories of people wearing a safety pin standing by while people are abused or frightened, and worse, stories of it being used to trick people into dangerous or abusive situations. The lack of courage I can relate to, would I really step in to a situation? It’s nice in theory but I’m not a brave person. In a very real sense the pin is a symbol of all the failures of my class that have led us to this point. The criticism, and yes the derision and anger that it has provoked as a symbol is just one part of what we need to hear. It is not a gold star from teacher, it is the reverse.

At the same time, as a gesture, a symbol it works in provoking the conversations we need to have. In our nice comfortable settings people ask questions, and perhaps we have the opportunity to amplify and transmit some of that criticism and anger, to make those of us who need to be a bit less comfortable. It is also a personal reminder. For those of us for whom this is new, who have not been there in the past, and are privileged enough to not be dealing with discrimination and abuse on a day to day basis, it is easy to forget. Indeed it is the easiest coping strategy. Having a reminder there to keep doing better is helpful. A reminder of the commitment to step up when necessary.

So this is where I’m at right now. My current plan is to continue wearing a safety pin. I am going to keep thinking about how to apply this guide’s approach to dealing with abusive situations and try to have a plan when such a thing happens. I will continue to use the pin to start conversations and raise issues and I believe it has a real value for that in the places I go and with the people I meet. And I will keep listening to and reading the criticism and anger. And seek to reflect on it and to transmit and amplify it. At the core of that criticism is the charge to get on and do the damn work, not expect a pat on the back. So that pin isn’t a merit badge, it has to be a commitment or its worse than nothing at all. At the moment at least I feel like as a symbol of that it has some value.

Licensing, ethics and patient privacy

Following one of “those” conversations on twitter, the ones where the 140 character limit just isn’t enough it seemed worth writing up a quick post. It’s that or follow the US election after all…

Richard Sever of Cold Spring Harbour Press posed the following question on:

…to which my answer was:

You can follow the conversation via the links above but here I just wanted to flesh out the disagreement and why I think this matters. This is a general class of a problem that we often see, where we reach for licensing as a tool to reassure or solve some sort of complex normative, and often ethical, issue. I’ve always had a problem with this because normative issues are ones of community, and therefore ones that we need to take responsibility for. Licensing (and other legal tools) place responsibility elsewhere, granting control to another community, in this case judges and the courts. Sometimes this is necessary, when there is an expectation that the interests of different communities will need to be arbitrated, but in the case of ethical issues I feel these are internal issues and ones that should be determined (and if necessary sanctioned) internally.

In the specific case here we’re talking about an identifiable image of a child, where the parents had apparently given permission for the image to appear “in a scientific journal” but hadn’t realised that this would be widely available. When they did realise this some years later they withdrew permission and the article was retracted, with the image blacked out in the retracted version. This is unfortunate and there are issues with the specific story but in some ways its a story of things working well. Permission was sought and given, when it was revoked the image was removed and the issue noted.

For me, what is at core here is the issue of informed consent and the degree of assurance that can offered to participants that the commitments made to them, particularly on issues of privacy, can be met. If there is data, including images, that should be restricted from public access then that needs to be made clear, but above and beyond that there needs to be clear communication about the risks of the access control that are put in place breaking down.

Why am I focussed on access control when this was an Open Access article? For me, the issue was a lack of appropriate clarity in the consenting for the use of the image. If the participant’s expectation is that an image or data will only be made available to professional medical staff or to researchers, then it should never go in a journal article of any kind. Journal articles are publicly accessible, in different ways, we cannot guarantee to prevent a journalist who has access to a research library taking a copy of an image from a print subscription journal and using that in an article. If the concern is public view or commercial use then once its in a journal we cannot guarantee that will not happen.

You might argue that the risk is much higher with online CC BY licensed article but ethical judgements err (sometimes radically) on the side of caution for good reason, because they are intended to deal with low probability events that can lead to substantial harm. I would argue that unless a participant explicitly consents to allowing liberal re-use then such data (including images) needs to be properly access controlled.

As John Wilbanks has argued for many years, copyright licenses are a very poor means of protecting participant privacy. There are far too many ways for it to fail, from people ignoring it, to technical systems failing to recognise a separate license for an image in a larger work, to the many conditions under which the license simply doesn’t apply because a use falls under Fair Use or Fair Dealing. Both to establish trust and meet ethical standards it is necessary to link access to contractual requirements that bind the user to limit their downstream uses in ways that licensing can not.

Now Richard was arguing from a different end. Without presuming to put words in his mouth his concern as I understood it was “given that key data needs to go in ‘the paper’ how can we best give participants assurances that make them comfortable with providing consent”. In the end I think we agree that access to sensitive material needs to be limited. My view is that copyright licensing provides little to no assurance of the type needed. Licensing is also implemented by players outside the control of the organisations responsible for consent.

Richard, I think would argue that having open licenses could be discouraging. In both cases I think we end at the point that where material is sensitive and where access controls are deemed necessary, whether by an IRB or by the participants themselves then an ethical approach requires appropriate safeguards. And for me that means robust access controls.

There might be a case in which participants consented to use for publishing, but only under a restricted license. I actually find this a little implausible. The consenting should be based on clear limitations on use, not on copyright licensing, for the reasons noted above about the limitations on enforcing licenses. Nonetheless its at least a theoretical possibility. For me, the importance of having a cleanly and consistently liberally licensed public record is enough to say that under these circumstances such materials should be kept separate to the formal public record, linked from it but not formally part of it.

At scale, re-use requires reasonable certainty and at the moment the wholescale re-use of images, even from Open Access literature runs into problems due to the embedding of differently licensed images and no consistent way of marking this. This is actually an inverse of the problem as above. Just as licensing can’t give sufficient assurances that inappropriate uses will be blocked, poorly expressed licensing doesn’t give clear assurance to users, particularly at scale that use is appropriate.

For me the argument from both ends, that a consistently licensed clean corpus has enormous value, and that licensing is not the right tool for carrying out ethical responsibilities, reaches the same point. If participant consent does not include liberal re-use then material should be maintained separately to the public, published record under appropriate access controls that limit uses to those that have been consented.

 

The Goods in the Scholarly Marketplace

This a set of notes for my talk at Duke University this week. It draws on the Political Economy of Publishing series as well as other work I’ve been involved with by Jason Potts at RMIT amongst others. The title of the talk is “Sustainable Futures for Research Communication” and you can find the abstract at the Duke event page.

The video is now available along with the slides. The lecture capture didn’t get such a clear view of the slides so you may want to bring both up and play along.

Sustainability is the big discussion point in Research Communication. Will journal’s survive? Are APCs the only credible route to sustainability? Or will they lead to inevitable destruction of journals? Will monographs survive? Scholarly societies? What, as is asked over and over again, is the future of the library? Of the institution? This week, frankly, of the nation state?

Figure: Two responses to the same tweet. Differing views on what “sustainability” is all about.

And into this space we see perspectives from economics and political economics start to filter in. Martin Eve and David Golumbia are concerned about taking labour – and implicitly – labour theory seriously. From within the publishing industry the perspective of consultants – mostly financial analysts rather than economists – looks to trends and balance sheet calculations. Scientific Reports up, PLOS ONE down, build a story out of the data. As someone who has seen the inside those stories are almost always roughly right on the trends and almost entirely wrong on the underlying reasons for them.

Financial analysis is important but it is a weak grounding to understand Research Communications. Here I agree with Golumbia and Eve, as well as Jason Potts amongst others, that “OA advocacy” but also industry advocacy “[…]lacks robust critical grounding for its propositions in credible Marxist or socialist economic theory” or indeed in any but the most facile theories of market operation. There are sound political reasons for this. Articulating a clear dichotomy between corporatist, self-enriching encumbent players, and the selfless and public spirited intent of scholars to communicate has been an affective mean of driving the debate.

The nature of “scholarly goods”

This dichotomy is implicitly framed in economic terms. The private goods of publishers are growing at the expense of the creation of public goods by scholars. Knowledge, must surely be a public good. The Jefferson quote we often use could hardly have been shaped better to make the case for which quadrant of Ostrom’s diagram knowledge belongs in. But its also obviously more complicated than that. Jefferson fails to mention to McPherson in his letter that you need your own candle to accept his flame, or at least a taper or lamp. That you can only receive that light by being close to the candle. The flame may be non-rivalrous but it is still exclusive – there remain hurdles to gaining access.

He who receives an idea from me, receives instruction himself without lessening mine; as he who lights his taper at mine, receives light without darkening me.

Thomas Jefferson, Letter to Issac McPherson, “No Patents on Ideas,” 13 August 1813.

As with the candle flame, so with knowledge. The aim of Open Access is not to simply nationalise the private goods of the artist formerly known as the publisher. To convert private goods to public goods. History and economics tell us this is not a generally successful path. Our goal, and indeed I would argue that a core goal of scholarship itself, is to make the exclusive goods made within small groups of scholars more public. Since Robert Boyle railed at alchemists, the importance of communicating scholarship has been central to science, and since Plato wrote down Socrates’ complaints about the dangers of writing scholars have debated how new technology affects that communication. In that sense the idea of “public making” and the tension that immediately arises from the question of “to which publics” has always been the central issue of scholarship.

Groups make knowledge

It is easy to fall down the rabbit hole of philosophy of knowledge. So I’m not going to seek to define it here. I will however assert that private experience is not knowledge; that knowledge is a characteristic of groups. In that sense it is born exclusive, as it is held within the group. The process of public-making is both an investment – it requires effort – and a risk – it reduces exclusion. There must therefore be a process of exchange, the group receives something back in return for this public-making. The process of exchange, as knowledge is absorbed and tested and refined by other groups, creates collective (or more public-like) goods that benefit a wider range of groups. New ideas unlock problems, new tools are built that can be generally used, research is applied by others to create wider benefits.

However classical economics, in particular the work of Mancur Olson, tells us that the value of collective goods is not a sufficient incentive for their creation. In large systems it is more in the interest of contributors to try to free-load on collective goods, and in a classic tragedy of the commons, the collective good is created at a lower than optimal level, if at all. It is not at all an accident that knowledge making groups make an exclusive good. Buchannan’s work on the 60s identifies goods with these characteristics as the ones that make groups (specifically clubs providing facilities in his work) attractive to members, and therefore sustainable. We therefore have two high level questions to answer if we want to address sustainability: how to sustain the groups that make knowledge, and how to make it in their interest to invest in making their exclusive “club” goods more public-like.

Institutionalising the scholar

These may not seem like the questions that I suggested I would answer at the beginning but that is deliberate. These are the foundational questions we have to address. Olson and Buchannan as well as Elinor Ostrom offer us some solutions. Olson notes that one solution to the challenge of collective action is to make it compulsory. He is thinking of taxation systems, but the effective compulsion for professional scholars to publish provides a similar mechanism. Ostrom notes that while in many situations facile economic analysis shows coordination is “impossible” – Hardin’s tragedy of the commons – that in the real world institutions can evolve so as to help communities solve collective action problems.

The institution of the research university in which a scholar has resources to work with, gains membership to a set of interlocking “clubs”, and in return is expected to publish is an example of this kind of evolved institution. By “institution” I don’t just mean university, but more broadly Ostrom’s sense of: “the prescriptions that humans use to organize all forms of repetitive and structured interactions”. Merton’s norms are an institution. But so are Mitroff’s anti-norms, the observation that the success of the research enterprise is in part dependent on behaviours exactly opposite to the ones that Merton prescribes: particularism, individualism, self-interestedness and organised dogmatism. To the extent that we recite standards of behaviour, and to the extent that we follow it, or indeed something else, culture is also an institution. This is where the economics intersects with the politics.

This is also where we finally get to publishing (and indeed to libraries). The publishing system is also an institution (or set of institutions) that function as infrastructure. The publishing system provides collective goods: access (albeit to a traditionally exclusive collective), discovery, archiving. The purpose of the institution that is the university is in part to provide a mechanism for compelling scholars to engage with that publishing system so as to provide that collective good. Not by any means the whole mechanism, nor is that its only purpose, but its a part.

Institutionalizing the publisher

The obvious follow-on question is what incentive do publishers have to engage with this collective good production? The obvious answer we would traditionally give is: to extract rents on monopoly rights to generate private goods. But that hasn’t always been the answer. In fact for most of the history of scholarly publishing its been a loss-making enterprise. We need to understand a bit more about that history and the trends that underly it before giving a definitive answer. I am borrowing a lot here from a workshop I went to which is reported in this set of papers in Notes and Records of the Royal Society.

Journal publishing arose out of clubs. First the clubs of emerging national academies like the Royal Society and then out of an increasing set of scholarly (including disciplinary) societies. The term “journal” actually dates from the 19th century and originally applied to serial publications that mixed both research and public interest material as a way to break even. Some of these were money making enterprises for the researchers involved but almost universally pure research and scholarly society journals lost money. They engaged publishers, or rather printers, by paying them for their services and the community subsidized the cost, both directly and through personal and institutional subscriptions.

In the mid to late 19thC relatively few researchers were paid by universities. Only those with endowed chairs were free to focus on research. Many active researchers were independently wealthy, some had wealthy patrons. Many, famously including Charles Darwin, interacted widely with correspondents from beyond the professional research sphere, including the full range of what we might now call citizen scientists. But at the same time professionalization was growing, and membership of the appropriate scholarly clubs, was only growing in importance. The gentleman scholar had the resources to engage, and more often than not the social status to demand attention. The growing class of professional researchers needed to gain access to the club, and publishing in the right place, being read in the right societies was a key part of that.

A journal is a club

My emphasis on clubs is deliberate. When journals (and to some extent the same is true of the University or Scholarly Press) were small, and focussed on a specific, recognizable community then the losses could be absorbed. The collective goods of the journal were created and individual members (subscribers, authors, community members) gained through association and identity. This continues to this day. The enterprise of creating a new scholarly discipline, a new club, invariably involves the creation of its cognate (or is that eponymous?) journal, even if that “journal” does not take a traditional form. See Chris Kelty’s fascinating “This is not an article” for a parallel example of this outside the journal world. The sustainability of these “clubs” with their various tiers of membership and benefits can be understood within the framework of Buchannan’s analysis of club economics.

Buchannan is a classical economist so his models make some assumptions. Specifically that all the club goods are ultimately exchangeable for money and that sustainability means an achievable equilibrium where positive externalities balance negative externalities. We are dealing with only partly exchangeable goods and equilibrium is a long way distant. Nonetheless there are useful lessons to be drawn about possible steady states. In particular Buchannan’s model turns on the question of what happens as the club changes in size. He shows that, where there is friction in access to the club goods – where it is not perfectly non-rivalrous – that this places limits on club size. In the print journal world friction arises in access for authors to the pages of the journal, access to the attention of readers and to the expert criticism of the community.

These trade-offs work when the community is small. Buchannan, Olson and Ostrom all tell us that negotiating these kinds of subsidies to support collective benefits can work when communities are small, homogeneous and have common goals. And to varying extents this was true of the various research communities up until WWII. But then they grew, an explosion of funding and an expansion of the institution of the university both geographically but also within traditional national centres of western scholarship, lead to a massive expansion. And arguably the system has been broken ever since.

The problem of scaling

While the set of readers and authors and funders of a journal are highly similar then there is a common body of culture and knowledge. Amongst other things this commonality reduces a number of costs, specifically of defining whether any given contribution is “acceptable”. Because there are common conceptions of what is within scope the internal sense of “quality” is cheap to determine. The “journal club” is contiguous with a “knowledge club”. This helps to define why society run journals are generally cheaper than commercially run ones, despite the claim that a market economist would make that the commercial run operation should be more competitive and efficient.

As we scale up the system a number of coping strategies emerge. One is specialisation, creating more and more specific communities and journals. This works and is a large part of the history of 20th century scholarly publishing. The price is paid in the creation of silos, an opportunity cost that is hard to delineate in which not only does cross fertilization not take place, but the modes of knowing and testing, the individual culture of each community becomes more and more incompatible with that of others.

If one approach is scale down, then the other is scale out. Scale up the journal to encompass more than one knowledge community. This also has potential to work, it grows the subscriber base and it can create prestige for the journal, attracting authors. But there’s a problem. Because the knowledge community is no longer singular or coherent the cost of determining inclusion becomes externalized. Because the club is no longer capable of determining a single standard a standard needs to be imposed. It turns out that PLOS ONE and Nature actually have the same problem. At PLOS ONE (and Scientific Reports) a small army of people are required to do the process of checking whether articles meet the standards. The most successful areas within PLOS ONE are those where there is a community (paleontology, neuroscience, some parts of genetics) that can do this policing, but across the broad remit of the journal its not possible to rely on academic editors and referees to get across everything and staff need to be brought on to manage that.

At Nature a small army of people (different job titles, different qualifications, but nonetheless) do exactly the same job. And again, they appear on the balance sheet. The complaint most frequently leveled at Nature is that “it doesn’t publish work of type X”. What’s actually happening is that individual editors within Nature have their own views, they create a community, one that is fundamentally predicated on prestige and excitement of being associated with the masthead. But the gaps, and indeed the failures of judgement, illustrate the exact same costs and challenges as the occasional dodgy papers that slip through the PLOS ONE system.

Separating equilibria and the luxury APC market

Another thing happens as the journals spread their scope. The attraction for membership shifts from being membership of the knowledge club, a form of identity politics if you like, to being prestige. Where there is high consistency between knowledge styles, a member of the club can reach an opinion about the quality of a given article. But the author of an astrophysics article in Nature cannot judge an article on neuroscience in the same journal, nor can the astrophysics editor judge that same article, at least not by any means that the neuroscience community would recognize. They can only judge the fact that it appears in that journal. The masthead is a proxy for something but it cannot possibly be a consistent characteristic of the work described in the article.

This might not be a problem in and of itself. In our manuscript on the issues around “Excellence”, we start by advancing the argument that the fact that the word is meaningless might not matter as long as it serves a useful political purpose in aligning the research community. But as we go on to show in that context the concept of “excellence” is actively harmful. It leads to hyper competition, and based on well established findings from behavioral psychology of rewards, leads to performance of excellence, not the characteristics of research we want.

Even this might not be a problem from an economic perspective. Markets need signals and can operate just fine with proxy signals that don’t actually mean anything. These markets tend to be volatile but that in and of itself is not a problem. But, as with excellence, it means that the market focusses on the proxy. In the subscription world this creates the “must-have” journals that authors seek to get into and strengthens their hold on the market. It concentrates power and helps to make big deals work. It focusses the publisher’s attention on maintaining prestige rather than quality of the process (for instance it took Science seven years to implement the same level of QA on statistical soundness as PLOS ONE).

In a subscription world this is just another set of perverse incentives, a lack of alignment between what would provide the greatest collective benefit, and the interests of monopoly rent seekers. This is neither surprising nor wholly avoidable. But something much worse happens with the shift to an APC model. There’s a lot of pretty naive economic analyses of this shift. Most recently you can see the interchange between David Schulenberger in a paper commissioned by the US Association of Research Libraries and Rick Anderson on Scholarly Kitchen.

In what should by now be a fairly familiar one-two the disagreement between the two centres on the power of authors in the market. Schulenberger argues that authors have zero market power, because they have no choice on where to publish, and therefore an effective author-side market can not emerge. Anderson argues that there is sufficient choice by listing prestigious journals in a number of fields, essentially showing there is more than one for any given field.

This form of simplistic analysis, which to be fair I’ve been guilty of in the past, however is unhelpful. The answer one gets tends to depend on how you frame the question. And that in turn is usually determined by how sloppy we are in defining what the market is. We need to understand how the author-side and reader-side markets differ and what is being purchased. Then we need to understand how those markets can break down. There’s a lot of work to be done here.

I want to cut to the chase here though and point out that both Schulenberger and Anderson can be right. Authors can have both substantial choice and buying power and prices can still rise. This happens in markets with specific properties. Markets where prestige sits at the centre. Markets where there is a disconnect between product quality and price. And worst of all, markets where price becomes a proxy for prestige. The APC market that we’ve built has all of these characteristics.

Luxury goods markets operate in situations where there are customers driven by prestige and some proportion are “rich”. Prices rise because it is in the interest of sellers to identify who is “rich”. Think of all those shops in airports, or most recently for me a Tesla dealership on a pedestrian mall in Sydney. Those shops are not for us. They are for people who can afford to walk in and just buy something. Sellers price otherwise unexceptionable goods at high prices to identify purchasers who will pay those high prices. If price can be set as a proxy for prestige (think cars, watches, handbags, haute couture) then the seller wins.

The rational economic move is for the rich person to not play the game. But in the APC world we have perfect conditions for a run away luxury goods market. Prestigious institutions demand prestigious scholars publish in prestigious venues to build up prestige. In the past they paid through the nose for this through subscriptions. But bringing authors into the loop and allowing the concretion of both traditional prestige markers and price as a proxy is potentially toxic. And without delving into the details, the situation seems even worse in the monograph space.

A brief segue on data

It’s worth taking a short sidestep into considering data and data sharing. For better or for worse data publishing doesn’t have the same prestige factor in play. We don’t fight to get our data into Zenodo and when rebuffed head to Figshare or vice versa. This is good – the same run away luxury goods market isn’t emerging – and bad – the institutions that motivate data sharing aren’t there for many disciplines. Where they are they are tied to formal article publication.

Data “publishers” perform much the same economic function as article publishers. They provide an arbitrage between the needs of readers and those of authors. They identify and define knowledge communities. They manage the quality assurance processes that the community defines. The objects are somewhat different but there is no reason to expect the economics to be fundamentally different. Differences in their funding will be signals of historical differences or differences in the value that a community places on articles vs data.

The issue of prestige we have already noted. This means, at least in part, that data sharing infrastructures must be more coupled to knowledge clubs, a factor that is clear in the disciplinary focus of many mature data sharing services. Adoption of broad-based data sharing services is driven more by either direct author benefits to sharing or by mandates. If a judgement were forced it might be argued that sharing through Figshare or Zenodo gains less prestige or community membership than submissions to PDB, Genbank or model organism databases. This actually mirrors community perceptions of general journals like Nature in the early part of 20th century. Publishing in disciplinary journals was viewed as more important – the idea that something of general interest is of more value is actually quite recent.

The funding of data infrastructures is also a challenge, and more so because the lack of prestige reduces the opportunities for creating commercial interest through monopoly rent opportunities. There are some examples, but they are generally residual monopolies from the print world. The Chemical Abstracts Service of the U.S. American Chemical Society stands out as an example. Other resources have gained the status of being two important to fail. The World Wide Protein Data Bank is an example of an infrastructure that is effectively top-sliced funding, albeit through the somewhat unstable mechanism of uncoordinated grants from a range of national funding agencies. Europe Pubmed Central is supported by a consortium of funders through a formal mechanism brokered by the UK’s Wellcome Trust.

Olson’s three options and the story of Crossref

These different stories map onto the paths that Olson described in the 60s as the means by which groups could solve collective action problems. Above a certain size he saw only three options. First is a kind of technicality, where the community is structured so that a small number of players dominate the space. Europe Pubmed Central is an example of this, the Wellcome Trust and a few other players, originally restricted to the UK could get around a table and just decide to act.

The second path is one where there is a non-collective benefit that contributors receive in exchange for supporting the collective benefit. Subscriptions are the traditional version of this, but membership models of all sorts generally involve a benefit. Sometimes, as I would suggest is the case for the Open Library of Humanities, that benefit is being seen to provide leadership, demonstrating a progressive stance. There will be more models like this in the future but there are questions about how they can scale – its not a cool kids club when everyone is a member. The Cambridge Crystallographic Data Centre is an example that tries the square the circle, providing some subscription access benefits while making most data available in some form.

The final approach is compulsion. At the nation state level general taxation funds collective goods and we (or at least most of us) don’t get a choice about contributing. Access to the Chemical Abstracts Service is pretty much a requirement of running a serious chemistry operation, one that the ACS protects by trying very hard to ensure its also a requirement for certification for chemistry courses. Regulatory capture is also a good strategy, as regulated professions like medicine and the law can attest. Standards, pace Malamud, operate in a similar manner.

The story of Crossref, the provider of article DOIs and de facto home of scholarly bibliographic metadata is instructive. Crossref was started essentially by fiat by a small number of publishers. In practice between five and nine publishers dominate the market. The heads of houses (they have been known to call themselves that on occasion) will meet and agree certain things. The setup and early funding of Crossref was one of those. This is Olson’s first option, effective oligopoly.

Once setup, Crossref offered a membership benefit. The growing infrastructure of DOI referrals and the improvements in article discovery led to traffic to publisher websites. And those numbers could be sold back to subscribing libraries. Both the ability to assign DOIs and the traffic are non-collective benefits of membership, and ones that increase in value as the collective good, the growing pool of public metadata and infrastructure to use it, grows in turn.

Finally today, membership of Crossref is effectively compulsory for any serious publisher of scholarly articles in STM, and increasingly for book and HSS publishers. It’s just part of the costs of doing business. A compulsory, tax-like, part of the system. Now compare the growth of ORCID. ORCID was also started largely because publishers decided it was a good idea. A few funders came on board but it was really publishers who put the money in up front. Gradually the funders have come in (again, really there are maybe 20 really important funders at a global scale), but the institutions? Institutions have been rubbish, despite actually being the place where the greatest benefits probably accrue. There are just too many to get past the collective action problem. We’re starting to see progress now, not because the universities have got their act together but because national level coordination is starting to kick in.

This story repeats itself pretty much every time the scholarly communications infrastructure needs an upgrade. Publishers move first, not because they have the most capital. The institutions do. Harvard could buy Elsevier and Wiley and T&F and SpringerNature and have cash to spare to throw in Calvariate or whatever they’re calling the rebranded ISI and PLOS for good measure. And not because the publishers have more freedom of movement. Publishers have share holders, or at least owners, who want a return. Many funders are in a much better place to call shots. The publishers move because they can, because there are few of them.

The future of the library

Which brings us to the university library and its unique place. What can we learn here? Maybe the IR infrastructure is a counter example? What do these models tell us about the success of institutional repositories? Well it suggests that those that started in a department (small, homogenous) will work better than those started at scale (Harvard and Southampton spring to mind). That those which provide a non-collective benefit back may be able to scale (Minho provides a great example of this with its competition with cash prizes for the best department). And that where those two do not hold that success will depend on effective compulsion (Liege, and the genius of Bernard Rentier obviously).

The library itself is a club good, membership of the university provides access to resources and people. The digital resources are non-rivalrous, but exclusive – even if that is exclusion is no access control. But Sci-hub and its inevitable successors blow an enormous hole in that. Privileged access to digital content will not survive as a sufficient value. Access to people, to expertise, now that’s different because that’s rivalrous. Proximity still matters. But services can also be bought elsewhere and the artist formerly known as the publisher is rapidly moving into that space. Neutral, non-biased advice in the interest of the researcher or broader university member is a good differential to start with, but focussing on how membership of the university is linked to access to the right kind of relevant expertise is key.

The institution itself, and the library within it, is pretty much at the worst of all worlds scale. Too large for collective action, too small and internally fractious to act unilaterally. Too many to solve the combined collective action problem. Collectives, thematic and geographic might provide solutions, and some actions the institutional leadership can take unilaterally and internally. But as currently disposed the university system globally is almost designed to prevent effective action.

The ways forward

The thing that the scholarly community has, if it can regain control of the institution is capital. I know it’s not proper to turn into a Marxist in middle age but this is fundamentally a question of control over deployment of capital and the position of labour. The labour sits almost entirely within the university and the scholarly community. Between libraries, scholarly societies, scholars and the institution.

But capital, or at least free capital, has been deployed largely by corporate interests and publishers in particular. Except that there is much more capital sitting with the funders and more importantly the institutions. Maybe it is time to challenge what the endowments and pension funds of institutions are supposed to be for. Maybe a trillion dollars is tied up, not being put to the use of the mission of these – often legally speaking charitable – institutions.

If you accept my argument that the institutional purpose of a university is to compel the publication of research outputs, as part of the contract of being a professional scholar, then the true Marxist would necessarily argue that the institutional capital, both financial and non-financial, must be liberated into the hands of labour. Ok, that may be a little radical. Particularly here in the Triangle.

There are some more practical goals. Ostrom shows us that institutions can grow in which collectives form collectives. If we focus on good community design and on shared principles of behavior, on building strong culture then our local communities, where we can solve the collective action problem, can band together at a higher level to solve the bigger problems. Stop seeing the university as a company with a CEO and see it as a collective of communities with a Community Manager. Leadership is not a dirty word in communities: leading from the front, leading from the middle, or leading from behind can all be successful. But leading by fiat will not be.

Still too radical? These communities are growing. Support them. The rise of collectives, OLH, Knowledge Unlatched, but also new forms of scholarly societies: the Research Data Alliance, ArXiv, FORCE11. Help these to grow, support them financially where you can, and with moral support where you can’t. We can work towards service definition. Let’s take the publisher’s at their word. They want to be a service operation: lets collectively define those services and our requirements. When I talk to the “big bad wolves” they are just as keen to have the space where they should bring their commercial sensitivities mapped out. But they can’t wait forever for us to make up our mind.

We need to reduce costs. I haven’t really talked about the structural aspects of costs in scholarly communications and how they are changing but there are opportunities for orders of magnitude reductions. To achieve that we need shared infrastructures. But for those we need funding models, which will necessarily be compulsory, and for those we need some form of shared governance. No taxation without representation. What does that mean in our space? And how can that be equitable. One of the other things Olson shows is that in delivering collective benefits the small will exploit the large. Time for the big institutional players to suck it up and play that role.

Finally we need to do everything in our power to sever the connection between prestige and price. By whatever means are necessary that link needs to be stomped on, disavowed or destroyed. Frankly, this one seems harder than the Marxist revolution. But if we fail at this we will reap the whirlwind. APC based models are growing. Separating equilibrium gives us two things, run away luxury prices, and a commodity product market for those that can’t afford the costs of luxury.

This isn’t just inequitable, and it doesn’t just damage the diversity of views on which scholarship depends. It is unaffordable at both ends. First because the luxury goods market will rise until it gets smaller and smaller, bankrupting the publishers that need to chase that extra differentiating piece of leather trim first, and then the research communities that are paying for it. But at the other end the commodity market is being supported by the rise, not of the predatory journal, which are frankly small fry in the scheme of things, but the predatory author.

The industrialization of fraud and plagiarism is already leading to an arms race which will rapidly raise prices. Just as we’ve seen at PLOS ONE and Scientific Reports, the potential savings from technology development will be swallowed up in layers of checks for author and referee identities, validating figures, checking for ever more sophisticated plagiarism. That’s the real story of the price rises – that the drop in the quality of submissions leads to an increase in the necessity for professional checks. And that is also unaffordable.

The optimal answer probably lies in smaller journals, and smaller, community based, data repositories, built on a common infrastructure that makes these cheap to run. An efficient market in services where innovation isn’t driven entirely by the possibility of a 100x sell out, but where commercial competition also plays its role. I don’t know if its possible to get there from here. But if we can then the big shift will be the role of knowledge communities, taking control of their own processes, within a shared framework that makes that easy to do.

To return to my original point. We want communities to create knowledge, and we want them to invest in sharing that to wider audiences in a way that is both principled and makes economic sense. The best way to do that is to provide platforms and infrastructures that make that investment both cheap, and provides good returns. What does the community, the club, get back in return? We will need new forms of economics to figure this out, the work on the economics of clubs and of commons, is still only beginning. But this will be the crucial knowledge we need to design new and improved institutions for the future.

Speculation: Sociality and “soundness”. Is this the connection across disciplines?

Italiano: network sociality
Italiano: network sociality (Photo credit: Wikipedia)

A couple of ideas have been rumbling along in the background for me for a while. Reproducibility and what it actually means or should mean has been the issue du jour for a while. As we revised the Excellence manuscript in response to comments and review reports, we also needed to dig a bit deeper into what it was that distinguishes the qualities of the concept of “soundness” from “excellence”. Are they both merely empty and local terms or is there something different about “proper scholarly practice” that we can use to help us.

At the same time I’ve been on a bit of a run reading some very different perspectives on the philosophy of knowledge (with an emphasis on science). I started with Fleck’s Genesis and Development of a Scientific Fact, followed up with Latour’s Politics of Nature and Shapin and Schaeffer’s Leviathan and the Air Pump, and currently am combining E O Wilson’s Consilience with Ravetz’s Scientific Knowledge and its Social Problems. Barbara Herrnstein Smith’s Contingencies of Value and Belief and Resistance are also in the mix. Books I haven’t read – at least not beyond skimming through - include key works by Merton, Kuhn, Foucault, Collins and others, but I feel like I’m getting a taste of the great divide of the 20th century.

I actually see more in common across these books than divides them. What every serious study of how science works agrees on is the importance of social and community processes in validating claims. While they disagree violently on what that validation actually means all of these differing strands of work show that it is explicit and implicit processes within communities, supported by explicit and implicit knowledge of how they operate and who is who, and who gets to say what, and when. The Mertonian norm of “organised scepticism” might be re-cast by some as “organised critique within an implicit frame of knowledge and power” but no-body is arguing that this process, which can be studied and critiqued, crucially can be compared, is not dependent on community processes. Whatever else it is, scholarship is social, occurring within institutions – that are the product of history – that influence the choices that individual scholars make.

In the Excellence pre-print we argued that “excellence” was an empty term, at best determined by a local opinion about what matters. But the obvious criticism of our suggesting “soundness” as an alternate is that soundness is equally locally determined and socially constructed: soundness in computational science is different to soundness in literature studies, or experimental science or theoretical physics. This is true, but misses the point. There is an argument to be made that soundness is a quality of the process by which an output is created, whereas “excellence” is a quality of the output itself. If that argument is accepted alongside the idea that the important part of the scholarly process is social then we have a potential way to audit the idea of soundness proposed by any given community.

If the key to scholarly work is the social process of community validation then it follows that “sound research” follows processes that make the outputs social. Or to be more precise, sound research processes create outputs that have social affordances that support the processes of the relevant communities. Sharing data, rather than keeping it hidden, means an existing object has new social affordances. Subjecting work to peer review is to engage in a process that creates social affordances of particular types. The quality of description that supports reproducibility (at all its levels) provides enhanced social affordances. What all of these things have in common is that better practice involves making outputs more social.

“More social” on its own is clearly not enough. There is a question here of more social for who? And the answer to that is going to be some variant of “the relevant scholarly community”. We can’t avoid the centrality of social construction, because scholarship is a social activity, undertaken by people, within networks of power and resource relationships. What forms of social affordance are considered necessary or sufficient is going to differ from one community to another, but this may be something that the all have in common. And that may be something we can work with.

 

FAIR enough? FAIR for one? FAIR for all!

English: Brussels Accessible Art Fair Logo
Brussels Accessible Art Fair Logo (Wikipedia)

The development of the acronym “FAIR” to describe open data was a stroke of genius. Standing for “Findable, Accessible, Interoperable and Reusable” it describes four attributes of datasets that are aspirations to achieve machine readability and re-use for an open data world. The short hand description provided by four attributes as well as a familiar and friendly word have led to its adoption as a touchstone for funders and policy groups including the G20 Hangzhao Concensus, the Amsterdam Call for Action on Open Science, the NIH Data Commons and the European Open Science Cloud.

At the FORCE11 Workshop on the Scholarly Commons this week in San Diego inclusion was a central issue. The purpose of the workshop was to work towards a shared articulation of principles or requirements that would define this shared space. To make any claim of a truly shared and global conception of this “scholarly commons” we clearly need to bake in inclusive processes. In particular we need our systems, rules and norms to remind us, at every turn, to consider different perspectives, needs and approaches. It is easy to sign up to principles that say there should be no barriers to involvement, but much harder to maintain awareness of barriers that we don’t see or experience.

The coining of FAIR was led by a community that want to emphasise that we need to expand our idea of audiences to include machine readers. As the Scholarly Commons discussion proceeded, and FAIR kept returning as a touch point I wondered whether we could use its traction for a further expansion, as a mnemonic that would remind us to consider the barriers that we don’t see ourselves. Can we embed in the idea of FAIR the inclusion of users and contributors from different geographies, cultures, backgrounds, and levels of access to research? And might something along the lines of making research “FAIR for All” achieve that?

As I looked at the component parts of FAIR it seemed like this could be a really productive approach:

Accessible

Originally conceived of as “available”, accessibility lends itself easily to expanding in scope to fit with this agenda. Can it be accessed without pay barriers online? Is it accessible to a machine? To a person without web access? To a speaker of a different language? To a non-expert? To someone with limited sight? There are many different types of accessibility but by forcing ourselves to consider a wider scope we can enhance inclusion. Many people have made excellent arguments that “access is not accessibility” and we can build on that strong base.

Interoperable

In the original FAIR Data Principles, Interoperability is concerned mainly with the use of  standard descriptions language and how resources make reference to related resources. For our purposes we can ask what systems and cultures can a project or resource interoperate with. Is it useable by policy makers? Can it be disseminated via print shops where internet access is not appropriate? Does it link into widely used information systems like Wikipedia (and in the future WikiData). Does the form of the resource, or the project, make it incompatible with other efforts?

Re-usable

For machine use of data, re-usability is reasonably easily defined. If we seek to expand the definition it gets more diffuse. This is more than just licensing (although open licensing can help) but also relates to formats and design. Is software designed to be multilingual and are resources provided in a form that supports translation? Are documents provided in editable form as well as print or PDF? While accessibility, interoperability and re-usability are all clearly related they give us a different lens to check our commitment to inclusion.

Findability

As I thought through the four components it seemed that discoverability might not fit the agenda well, but as I thought it through it became clear that discoverability is perhaps the most important aspect to consider. As an extreme example, something indexed in Google, or available via Wikimedia Commons doesn’t help if there is no network access. But more generally, the way in which we all search for information shapes the things we discover, and is the first necessary condition for engagement. From the challenges of getting Open Access books into library catalogues to the question of how patients can efficiently search for relevant research, via the systemic problems of how consumer search engines increasingly fail to provide clear provenance for information, the issue of inclusion and engagement starts, and far too often ends, with the challenges of discovery.

Conclusions

A few things become clear in considering this expansion of scope. The FAIR Data principles provide some clear proscriptions and tests for compliance. Issues of inclusion are much more open ended. When have we done enough? What audiences do we need to consider? In that sense it becomes much more a direction of travel than an easily definable goal to reach. But actually that was the initial goal, to prompt and provoke us to think more.

It also expands the question of the thing that is FAIR. For FAIR data we need only consider the resource, generally a dataset. With this expansion it is clear that it is both resources and the projects that generate them that we need to consider. A project could generate FAIR outputs without being FAIR itself. But again, this is a journey, not a destination. If we can hold ourselves to a higher standard then we will make progress towards that goal. With limited resources there will be difficult choices to make, but we can still use this idea as a prompt, to ask ourselves if we can do better.

If our goal is to do research that is “FAIR for All”, then we can test ourselves as we improve towards that goal by continuing to ask ourselves at each stage.

Is this FAIR enough?

 

 

Submission to the European Commission Expert Group on Altmetrics

As part of the broader Open Science agenda of the European Commission an expert group on “altmetrics” has been formed. This group has a remit to consider how indicators of research performance can be used effectively to enhance the strategic goals of the commission and the risks and opportunities that new forms of data pose to the research enterprise. This is my personal submission. 

Next Generation Altmetrics

Submission by Cameron Neylon, Professor of Research Communications, Curtin University

1. Introduction

The European Commission has an ambitious program for Open Science as part of three aspirations, Open Innovation, Open Science, and Open to the World. Key to defining the role of evaluation, and within that the role of metrics, in achieving all these aspirations is a clear understanding of the opportunities and limitations that our new, data-rich, environment creates. I therefore welcome the Commission’s call for evidence and formation of the expert group.

My expertise in this area is based on a long term interest in the intersection between research evaluation and policy implementation, specifically the role that new indicators can play in helping to drive (or hinder) cultural change. I was an author of the Altmetrics Manifesto[1] as well as the first major paper on Article Level Metrics[2]. I have more recently (in my previous role as Advocacy Director at PLOS) been closely engaged in technology and policy development, and wrote the PLOS submission to the HEFCE Metrics enquiry[3]. Since leaving PLOS I have been focusing on developing a research program looking at how technology and incentives combine to effect the culture of research communities. In this context recent work has included the preprint (currently under review) Excellence R Us[4] which has gained significant attention, and two reports for Jisc[5,6], that address related issues of evaluation and culture.

2. Next generation Metrics for open science

A. How do metrics, altmetrics & ‘responsible metrics’ fit within the broader EC vision & agenda for open science?

Delivering on the Commission’s agenda across the research policy platform requires a substantial culture change across a range of stakeholders. The cultures of research communities, and the practices that they support are diverse and often contradictory. It is important to separate the question of evaluation and how indicators support this, how evaluation contributes to the overall incentives that individuals and organisations experience, and what effect changes in incentives have on culture. Thoughtful evaluation, including the application of new and improved indicators can contribute to, but will not, on its own, drive change.

B. What are the key policy opportunities and tensions in this area? What leadership role can the EU play in wider international debates?

There are two opportunities that the current environment offers. First, the Commission can take a progressive leadership position on research evaluation. As the HEFCE Metrics enquiry and many others have concluded, much research evaluation has the tail wagging the dog: available indicators drive targets and therefore behaviour. It is necessary to reframe evaluation around what public research investment is for and how different stakeholder goals can be tensioned and prioritised. The Commission can take a leadership role here. The second opportunity is in using new indicators to articulate the values that underpin the Commission’s policy agenda. In this sense using indicators that provide proxies of the qualities that align with the Open Science agenda can provide a strong signal to research communities, researchers and RPOs that these aspects (collaboration, open access, data accessibility, evidence of re-use) are important to the Commission.

3. Altmetrics: The emerging state of the art

A. How can we best categorise the current landscape for metrics and altmetrics? How is that landscape changing? How robust are various leading altmetrics, and how does their robustness compare to more ‘traditional’ bibliometrics?

The landscape of available indicators is diverse and growing, both in the range of indicators available and the quality of data underpinning them. That said, this increase is from a low base. The current quality and completeness of data underlying indicators, both new and traditional, does not meet basic standards of transparency, completeness or equity. These indicators are neither robust, stable nor reliable. Auditing and critical analysis is largely impossible because data is generally proprietary. On top of this, the analysis of this data to generate indicators is in most cases highly naïve and undertheorized. This can be seen in a literature providing conflicting results on even basic questions of how different indicators correlate with each other. Bibliometrics while more established suffer from many of the same problems. There is greater methodological rigour within the bibliometrics research community but much of the use of this data are by users without this experience and expertise.

B. What new problems and pitfalls might arise from their usage?

The primary risk in the use of all such poorly applied indicators and metrics is that individuals and organizations refocus their efforts on performing against metrics instead of delivering on the qualities of research that the policy agenda envisions. Lack of disciplinary and output-type coverage is a serious issue for representation, particularly across the arts and humanities as noted in the HEFCE Metrics report.

C. What are some key conclusions and unanswered questions from the fast-growing literature in this area?

With some outstanding exceptions the literature on new indicators is methodologically weak and under-theorized. In particular, there is virtually no work looking at the evolution of indicator signals over time. There is a fundamental failure to understand these indicators as a signal of underlying processes. As a result there is a tendency to seek indicators that match particular qualities (e.g. “influence”) rather than understand how a particular process (e.g. effective communication to industry) leads to specific signals. Core to this failure is the lack of a framework for defining how differing indicators can contribute to answering a strategic evaluative question, and a tendency to create facile mathematical constructs of available data and defining them as a notionally desired quality.

4. Data infrastructure and standards

I refer the expert group to the conclusions of the HEFCE Metrics report, the PLOS submission to that enquiry[3] and to my report to Jisc[6] particularly on the issues of access to open citations data. Robust, trusted and responsible metrics require an open and transparent data infrastructure, with robust and critical data quality processes, alongside open processes subjected to full scholarly critical analysis.

The Commission has the capacity and resources to lead infrastructure development, including in data and technology as well as social infrastructures such as standards. My broad recommendation is that the Commission treat administrative and process data with the same expectations of openness, quality assurance, re-usablity and critical analysis as the research data that it funds. The principles of Open Access, transparency, and accountability all apply. As with research data, privacy and other issues arise and I commend the Commission’s position that data should be “as open as possible, as closed as necessary”.

5. Cultures of counting: metrics, ethics and research

A. How are new metrics changing research cultures (in both positive and negative ways)? What are the implications of different metrics and indicators for equality and diversity?

The question of diversity has been covered in the PLOS submission to the HEFCE Enquiry[3]. Indicators and robust analysis can both be used to test for issues of diversity but can also create issues for diversity. These issues are also covered in detail in our recent preprint4. Culture has been changing towards a more rigid, homogeneous and performative stance. This is dangerous and counter to the policy goals of the Commission. It will only be addressed by developing a strong culture of critical evaluation supported by indicators.

B. What new dynamics of gaming and strategic response are being incentivized?

Gaming is a complex issue. On one side there is “cheating”, on the other an adjustment of practice towards policy goals (e.g. wider engagement with users of research through social media). New indicators are arguably more robust to trivial gaming than traditional single data-source metrics. Nonetheless we need to develop institutional design approaches that promote “strategic responses” in the desired direction, not facile performance against quantitative targets.

6. Next generation metrics: The way forward

A. Can we identify emerging best practices in this area? What recommendations might we as a group make, and to which actors in EU research systems?

There are structural reasons why it is difficult to identify specific examples of best practice. I take the thoughtful use of data and derived indicators to support strategic decision making against clearly defined goals and values as the ideal. The transparency and audit requirements of large scale evaluations make this difficult. Smaller scale evaluation that is not subject to external pressures is most likely to follow this path. Amongst the large scale efforts that best exemplify efforts to reach these goals is the UK REF, where the question of what “excellence” is to be determined is addressed with some specificity and in Impact Narratives where data was used to support a narrative claim against defined evaluation criteria.

Overall we need to develop a strong culture of evaluation.

  • The Commission can support this directly through actions that provide public and open data sources for administrative and activity data and through adopting critical evaluative processes internally. The Commission can also act to lead and encourage adoption of similar practice across European Funding Organisations, including through work with Science Europe.
  • Institutions and funders can support development of stronger critical evaluation processes (including that of evaluating those processes themselves) by implementing developing best practice as it is identified and by supporting the development of expertise, including new research, within their communities.
  • Scholarly Societies can play a strong role in articulating the distinctive nature of their communities’ work and what classes of indicators may or may not be appropriate in assessment of that. They are also valuable potential drivers of the narratives that can support culture change
  • Researchers can player a greater role by being supported to consider evaluation as part of the design of research programs. Developing a critical capacity for determining how to assess a program (as opposed to developing the skills required to defend it all costs) would be valuable.
  • Publics can be engaged to define some of the aspects of what matters to them in the conduct and outcomes of research and how performance against those measures might be demonstrated and critically assessed to their satisfaction.

References

  1. Priem et al (2010), altmetrics: a manifesto, http://altmetrics.org
  2. Wu and Neylon (2009), Article Level Metrics and the Evolution of Scientific Impact, PLOS Biology, http://dx.doi.org/10.1371/journal.pbio.1000242
  3. PLOS (2013), PLOS Submission to the HEFCE RFI on Metrics in Research Assessment, http://dx.doi.org/10.6084/m9.figshare.1089555
  4. Moore et al (2016): Excellence R Us: University Research and the Fetishisation of Excellence. https://dx.doi.org/10.6084/m9.figshare.3413821.v1
  5. Neylon, Cameron (2016) Jisc Briefing Document on Data Citations, http://repository.jisc.ac.uk/id/eprint/6399
  6. Neylon, Cameron (2016) Open Citations and Responsible Metrics, http://repository.jisc.ac.uk/id/eprint/6377