Tag: institution

Posted on

Tracking, Valuing and Evaluating the Diversity of HSS Research Outputs

This was originally a script for a video I recorded for a meeting of the Australian Deans and Directors of Creative Arts (DDCA) group. It is addressed to those who are in institutional positions who need to make judgements about the qualities of research outputs in Humanities and Social Sciences with a particular focus on outputs that are “non-traditional” (actually often very “traditional” in their field!) or otherwise “not easily measurable by a STEM audience”.

In my view there are two key roles for those specifically engaged in supporting those doing HSS research. The first is to advocate for financial systems that can support small and numerous research projects and the funders who want to support them. The second key role is to defend the diversity of research practices, outputs and impacts in HSS and to create a meaningful narrative around their qualities for institutional audiences.

I’m going to focus on the second here, but I wanted to link both to underline the point that what is of value in HSS research is precisely its diversity, and its difference from the homogeneity and scale of conventional STEM research and its outputs. There is not infrequently a greater diversity of research outputs from single HSS groups, centres and departments than there is from entire faculties of science, engineering and medicine.

That diversity is at the core of the value that HSS research and scholarship brings to the table. 

Every social epistemology of knowledge from Latour to Longino via Merton, Ravetz, Kuhn and Fleck have at their heart the idea that the core is the negotiation of knowledge across boundaries, or between groups. This places diversity, of all kinds, at the heart of this process, and linked to it the institutional systems that keep it alive. Arguably it is the purpose of a university to support and bring into contact diverse approaches, experiences and ways of knowing.

The issue of course is that managing that diversity is a challenge. At the scale of a university, systems are a necessity. Consistency is needed and economies of scale are real. Particularly in HSS but across other disciplines as well, it is impossible for any leader to be across the detail of how good every specific body of work is. If you believe in the argument for diversity then this as it should be – anything else would be a sign of insufficient diversity. That doesn’t mean that leaders can’t be effective at telling the stories about why diverse research matters. Nor does it mean that they can’t make judgement calls on the distribution of resources. But it means those decisions are subjective and contextual. And that the evidence that supports them will be specific to each case.

But that doesn’t solve the problem either. So what are the pragmatic and principled approaches available to addressing this complexity? In my view the key lies in separating out the challenge into two parts, one relatively straightforward and immediately useful, the other more complex but I would argue ultimately tractable.

An important, but oddly overlooked distinction in research evaluation and particularly technology discussions is the distinction between tracking outputs and evaluating them. There is often a tendency to lump these together and to imagine for example, that if scholarly books were better tracked then that would necessarily mean there would be improved citation data for them. The homogeneity of STEM research outputs means that issues with data coverage and completeness are often conflated to the question “is it indexed” using the presence of a specific journal in a specific database both as a reliable marker of quality itself but also of confidence in the quality and relevance of generally accepted performance data.

So, although this is an oversimplification, in STEM the tracking of outputs is standardised and homogenous. You look it up in an index like Web of Science.

In HSS, and particularly for creative works, the situation is far more complex. Institutions generally have very poor data on the scope and volume of creative outputs, or other forms of (so-called) “non-traditional” outputs. Work done by Niamh Quigley, within the COKI team (Quigley, 2022) showed generally used recording systems (systems like Symplectic, Pure and the like) tended to be poor at capturing aspects of research outputs important to creative researchers. Further, there was a tendency for the information that was manually inputted by researchers (that they care enough to type in!) to be stripped away as it made its way through institutional systems to the data warehouses used for high level analysis.

From a systems perspective what is needed is systems that make it easy for creative practice researchers to register outputs, to flexibly record the evidence that they consider important about the qualities of these works, and to also make it easy for institutions in turn to gather that information up. Recent technical developments provide improved paths to making this happen.

The Open Research Contributor and ID (ORCID) system provides a mechanism for researchers to authoritatively identify themselves, and to record outputs they have contributed to. In 2025 ORCID have increased the number of work types (Petro, 2025) to include a range relevant in humanities (including images, video, designs, sound and other creative works) as well as improving the mapping of some other relevant work types including reports. A researcher can choose what works they make visible, and thereby designate as “research”. This is an important distinction for some creative workers. Institutions can then harvest those outputs without requiring further work from researchers. 

ORCID doesn’t solve all the problems though. Badly implemented as an institutional system it is yet another place creative researchers have to battle through manually adding their work while colleagues in STEM subjects have it all done for them by journal publishers. Critical to making this work is a commitment from you and your teams that ORCID be the only place researchers have to make sure their work is available and pulling that information from ORCID is an institutional responsibility.

But that still leaves the manual input issue. There isn’t – currently – a complete solution for that (although one – or several – could be built) but there is at least a solution that offers some additional value for creative researchers. Zenodo is a large scale repository, managed by CERN (yes the particle accelerator in Switzerland CERN) that offers a place for anyone to deposit research outputs. It provides a safe, publicly accessible platform from which researchers can showcase their work. It even allows for dark deposit for those creative researchers who need to restrict access to the core of their work for artistic, ethical or commercial reasons.

Most importantly Zenodo provides DataCite DOIs and this means that – while you do have to manually put information into Zenodo – that information can automatically flow through to ORCID. One place to input the information and then it can flow to the other places it needs to be.

This flow of information should be the case for any repository that integrates with DataCite (or Crossref) to provide DOIs. Zenodo is large scale and flexible, but there can be an argument for specialist repositories. I am generally sceptical of the value of creating new local repositories, whether at institutions or nationally, but the key questions to ask of any repository are “does it integrate with DOI providers” and “does it provide the recording functionality creative research practitioners need”? That flexibility is key and Zenodo has some strengths here that can also help us to address, if not yet fully solve, the second problem. 

Creative practice researchers know the reasons that their work is important, and can help provide evidence of that. But that evidence is highly diverse. 

I know the pressures that institutional leaders face to articulate a comparative quantitative value of research outputs to the rest of the university. And I would argue that we nonetheless need to collectively hold the line that this makes no sense. If you value a diversity of research and researchers, you can’t protect that diversity by homogenising the way that value is measured. 

But that doesn’t mean we can’t articulate value. It doesn’t mean we can’t compare how disparate research projects and outputs achieve specific goals. It just means, in the best tradition of HSS scholarship, that the evidence that we use is contextual. That it is diverse.

And here is how repositories like Zenodo can help. Where they support the creation of packages of work, researchers can bundle up the work itself with evidence of its value and impact. This will start unstructured, but over time, and in collaboration with research communities it will be possible to develop some standards. For specific cases. Where appropriate. This might mean giving some background on the gallery that invited an exhibition, the footfall or an audience survey. It might be details of the film festival that selected a documentary for screening, or audience numbers when it was syndicated or broadcast. It might be structured in a particular way to make it easier to process (or even generate). But the key is to build on systems that always allow for the flexibility of a narrative statement that allows for the flexibility that research statements already provide. 

This workflow places researchers in control, gives them flexibility they need to provide access and to evidence the value of their work. It provides a single point of entry from which communities, institutions, and others can draw in the data to help track those outputs and collate the evidence of their value. It is not controlled by corporate interests but by academic communities. It is not subject to the funding whims of one national government but an international set of consortia.

Perhaps best of all, this is an area where the humanities and social sciences can lead. It solves a problem by putting researchers in control while supporting institutional needs. It helps and simplifies monitoring while offering a path to gathering more sophisticated evidence to tell better stories. These are not problems that are restricted to HSS, but you could argue that much of STEM evaluation has lost its way, focusing on numbers instead of substance, journal lists instead of journal content, impact factors instead of actual impact. This recontextualisation, and re-situation is where humanities and social sciences scholars, and leadership can show a way to better forms of evaluation.

References

Posted on

.everyone or .science? Or both? Reflections on Martha Lane Fox’s Dimbleby Lecture

English: Martha Lane Fox
Martha Lane Fox (Photo: The Cabinet Office License: OGL v1.0)

On March 30 the BBC broadcast a 40 minute talk from Martha Lane Fox. The Richard Dimbleby Lecture is an odd beast, a peculiarly British, indeed a peculiarly BBC-ish institution. It is very much an establishment platform, celebrating a legendary broadcaster and ring marshaled by his sons, a family that as our speaker dryly noted are “an entrenched monopoly” in British broadcasting.

 

Indeed one might argue Baroness Lane Fox, adviser to two prime ministers, member of the House of Lords, is a part of that establishment. At the same time the lecture is a platform for provocation, for demanding thinking. And that platform was used very effectively to deliver a brilliant example of another very British thing, the politely impassioned call for radical (yet moderate) action.

The speech calls for the creation of a new public institution. Dubbed “Dot Everyone” such an institution would educate, engage and inform all citizens on the internet. It would act as a resource, it would show what might be possible, it would enhance diversity and it would explore and implement a more values based approach to how we operate on the web. I have quibbles, things that got skipped over or might merit more examination, but really these are more the product of the space available than the vision itself.

At the centre of that vision is a call for a new civics supported by new institutions. This chimes with me as it addresses many of the same issues that have motivated my recent thinking in the research space. The Principles for Open Infrastructures I wrote with Geoff Bilder and Jennifer Lin, could as easily have been called Principles for Institutions – we were motivated to work on them because we believe in a need for new institutions. For many years I have started talks on research assessment by posing the question “what are your values” – a question implicit in the speech as it probes the ethics of how the internet is built in practice.

I was excited by this speech. And inspired.

And yet.

One element did not sit easily with me. I emphasized the British dimension at the top of this piece. Martha Lane Fox’s pitch was to “make Britain brilliant at the internet” and was focused on the advantages for this country. By contrast the first of the Principles for Open Infrastructures is that these new institutions must transcend geography and have international reach. Is this a contradiction? Are we pushing in different directions? More particularly is there a tension between an institution “for everyone” and one having a national focus?

The speech answers this in part and I think the section is worth quoting in full:

We should be ambitious about this. We could be world leading in our thinking.

In this 800th year anniversary of Magna Carta, the document widely upheld as one of the first examples of the rule of law, why don’t we establish frameworks to help navigate the online world?

Frameworks that would become as respected and global as that rule of law, as widely adopted as the Westminster model of parliamentary democracy.

Clearly this new institution, “our new institution” as it is referred to throughout, has international ambitions. But I don’t imagine I am the only person to find something almost neo-colonial in these words. Britain has sought to export its values to the world many times, and been remarkably successful. But in the past this has also been paternalistic. The very best possible assessment of what was in many cases well intentioned imposition of British values is equivocal. Lane Fox sets up the “good” values of Britain against the lack of values that inhere in the big commercial players building the web. What is it today that make “our” values those that should inspire or lead any more than the, now questionable, values of the past?

To be clear I am absolutely not suggesting that these are issues that have escaped the speaker’s notice. Martha Lane Fox is an outstanding and effective campaigner for diversity and inclusion and the section of her talk that I have taken out of context above comes after a substantial section on the value of inclusion, focused largely on gender but with a recognition that the same issues limit the inclusion and contribution of many people on the basis of many types of difference. In truth, her views on the how and the why of what we need to change, on what those values are, are highly aligned with mine.

But that’s kind of the point.

If we are to have a new civics, enabled by the communications infrastructure that the web provides, then diversity will lie at the heart of this. Whether you take the utilitarian (not to say neo-liberal) view that inclusion and diversity drives the creation of greater value, or see it as simply a matter of justice, diversity and inclusion and acceptance of difference are central.

But at the same time the agile and flat governance models that Lane Fox advocates, to be fair in passing, for our new institution arise out the concept that “rough consensus and running code” are the way to get things done. But whose consensus matters? And how does the structural imbalance of the digital divide affect whose code gets to run first? This seems to me the central question to be resolved by this new civics. How do we use the power of web to connect communities of interest, and to provide infrastructures that allow them to act, to have agency, while at the same time ensuring inclusion.

At its best the web is an infrastructure for communities, a platform that allows people to come together. Yet communities define themselves by what they have in common, and by definition exclude those who do not share those characteristics. My implicit claim above that our institutional principles are somehow more inclusive or more general than Lane Fox’s is obviously bogus. Our focus is on the research community, and therefore just as exclusive as a focus on a single nation. There are no easy answers here.

The best answer I can give is that we need multiple competing centres. “Dot Everyone” is a call for a national institution, a national resurgence even. Alone it might be successful, but even better is for it to have competition. Martha Lane Fox’s call is ambitious, but I think it’s not enough. We need many of these institutions, all expressing their values, seeking common ground to build a conversation between communities, domains, geographies and nations.

The tension between facilitating community and diversity can be a productive one if two conditions are satisfied. First that all can find communities where they belong, and secondly that the conversation between communities is just and fair. This is a huge challenge, it will require nothing less than a new global infrastructure for an inclusive politics.

It is also probably a pipe dream, another well meaning but ultimately incomplete effort to improve the world. But if the lesson we learn from colonialism is that we should never try, then we should give up now. Better is to do our best, while constantly questioning our assumptions and testing them against other’s perspectives.

As it happens, we have some new systems that are pretty good for doing that. We just need to figure out how best to use them. And that, at core, was Martha Lane Fox’s point.