Home » Blog

Pub-sub/syndication patterns and post publication peer review

20 June 2009 24 Comments

I think it is fair to say that even those of us most enamored of post-publication peer review would agree that its effectiveness remains to be demonstrated in a convincing fashion. Broadly speaking there are two reasons for this; the first is the problem of social norms for commenting. As in there aren’t any. I think it was Michael Nielsen who referred to the “Kabuki Dance of scientific discourse”. It is entirely allowed to stab another member of the research community in the back, or indeed the front, but there are specific ways and forums in which it is acceptable to do. No-one quite knows what the appropriate rules are for commenting on online fora, as best described most recently by Steve Koch.

My feeling is that this is a problem that will gradually go away as we evolve norms of behaviour in specific research communities. The current “rules” took decades to build up. It should not be surprising if it takes a few years or more to sort out an adapted set for online interactions. The bigger problem is the one that is usually surfaced as “I don’t have any time for this kind of thing”. This in turn can be translated as, “I don’t get any reward for this”. Whether that reward is a token for putting on your CV, actual cash, useful information coming back to you, or just the warm feeling that someone else found your comments useful, rewards are important for motivating people (and researchers).

One of the things that links these two together is a sense of loss of control over the comment. Commenting on journal web-sites is just that, commenting on the journal’s website. The comment author has “given up” their piece of value, which is often not even citeable, but also lost control over what happens to their piece of content. If you change your mind, even if the site allows you to delete it, you have no way of checking whether it is still in the system somewhere.

In a sense, when the Web 2.0 world was built it was got nearly precisely wrong for personal content. For me Jon Udell has written most clearly about this when he talks about the publish-subscribe pattern for successful frameworks. In essence I publish my content and you choose to subscribe to it. This works well for me, the blogger, at this site, but it is not so great for the commenter who has to leave their comment to my tender mercies on my site. It would be better if the commenter could publish their comment and I could syndicate it back to my blog. This creates all sorts of problems; it is challenging for you to aggregate your own comments together and you have to rely on the functionality of specific sites to help you follow responses to your comments. Jon wrote about this better than I can in his blog post.

So a big part of the problem could be solved if people streamed their own content. This isn’t going to happen quickly in the general sense of everyone having a web server of their own – it still remains too difficult for even moderately skilled people to be bothered doing this. Services will no doubt appear in the future but current broadcast services like twitter offer a partial solution (its “my” twitter account, I can at least pretend to myself that I can delete all of it). The idea of using something like the twitter service at microrevie.ws as suggested by Daniel Mietchen this week can go a long way towards solving the problem. This takes a structured tweet of the form @hreview {Object};{your review} followed optionally by a number of asterisks for a star rating. This doesn’t work brilliantly for papers because of problems with the length of references for the paper, even with shortened dois, the need for sometimes lengthy reviews and the shortness of tweets. Additionally the twitter account is not automatically associated with a unique research contributor ID. However the principle of the author of the review controlling their own content, while at the same time making links between themselves and that content in a linked open data kind of way is extremely powerful.

Imagine a world in which your email outbox or local document store is also webserver (via any one of an emerging set of tools like Wave, DropBox, or Opera Unite). You can choose who to share your review with and change that over the time. If you choose to make it public the journal, or the authors can give you some form of credit. It is interesting to think that author-side charges could perhaps be reduced for valuable reviews. This wouldn’t work in a naive way, with $10 per review, because people would churn out  large amounts of rubbish reviews, but if those reviews are out on the linked data web then their impact can be measured by their page rank and the authors rewarded accordingly.

Rewards and control linked together might provide a way of solving the problem – or at least of solving it faster than we are at the moment.


24 Comments »

  • Nash said:

    With respect to centralizing commenting :
    A simple approach is to replace your own commenting system with one of the few popular services out here. IntenseDebate , which I use, and Diqus come to mind.

    More often than not, these offer a much more advanced commenting and system, and since users can create profiles and have their comments centralized – allows for citation and a public record.

  • Nash said:

    With respect to centralizing commenting :
    A simple approach is to replace your own commenting system with one of the few popular services out here. IntenseDebate , which I use, and Diqus come to mind.

    More often than not, these offer a much more advanced commenting and system, and since users can create profiles and have their comments centralized – allows for citation and a public record.

  • Nash said:

    Edit above Line 3: Diqus = Disqus

  • Nash said:

    Edit above Line 3: Diqus = Disqus

  • Cameron Neylon said:

    Nash, that’s a good point. I have to admit that I tried a few of those and gave up because they just didn’t seem to work for me (as a commenter). The problem being that as the visitor you are still at the mercy of the site owner’s choice of service – if you are running Disqus I can’t choose to comment via Intense Debate – argument for data portability again I guess.

    Nonetheless this is something journals could run. Problems I would forsee are the same roundtripping issues that sites have with OpenID where people get confused being sent to a new site to register. Hopefully these kind of problems will go away soon though…

  • Cameron Neylon said:

    Nash, that’s a good point. I have to admit that I tried a few of those and gave up because they just didn’t seem to work for me (as a commenter). The problem being that as the visitor you are still at the mercy of the site owner’s choice of service – if you are running Disqus I can’t choose to comment via Intense Debate – argument for data portability again I guess.

    Nonetheless this is something journals could run. Problems I would forsee are the same roundtripping issues that sites have with OpenID where people get confused being sent to a new site to register. Hopefully these kind of problems will go away soon though…

  • Daniel Mietchen said:

    Just a brief extension of your note on bulks of rubbish reviews (or commentaries): to avoid them, it is sufficient to make the review citable, to allow it to be reviewed too, and to aggregate the reviews across authors, reviews and reviewers.

  • Daniel Mietchen said:

    Just a brief extension of your note on bulks of rubbish reviews (or commentaries): to avoid them, it is sufficient to make the review citable, to allow it to be reviewed too, and to aggregate the reviews across authors, reviews and reviewers.

  • Matt Leifer said:

    As a commenter, I have found Backtype to be much more useful than systems like Disqus and Intense debate. It aggregates your comments from blogs, friendfeed, etc. by using the URL that you enter into the comment form as an identifier. You can then syndicate the comments wherever you like.

    However, this is really the inverse of what Cameron is talking about, i.e. the publisher owns the comment and you syndicate it rather than the other way round.

  • Matt Leifer said:

    As a commenter, I have found Backtype to be much more useful than systems like Disqus and Intense debate. It aggregates your comments from blogs, friendfeed, etc. by using the URL that you enter into the comment form as an identifier. You can then syndicate the comments wherever you like.

    However, this is really the inverse of what Cameron is talking about, i.e. the publisher owns the comment and you syndicate it rather than the other way round.

  • David Crotty said:

    I don’t know, I get the feeling you’re coming at this from too “techie” a point of view, and that control isn’t a central issue here. Most people seem fine with putting their e-mail under someone else’s control (GMail), their photos under someone else’s control (Flickr), their daily diaries and communication with associates under someone else’s control (Facebook). Most people don’t want to intricately tinker with their operating system, and are fine with Windows rather than Linux. Most people don’t want to assemble or upgrade their hardware and instead buy the stock Dell or Mac computer. I’ve never heard anyone complain that the reason they’re not leaving comments on scholarly papers is a lack of control over those comments.

    The point of “I don’t have time/I get no reward” is that writing an intelligent, well-reasoned, well-written critique of a journal article takes time and effort. A quick response to a blog article can be dashed off, but something that carries your name and reputation that is directly in the field in which you work requires more care. And why bother when that time and effort could instead go into something with a direct professional reward instead of something that may actually cause you more grief than benefit?

    The question that needs to be answered is “why should I do this?”, not “how can I have better control over doing something that I’m not interested in doing?”

  • David Crotty said:

    I don’t know, I get the feeling you’re coming at this from too “techie” a point of view, and that control isn’t a central issue here. Most people seem fine with putting their e-mail under someone else’s control (GMail), their photos under someone else’s control (Flickr), their daily diaries and communication with associates under someone else’s control (Facebook). Most people don’t want to intricately tinker with their operating system, and are fine with Windows rather than Linux. Most people don’t want to assemble or upgrade their hardware and instead buy the stock Dell or Mac computer. I’ve never heard anyone complain that the reason they’re not leaving comments on scholarly papers is a lack of control over those comments.

    The point of “I don’t have time/I get no reward” is that writing an intelligent, well-reasoned, well-written critique of a journal article takes time and effort. A quick response to a blog article can be dashed off, but something that carries your name and reputation that is directly in the field in which you work requires more care. And why bother when that time and effort could instead go into something with a direct professional reward instead of something that may actually cause you more grief than benefit?

    The question that needs to be answered is “why should I do this?”, not “how can I have better control over doing something that I’m not interested in doing?”

  • Cameron Neylon said:

    David, I think you’ve actually made my point for me. The point about services like Flickr and Facebook, and even email, is that the account “belongs” to the user. There is the perception, even if not true, that I can close down my FlickR account, I can moderate comments, it is my own space. My point was less about some tool or some service and more that I wanted to make the suggestion that people would be more comfortable having comments in their own space. In sense this is a no-brainer because people do leave comments in “their own space” where that is the margins of the pdf.

    I’ve heard many people say are uncomfortable with leaving comments “on the paper” or “out in the open”, and a real sense that people are more comfortable in their own space. One interpretation of that is that people will just never be prepared to make criticism openly, but I find that unlikely. People do leave comments in private email, on group sites and I would suggest it’s a question of degree to move from there to something open.

    And the final point is that there is the potential for a more direct route towards reward if a comment or critique is clearly published by the commenter. The “why should I do this” is because without public critique we have no science, the question for me is therefore how to then create rewards that help to make it happen.

    So as usual I agree with everything you say but just see it from a different direction :-)

  • Cameron Neylon said:

    David, I think you’ve actually made my point for me. The point about services like Flickr and Facebook, and even email, is that the account “belongs” to the user. There is the perception, even if not true, that I can close down my FlickR account, I can moderate comments, it is my own space. My point was less about some tool or some service and more that I wanted to make the suggestion that people would be more comfortable having comments in their own space. In sense this is a no-brainer because people do leave comments in “their own space” where that is the margins of the pdf.

    I’ve heard many people say are uncomfortable with leaving comments “on the paper” or “out in the open”, and a real sense that people are more comfortable in their own space. One interpretation of that is that people will just never be prepared to make criticism openly, but I find that unlikely. People do leave comments in private email, on group sites and I would suggest it’s a question of degree to move from there to something open.

    And the final point is that there is the potential for a more direct route towards reward if a comment or critique is clearly published by the commenter. The “why should I do this” is because without public critique we have no science, the question for me is therefore how to then create rewards that help to make it happen.

    So as usual I agree with everything you say but just see it from a different direction :-)

  • David Crotty said:

    I still feel like you’re overthinking things because you are deeply enmeshed in the technology and issues of content ownership. Most people don’t think about these issues and readily turn control over to other services that they trust. If it was really such a big issue, no one would publish in journals, which are under someone else’s control. People are a lot more protective of their data and life’s work than they are of their comments. I’m not sure it’s really an issue here.

    I think people are grudgingly willing to leave comments in the open, but doing so is a commitment. If you’re going to go public, you need to be sure you have all your ducks in a row, thoroughly vet what you’re saying before you attack the work of a colleague. Think in terms of writing to a journal to demand someone else’s paper be retracted because the data is falsified or the conclusions are incorrect. You wouldn’t just dash that off, you’d want to be really sure you were correct before taking that step. And that requires a lot of time and effort. It’s not something you’re going to do for every paper you read, only in extreme cases where the effort is warranted.

    I’ve had grant applications slammed not because they were scientifically poor (the same application was funded by another agency) but because someone on the review committee had a beef with my PI. How many scientists are willing to publicly call out a colleague and start potential conflicts like this? We’d like to think that we’re all objective and thick-skinned enough to accept constructive criticism, but in my experience, this is far from the case with most scientists.

    One other thought, most papers probably don’t call for any useful commentary–how many articles in a given issue of a journal would you want to discuss in a journal club?

  • David Crotty said:

    I still feel like you’re overthinking things because you are deeply enmeshed in the technology and issues of content ownership. Most people don’t think about these issues and readily turn control over to other services that they trust. If it was really such a big issue, no one would publish in journals, which are under someone else’s control. People are a lot more protective of their data and life’s work than they are of their comments. I’m not sure it’s really an issue here.

    I think people are grudgingly willing to leave comments in the open, but doing so is a commitment. If you’re going to go public, you need to be sure you have all your ducks in a row, thoroughly vet what you’re saying before you attack the work of a colleague. Think in terms of writing to a journal to demand someone else’s paper be retracted because the data is falsified or the conclusions are incorrect. You wouldn’t just dash that off, you’d want to be really sure you were correct before taking that step. And that requires a lot of time and effort. It’s not something you’re going to do for every paper you read, only in extreme cases where the effort is warranted.

    I’ve had grant applications slammed not because they were scientifically poor (the same application was funded by another agency) but because someone on the review committee had a beef with my PI. How many scientists are willing to publicly call out a colleague and start potential conflicts like this? We’d like to think that we’re all objective and thick-skinned enough to accept constructive criticism, but in my experience, this is far from the case with most scientists.

    One other thought, most papers probably don’t call for any useful commentary–how many articles in a given issue of a journal would you want to discuss in a journal club?

  • Anna said:

    “One other thought, most papers probably don’t call for any useful commentary–how many articles in a given issue of a journal would you want to discuss in a journal club?”

    I would attribute this to the publish or die accountancy culture – a whole other debate.

    And for the record, (Cameron), I’d actually post more comments here if there were a delete button. This is primarily because I lack confidence in what I might say quite often, and I don’t want to look stupid in a public forum (too late you might say). This even applies for such small things as the potential to make a typo. And I might change my opinion and not want to be bound to a previous one in such a permanent way.

  • Anna said:

    “One other thought, most papers probably don’t call for any useful commentary–how many articles in a given issue of a journal would you want to discuss in a journal club?”

    I would attribute this to the publish or die accountancy culture – a whole other debate.

    And for the record, (Cameron), I’d actually post more comments here if there were a delete button. This is primarily because I lack confidence in what I might say quite often, and I don’t want to look stupid in a public forum (too late you might say). This even applies for such small things as the potential to make a typo. And I might change my opinion and not want to be bound to a previous one in such a permanent way.

  • David Crotty said:

    Anna, I’m not talking about poor papers–I just think there are a lot of valuable pieces of data that are published that don’t warrant further comment. There’s nothing wrong with publishing a useful piece of information that adds to a understanding a field but doesn’t inspire heated debate.

  • David Crotty said:

    Anna, I’m not talking about poor papers–I just think there are a lot of valuable pieces of data that are published that don’t warrant further comment. There’s nothing wrong with publishing a useful piece of information that adds to a understanding a field but doesn’t inspire heated debate.

  • Cameron Neylon said:

    David, it’s my job to be overly optimistic and enmeshed in the world of technology and content ownership isn’t it? And yours to pick out the holes in my arguments :-)

    More seriously I think what I have in mind is much more modest than a full A4 sheet of closely argued commentary. What I’m interested in is precisely the off the cuff commentary and small pieces of related information and connections that already exist, but largely in a closed form in the margins of PDFs or in emails to colleagues, or perhaps as notes in your own database.

    The reason for focussing on the control issue is precisely because people change their minds, realise they’ve got something wrong, and want to retract or modify. But the bigger question is how you create a market for these pieces of content that make people want to make them available. If they are on someone else’s website then you can’t even start with that.

    The issue is a bit less as to whether the small number of people who might be commenting at the moment are worried about it, and more about how you make it easier for all of those people who aren’t to add value and get value back out. My view is that the message from successful web-based systems is that pub-sub is a better model when you can get it to work wells so the question is how we might be able to make systems that make this more natural for the average scientist.

    If the technology worked (which it mostly doesn’t) then the idea of a feed of stuff being generated by each individual would seem a sensible kind of model – after that it can be syndicated by databases, journals, media, educators etc etc. But you lower the barrier to communication so as to make it easier for people to filter. In a sense this is a way of solving the identity problem, which is fundamentally about _re_-aggregating all of this stuff under one person’s name by putting it there in the first place. It just seems a more natural pattern to me.

    Behaviour is a problem – and I’m tempted to just say people should grow up, but human nature being what it is…nonetheless I am a great believer in the power of sunlight. If you do play in the open then you tend to behave more respectfully, or at least I find that I do. I probably bin 50% of the off-the-cuff comments that I start to write online. My personal belief is that communities will evolve online with quite distinct rules of behaviour – we just need to figure out what works. If science remains secretive and uncommunicative it will just dissappear – those that communicate both to each other and the public (and government) most successfully will survive and that will by its nature involve being online and being as visible as possible (ideally for quality not quantity though)

    And as for all those papers that aren’t worth commenting, well I would say they’re not worth publishing as papers. Just should be dropped into a database. If it’s not worth arguing about the details of what it means, or it doesn’t change things to the extent that it is worth discussing, then lets treat that as a small, but valuable, piece of a bigger puzzle. Or to put it another way, if it isn’t worth post-publication peer review, then why should we spend the trouble on peer reviewing it in the first place?

  • Cameron Neylon said:

    David, it’s my job to be overly optimistic and enmeshed in the world of technology and content ownership isn’t it? And yours to pick out the holes in my arguments :-)

    More seriously I think what I have in mind is much more modest than a full A4 sheet of closely argued commentary. What I’m interested in is precisely the off the cuff commentary and small pieces of related information and connections that already exist, but largely in a closed form in the margins of PDFs or in emails to colleagues, or perhaps as notes in your own database.

    The reason for focussing on the control issue is precisely because people change their minds, realise they’ve got something wrong, and want to retract or modify. But the bigger question is how you create a market for these pieces of content that make people want to make them available. If they are on someone else’s website then you can’t even start with that.

    The issue is a bit less as to whether the small number of people who might be commenting at the moment are worried about it, and more about how you make it easier for all of those people who aren’t to add value and get value back out. My view is that the message from successful web-based systems is that pub-sub is a better model when you can get it to work wells so the question is how we might be able to make systems that make this more natural for the average scientist.

    If the technology worked (which it mostly doesn’t) then the idea of a feed of stuff being generated by each individual would seem a sensible kind of model – after that it can be syndicated by databases, journals, media, educators etc etc. But you lower the barrier to communication so as to make it easier for people to filter. In a sense this is a way of solving the identity problem, which is fundamentally about _re_-aggregating all of this stuff under one person’s name by putting it there in the first place. It just seems a more natural pattern to me.

    Behaviour is a problem – and I’m tempted to just say people should grow up, but human nature being what it is…nonetheless I am a great believer in the power of sunlight. If you do play in the open then you tend to behave more respectfully, or at least I find that I do. I probably bin 50% of the off-the-cuff comments that I start to write online. My personal belief is that communities will evolve online with quite distinct rules of behaviour – we just need to figure out what works. If science remains secretive and uncommunicative it will just dissappear – those that communicate both to each other and the public (and government) most successfully will survive and that will by its nature involve being online and being as visible as possible (ideally for quality not quantity though)

    And as for all those papers that aren’t worth commenting, well I would say they’re not worth publishing as papers. Just should be dropped into a database. If it’s not worth arguing about the details of what it means, or it doesn’t change things to the extent that it is worth discussing, then lets treat that as a small, but valuable, piece of a bigger puzzle. Or to put it another way, if it isn’t worth post-publication peer review, then why should we spend the trouble on peer reviewing it in the first place?

  • Christina Pikas said:

    is co-comment still around? also weren’t trackbacks supposed to help? It’s depressing, but spammers ruin everything. it makes sense to use a general purpose tool, but it could be that some approval system like arXiv’s might be called for.

  • Christina Pikas said:

    is co-comment still around? also weren’t trackbacks supposed to help? It’s depressing, but spammers ruin everything. it makes sense to use a general purpose tool, but it could be that some approval system like arXiv’s might be called for.