Girl, Interrupting

January 27, 2011

Peer review – a bad example

Filed under: Peer review — sylviamclain @ 4:39 pm

Cameron Neylon has published one of 2 posts defending his opinion of peer-review –
standing by the quote ‘it makes more sense in fact to publish everything and filter after the fact’ though he admits this is somewhat of an over simplification.

What Neylon doesn’t suggest is what the alternative actually is yet – hopefully he will cover this in Post 2.

Meanwhile back at Occam’s Typewriter, Austin Elliot gives a nice balanced view of peer-review, in my opinion, with its pluses and minuses.

But I’d like to add something that is perhaps a bit orthogonal ( the picture below shows orthogonality; the blue lines are orthogonal to the red curves at the points they touch each other (or intersect) …

Orthogonal curves

The metaphoric example which Neylon uses in his blog post of the UK government giving money to a business isn’t, I don’t think, really equivalent or applicable to the majority of scientific publication.

He starts the argument as such:

“The UK government gives £3B to a company, no real strings attached, except the expectation of them reporting back. At the end of the year the company says “we’ve done a lot of work but we know you’re worried about us telling you more than you can cope with, and you won’t understand most of it so we’ve filtered it for you.””

First – companies and scientific research aren’t the same thing – this is a confusion between science and industry. Industry gives deliverables and science industry develops technology – basic science isn’t the same thing – you never know where discoveries come from in basic science AND basic science research doesn’t always lead to the conclusion you want it to. Its not really possible (if you are doing basic science) to say we will absolutely get this result, this is much easier to do in industry.

When you write a grant you DO have to put in milestones and goals and objectives – its not just a matter of here’s some money – do what you want. Also these are checked at many grant funding institutions, you have to write a progress report to get the rest of the money each year or 1/2 through the grant for example the ERC (European Research council).

With regard to publishing a paper the statement ‘we were worried about telling you more than you can cope with and you won’t understand most of it?’ I think is a bit of a fallacy

Do most scientists do this? I don’t think so – I never assume when I am writing a paper that the reviewers ‘won’t understand it’ and leave out data they ‘won’t understand’.

But I do ‘filter’ if that is what you want to call it as I don’t present every single piece of data I collect. Why? Not because most people can’t ‘understand it’ or can’t ‘cope’ but because alot of it is redundant information, and doesn’t enhance or detract from the central results.
Also you can add more detailed information into ‘supplementary materials’ which you can attach to many publications, and many people routinely do this!

So a real life example – I have a paper in a journal called Angewandte Chemie – it is a generalist journal for chemistry research so covers ALL of chemistry – its for a pretty wide audience. The article in question is a communication and there are rules for how much information you can put in an Angewandte Chemie Communication; roughly – maximum size is 3 to 4 figures – 4 pages – about 10,000 characters per article total – other stuff can be put into supplementary information for the more ‘specialist audience’ (which I have also done).

So I am not sure that I understand what the alternative would be? – I guess this is the ‘can’t cope’ argument however, to a generalist journal which covers lots of subjects – do you get rid of these journals, which have short papers? Or do you fill a whole journal with loads and loads of data figures for each paper – is this a realistic alternative?

Even if I didn’t have my doubts about this argument – how would the alternative to peer-review deal with leaving out data? If its out its out whether you publish it in a peer-reviewed or non-peer-reviewed journal.

The other thing he goes on to use in his example is that of a reporter asking the fictitious business dude the question:

Reporter: “So you’ll be making the whole record available as well as the stuff that you’ve said is most important presumably? I mean that’s easy to do?”

Business dude: “No we’d be worried about people getting the wrong idea so we’ve kept all of that hidden from them.”

Do most scientists keep things hidden when submitting a paper for peer review? I don’t think so – this is part of the supplementary information and the email addresses (usually the Senior Author’s email address is included in the publication for further queries) available both to the people doing the reviewers and the people that read the paper later. Having participated on both sides of the peer-review process often referees DO ask for additional information or even experiments.

Also again, even if this is a good argument for finding an alternative to peer-review – how is this alternative going to deal with ‘hidden data’ ?

He concludes in his example the following:

“Well we can’t show any evidence that the filtering is any good for deciding what is important or whether it’s accurate, but our employees are very attached to it. I can get some of them in, they’ll tell you lots of stories about how its worked for them…”

You also can’t show any evidence that ‘filtering’ is bad at deciding what is important, or can you? I may have it around my neck on this one – I don’t know or couldn’t find any studies that say this..
Neylon also accuses those who are proponents of peer-review of being un-scientific about believing in peer review – but I would counter that peer-review and its benefits are indeed hard to measure. For instance, I have a peer-review response which I think is helpful and think that it makes the quality of my research better and tunes me into other research going on out there- how do I measure this empirically? Survey? If we need data on this, I think we need to think carefully about how to get it – and if there is data, I am hoping that Neylon will cover it in post 2. Moreover, what is wrong with stories about how ‘it worked for them’ in fact there is a whole scientific discipline about people telling stories about how things worked for them and how they feel – its called psychology – and you can come up with measures to quantify this.

The problems I have with Neylon’s example is that to me it seems to imply that scientists routinely try to cover up things ‘they don’t think other people will understand’ or think others ‘can’t cope with’ and I don’t think this is the case.

I worry over my publications, I try to quadrupole check everything at the minimum. Do my publications (even after peer-review) contain mistakes that I myself didn’t catch? Of course – normally they are small, misspellings or my graph isn’t quite right, or I mis-referenced something by complete accident – but does this fundamentally change my scientific results? Nope not at all

What if it does? What if I missed something essential – I can issue an erratum, retract the paper, apologize, publish a new paper explaining the difference, there are options – and again how is an alternative to peer review going to deal with this?

I have made a somewhat big mistake in a paper before, which didn’t change my actual results or my interpretation of my results but definitely needed to be fixed – so what did I do? I published an erratum, which are available in most journals; likely because they do understand that people make mistakes.

Publishing scientific results, whether with peer-review or not, entails a certain amount of trust – if I look at a graph produced by a research group, I have to trust they did the actual experiment and are showing a graph of something they actually measured and mistakes, especially small ones, aren’t easy to catch – either before or after publication, either in peer reviewed literature or something I publish on the web myself. So in this respect I don’t understand how the argument for changing peer-review will help with most if any of the issues that Neylon brings up in his metaphorical example – but I am looking forward to his second post on the matter to clarify further.



  1. I agree that lack of evidence is not necessarily evidence of lack. The problem is that where the potential positive effects of peer review have been studied they really do show no benefit. I think its incumbent on us to show that what we do has a benefit and particularly where it costs public money. I’d love to see some better evidence on this either way to be honest. What I can show is the evidence that peer review costs several hundred million pounds a year in the UK (incidentally a similar amount to the shortfall over the CSR period) and that there is little or no evidence that it has any benefit. That seems like something we ought to be worried about to me.

    As far as the dialogue goes I don’t think I meant it the way you seem to have taken. My point is not that people are hiding anything particularly, but that the public perception that they might be hiding something is something we need to be aware of. From the perspective of the public, and potentially the media, there is no difference between the research community and a company. They expect both to be equally “honest” and “open”, recognising that those are loaded and subjective terms. My point is that if someone outside the research community presented you with this kind of argument you’d be shocked and appalled. You’ve fallen back on a “but we’re different” position, which is fine, but then you have to justify what the difference is, not in terms of communications within the community but in terms of communication beyond the community.

    I’ll try to cover some alternatives in the second post but as a starter physics seems to work just fine with the ArXiv. I think its interesting to ask why people feel so strongly that a similar approach can’t or won’t work beyond physics.

    Comment by Cameron Neylon — January 27, 2011 @ 7:06 pm | Reply

    • I honestly don’t think I have fallen back on the ‘but we’re different’ catagory at all
      not publishing all of your data isn’t the same as hiding data, they are clearly different –

      but there is a HUGEdifference between a research community and a company – and maybe this is the problem there IS a fundamental difference so maybe that is what we have to get better? the explanationI mean

      I also think there is a confusion here between scientific publications and press-releases – these are two completely different beasts. BOTH I would argue are very important – but also I think we forget that most science publications never even come close to the media, not that they shouldn’t be clear, but there is a difference. Usually, in my opinion press releases go wrong when people over interepret their results, which is why science goes wrong as well. but for different reasons – think about press releases about banker’s bonuses for instance – we just hear ‘bankers get bonuses of X’ but don’t you think the real situation is much more complex? news often truncates MOST stories to somthing much simplier than it is

      Yep but the thing about the ArXiV journal thing is its not a good example – a) many physics disciplines don’t use it and b) they still publish those papers (after ArXiV) in peer review journals – also as someone pointed out on a comment to your blog – there is more reserach that goes uncommented on (and unreviewed) – there is an excellent blogpost about this –

      I am looking forward to your next post, maybe that will clear some of this up…. and I would like to see the links about money – How does it cost so much money? Do you have a link, how do they come up with this figure – who does it cost money? also is there really a free alternative?

      By the way did you see that review by Parliament? You should go to that you have lots to say!

      Comment by sylviamclain — January 27, 2011 @ 7:20 pm | Reply

      • Ok. Just to be clear what I’m talking about here is the notional “papers” that are being blocked from publication by a peer review process. So this isn’t redundant data but something that someone thought was worth publishing. But at some level I’m thinking about data as well. In a world where it is essentially free to release everything then why not do it? I have no problem with having a process that marks up stuff of higher quality but why not release everything else as well, just on the off chance that someone might find it useful?

        In terms of perceptions it doesn’t actually matter whether a research community is different to a company (or a research charity, or the charitable foundation of a research company etc etc). What matter is whether the public would accept these arguments from the research community. My assertion is that the world has changed and that this is no longer acceptable. Some scientists still seem to be genuinely bewildered that people (and I don’t mean climate change sceptics) were actually disturbed by the apparent secrecy at UEA. The assumption that information, particularly publicly funded information, will be public is strong and I think getting stronger. We ignore that at our peril.

        I use the ArXiv example carefully. My point here is that the sky hasn’t fallen in because work is being published before peer review. I’d also note that, in those fields that use it, people tend not to actually read the journals. They get the papers, and do their searching at ArXiv. The point is that such systems can work, at least in some disciplines. The interesting question is why they seem to be unable to work in e.g. the biological sciences.

        Costing of peer review: its quite controversial but the numbers that have been pulled together by the Research Information Network seem amongst the most credible to me. There are a couple of sources. First a report on the overall costs of scholarly comms which is worth a read in general: Activitie,s costs and funding flows. The link I put in the post is to a PDF report on peer review Peer Review, a researcher’s guide. I got the numbers from p12 in the box, second para. The cost is in time, paid in the UK largely through HEFCE (hence the comment about it being charged to a different department in my dialogue).

        Comment by Cameron Neylon — January 27, 2011 @ 7:38 pm

      • ok from your first comment we are talking about different things – but I agree – this is the point of supplemental information – secondly, I think many researchers are indeed happy to give their data out – this is why we have an email address in the paper – people can ask for this. In my personal case I would love to set up a data base of information from my experiments – but I haven’t largely due to time not because I have some kind of problem with this

        I think it does matter that reserach communities, like NGOs are different than businesses – they do different things. But keeping information secret is a different arguement – not that I have anything readily google-able but I am happy to share my research with those who ask.

        Maybe you should set up a BioArXiv yourself – also I think if you are working in a highly competitve area, perhaps you are not putting stuff out there before peer-review for a reason.

        Thank you for the links – but as a side note does HEFCE care about a researchers time? Most universities don’t you just have to fit things like peer-review in in between publishing, teaching and writing for grants so most academics work far more then their allotted 37.5 hours a week by HEFCE so I would say HEFCE isn’t loosing money in ‘real terms’ also this same amount of time would be spent on reading stuff extra to peer-review – to keep up in your field – so where is the real time lost?

        Comment by sylviamclain — January 27, 2011 @ 9:18 pm

      • Re the arXiV, essentially all physics *in some disciplines* is published there. “All physics” is shorthand, well understood by arXiV users but maybe unclear to others in a discussion such as this.

        Comment by Nick Barnes — January 27, 2011 @ 9:50 pm

      • good point – I work in physics and I have I think 1 paper there – don’t routinely do it – thanks for clearing that up

        Comment by sylviamclain — January 27, 2011 @ 9:51 pm

  2. Do you publish every value measured by every instrument, and every line of code you ever write? If so, you are nearly unique in the science community. If not, you are ‘hiding’ things. And page counts are a red herring: Neylon is not suggesting that all this data should go in the core publication (which in fact is one of the least useful things you can do with it, see also “Beyond the PDF”, or Peter Murray-Rust’s excellent examples of the ways valuable data can be destroyed in the process).

    Comment by Nick Barnes — January 27, 2011 @ 8:15 pm | Reply

    • that is what I am saying – we don’t ever publish absolutely everything! but it is availble should someone ask..

      Comment by sylviamclain — January 27, 2011 @ 9:11 pm | Reply

      • Many scientists won’t provide “everything” even if people do ask. What Neylon is saying, I think, is that everything should be published, without asking. Avoid confusion about the word “published” here: what I mean is “available at no fee to any interested person”. Some people (probably including Neylon) would add “under an open license”.

        Comment by Nick Barnes — January 27, 2011 @ 10:05 pm

  3. Ok, threading doesn’t work quite so deep, so a couple of responses. The data is a bit of a side issue, although an important one. There are some studies on this and they show pretty universally that getting backround information and data out of scientists that relates to published papers is like pulling teeth. Compliance rates, even for journals that have tough policies is around 10-20%. The other point is that emailing the author doesn’t scale. If I want to collect all the data in your field, much easier to just grab it from a repo than have to email every author. Also it is much easier on the authors to actually prepare data in a form that is easily publishable as it is collected than to do it later. Imagine if I really did send you that email today, how much of a pain would it be to collect the relevant data into a useable form, vs just pointing me at a URL. Obviously that requires upfront work tho

    BioArXiv has been tried in a couple of forms, most recently and probably most successfully with Nature Precedings but this is hardly a raging success. There do seem to be real issues in adopting this approach in the biosciences. Primarily this is tied up with a concern that a preprint is not a “real paper” and therefore you are increasing your chance of being scooped by making information available. Physicists (at least those disciplines that use ArXiv) find this argument bizarre. Their view is how can you afford not to put out the preprint because someone might scoop you! This is because to the physicists the ArXiv version is a “real version”. This subtle difference plays out in a lot of ways and is frankly totally bizarre but it has to do with the degree of competition and stability of incumbent systems.

    Finally for the moment: the issue of HEFCE and elastic time. I think it is very dangerous to assume that time can just expand. If that were true why can’t you just do more research and write more papers? The reality is that there are limits. They may not be 40 hours a week, but there are limits to what any single person can achieve within their work space and time and these things do get tensioned against each other. An hour on peer review may not be a full hour lost from the lab or data analysis but it is some time lost. So what is the value we assign to peer review vs lab work. We need to do both (in some form) to get ahead so what should the balance be? And how much effort should we put into making each of these more efficient?

    Comment by Cameron Neylon — January 28, 2011 @ 9:16 am | Reply

  4. […] review has a central place is modern-day science. There’s a more muttering about peer review in other places, including in England’s House of Commons. (Peer review needs to be peer reviewed?) I’d […]

    Pingback by A structured procrastination | Code for Life — February 6, 2011 @ 4:19 am | Reply

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at

%d bloggers like this: