Girl, Interrupting

January 27, 2011

Peer review – a bad example

Filed under: Peer review — sylviamclain @ 4:39 pm

Cameron Neylon has published one of 2 posts defending his opinion of peer-review –
standing by the quote ‘it makes more sense in fact to publish everything and filter after the fact’ though he admits this is somewhat of an over simplification.

What Neylon doesn’t suggest is what the alternative actually is yet – hopefully he will cover this in Post 2.

Meanwhile back at Occam’s Typewriter, Austin Elliot gives a nice balanced view of peer-review, in my opinion, with its pluses and minuses.

But I’d like to add something that is perhaps a bit orthogonal ( the picture below shows orthogonality; the blue lines are orthogonal to the red curves at the points they touch each other (or intersect) …

Orthogonal curves

The metaphoric example which Neylon uses in his blog post of the UK government giving money to a business isn’t, I don’t think, really equivalent or applicable to the majority of scientific publication.

He starts the argument as such:

“The UK government gives £3B to a company, no real strings attached, except the expectation of them reporting back. At the end of the year the company says “we’ve done a lot of work but we know you’re worried about us telling you more than you can cope with, and you won’t understand most of it so we’ve filtered it for you.””

First – companies and scientific research aren’t the same thing – this is a confusion between science and industry. Industry gives deliverables and science industry develops technology – basic science isn’t the same thing – you never know where discoveries come from in basic science AND basic science research doesn’t always lead to the conclusion you want it to. Its not really possible (if you are doing basic science) to say we will absolutely get this result, this is much easier to do in industry.

When you write a grant you DO have to put in milestones and goals and objectives – its not just a matter of here’s some money – do what you want. Also these are checked at many grant funding institutions, you have to write a progress report to get the rest of the money each year or 1/2 through the grant for example the ERC (European Research council).

With regard to publishing a paper the statement ‘we were worried about telling you more than you can cope with and you won’t understand most of it?’ I think is a bit of a fallacy

Do most scientists do this? I don’t think so – I never assume when I am writing a paper that the reviewers ‘won’t understand it’ and leave out data they ‘won’t understand’.

But I do ‘filter’ if that is what you want to call it as I don’t present every single piece of data I collect. Why? Not because most people can’t ‘understand it’ or can’t ‘cope’ but because alot of it is redundant information, and doesn’t enhance or detract from the central results.
Also you can add more detailed information into ‘supplementary materials’ which you can attach to many publications, and many people routinely do this!

So a real life example – I have a paper in a journal called Angewandte Chemie – it is a generalist journal for chemistry research so covers ALL of chemistry – its for a pretty wide audience. The article in question is a communication and there are rules for how much information you can put in an Angewandte Chemie Communication; roughly – maximum size is 3 to 4 figures – 4 pages – about 10,000 characters per article total – other stuff can be put into supplementary information for the more ‘specialist audience’ (which I have also done).

So I am not sure that I understand what the alternative would be? – I guess this is the ‘can’t cope’ argument however, to a generalist journal which covers lots of subjects – do you get rid of these journals, which have short papers? Or do you fill a whole journal with loads and loads of data figures for each paper – is this a realistic alternative?

Even if I didn’t have my doubts about this argument – how would the alternative to peer-review deal with leaving out data? If its out its out whether you publish it in a peer-reviewed or non-peer-reviewed journal.

The other thing he goes on to use in his example is that of a reporter asking the fictitious business dude the question:

Reporter: “So you’ll be making the whole record available as well as the stuff that you’ve said is most important presumably? I mean that’s easy to do?”

Business dude: “No we’d be worried about people getting the wrong idea so we’ve kept all of that hidden from them.”

Do most scientists keep things hidden when submitting a paper for peer review? I don’t think so – this is part of the supplementary information and the email addresses (usually the Senior Author’s email address is included in the publication for further queries) available both to the people doing the reviewers and the people that read the paper later. Having participated on both sides of the peer-review process often referees DO ask for additional information or even experiments.

Also again, even if this is a good argument for finding an alternative to peer-review – how is this alternative going to deal with ‘hidden data’ ?

He concludes in his example the following:

“Well we can’t show any evidence that the filtering is any good for deciding what is important or whether it’s accurate, but our employees are very attached to it. I can get some of them in, they’ll tell you lots of stories about how its worked for them…”

You also can’t show any evidence that ‘filtering’ is bad at deciding what is important, or can you? I may have it around my neck on this one – I don’t know or couldn’t find any studies that say this..
Neylon also accuses those who are proponents of peer-review of being un-scientific about believing in peer review – but I would counter that peer-review and its benefits are indeed hard to measure. For instance, I have a peer-review response which I think is helpful and think that it makes the quality of my research better and tunes me into other research going on out there- how do I measure this empirically? Survey? If we need data on this, I think we need to think carefully about how to get it – and if there is data, I am hoping that Neylon will cover it in post 2. Moreover, what is wrong with stories about how ‘it worked for them’ in fact there is a whole scientific discipline about people telling stories about how things worked for them and how they feel – its called psychology – and you can come up with measures to quantify this.

The problems I have with Neylon’s example is that to me it seems to imply that scientists routinely try to cover up things ‘they don’t think other people will understand’ or think others ‘can’t cope with’ and I don’t think this is the case.

I worry over my publications, I try to quadrupole check everything at the minimum. Do my publications (even after peer-review) contain mistakes that I myself didn’t catch? Of course – normally they are small, misspellings or my graph isn’t quite right, or I mis-referenced something by complete accident – but does this fundamentally change my scientific results? Nope not at all

What if it does? What if I missed something essential – I can issue an erratum, retract the paper, apologize, publish a new paper explaining the difference, there are options – and again how is an alternative to peer review going to deal with this?

I have made a somewhat big mistake in a paper before, which didn’t change my actual results or my interpretation of my results but definitely needed to be fixed – so what did I do? I published an erratum, which are available in most journals; likely because they do understand that people make mistakes.

Publishing scientific results, whether with peer-review or not, entails a certain amount of trust – if I look at a graph produced by a research group, I have to trust they did the actual experiment and are showing a graph of something they actually measured and mistakes, especially small ones, aren’t easy to catch – either before or after publication, either in peer reviewed literature or something I publish on the web myself. So in this respect I don’t understand how the argument for changing peer-review will help with most if any of the issues that Neylon brings up in his metaphorical example – but I am looking forward to his second post on the matter to clarify further.


January 22, 2011

Peer review here we go again

Filed under: Peer review,Trial by Twitter — sylviamclain @ 8:23 pm
Tags: , ,

Once again the peer-review vs. science online debate appears!

In an article by Peer review: Trial by Twitter – Apoorva Mandavilli talks about a lot of things but it mentions that science is getting ‘torn apart’ in the online media… which is ‘scary’ @rpg7twit (aka. Richard P Grant) has a nice response to this in the F1000 online magazine Naturally Selected from The Scientist…

OK maybe it is ‘scary’ but, sorry this, as a scientist is part of your damn job, presenting your work. It isn’t always pleasant going through peer-review, but it is a part of the process of presenting your science. And if your science makes it into a spot in one of the ‘big’ journals – like Nature well you want people to read it, or you wouldn’t have put it there in the first place. And like movie stars, you can’t have the fame without the Paparrazzi – that is you can’t only have praise with no critique, sorry life doesn’t work that way. Science is about, for better or worse, proving your hypothesis, it has to stand up to the test of time. It occurs to me perhaps the ‘test of time’ is just accelerated by the social media/internet process. This is a GOOD thing – maybe not for the scientist who got something wrong, but wake up – scientists get things wrong.

And it seems to me, as has been said, all this ‘taking apart of papers online’ just shows the public what scientists actually do – they take apart – or agree with – other’s work and this leads them to retest or look for new phenomena – that is a part of what we are supposed to be doing as scientists.

A few years ago in a field related to my own – somebody published a finding which said that liquid water (as opposed to being a tetrahedral network as most people have measured that it is) was not tetrahedral but actually linear. This, most people in the field thought, was crap. But instead of them all just sitting around and saying online it was crap – which they did – they also went back and re-measured and retested old data on liquid water. And as a much respected colleague of mine said at the time – this probably wasn’t such a bad thing for our field it shook it up and got people really explaining in better detail than before WHY the data showed what it WAS.

tetrahedral ice

But this isn’t actually why I am writing this blog post.

One thing that seems to come up in this debate often is something that is advocated by Cameron Neylon quite strongly which is:

‘it makes much more sense to publish everything and filter after the fact’

And this is where I’d like to archly raise an eyebrow myself.

I really cannot see, nor have I heard a very good argument as to why this makes more sense?

Its hard enough trying to filter through all of the work IN peer review journals if people just publish everything, imagine the volume. And imagine the volume of crap – as I pointed out in a blog post about a year ago – the internet now, and just the newspapers in the past already sometimes serve this function – look at the cold fusion story – where the science never even made it to peer review.

Also what if a ton of people think its crap but it really isn’t crap – which would be sort of opposite of the recent arsenic story.

This sort of implies and it has been implied that peer-review as it stands now is a bad thing
Why do so many people think everything is wrong with peer review? Perhaps it is these articles we see on Twitter – the ones where we see where the peer review process has missed something, but there are far more stories about where this doesn’t happen, I would suspect, you just never hear about them.

Does peer review go wrong? Yes
Do we hear about where peer review failed in the news? Yes – think of the recent example of arsenic.

Are there problems with the peer-review process? Of course.. look at Jan-Hendrick Shoon and recently Amil Potti

Do most peer-reviews articles NEVER make it to the news? Yes

But will publish first filter later make this better? I really doubt it.

Peer review is a team of people who are ‘experts’ in your field who look at your work and assess it – why is this altogether a bad thing?

In fact it can be, and largely is, a good thing. I have written papers – especially my first papers as corresponding author, where I was so focused on the forest I didn’t just miss the trees I actually, unknowingly, cut down the trees. After these manuscripts were peer reviewed – it was clear to me, from the reviewer’s comments – that they had no idea what I was talking about. Why? Because I wasn’t at all clear – so I had to fix it and as a result had a better paper. I have also had reviewers suggest work to me by other that I wasn’t aware of, often in agreement of what I was saying. That not only strengthened what I was saying in a particular paper but also my research in general.

If everything was published first – would be people take the time to even look at my piece of research to tear it apart? I am not an FRS or a Nobel prize winner, nor am I likely to be, but when I submit my paper to a suitable journal I have the same equality as anyone else in getting it read (this isn’t true for all journals, but specialists journals, certainly) – and I want it to be ‘torn apart’ to see if my science withstands the test because THIS is the point, whether it scares me or not.

Blog at

%d bloggers like this: