Girl, Interrupting

January 22, 2011

Peer review here we go again

Filed under: Peer review,Trial by Twitter — sylviamclain @ 8:23 pm
Tags: , ,

Once again the peer-review vs. science online debate appears!

In an article by Peer review: Trial by Twitter – Apoorva Mandavilli talks about a lot of things but it mentions that science is getting ‘torn apart’ in the online media… which is ‘scary’ @rpg7twit (aka. Richard P Grant) has a nice response to this in the F1000 online magazine Naturally Selected from The Scientist…

OK maybe it is ‘scary’ but, sorry this, as a scientist is part of your damn job, presenting your work. It isn’t always pleasant going through peer-review, but it is a part of the process of presenting your science. And if your science makes it into a spot in one of the ‘big’ journals – like Nature well you want people to read it, or you wouldn’t have put it there in the first place. And like movie stars, you can’t have the fame without the Paparrazzi – that is you can’t only have praise with no critique, sorry life doesn’t work that way. Science is about, for better or worse, proving your hypothesis, it has to stand up to the test of time. It occurs to me perhaps the ‘test of time’ is just accelerated by the social media/internet process. This is a GOOD thing – maybe not for the scientist who got something wrong, but wake up – scientists get things wrong.

And it seems to me, as has been said, all this ‘taking apart of papers online’ just shows the public what scientists actually do – they take apart – or agree with – other’s work and this leads them to retest or look for new phenomena – that is a part of what we are supposed to be doing as scientists.

A few years ago in a field related to my own – somebody published a finding which said that liquid water (as opposed to being a tetrahedral network as most people have measured that it is) was not tetrahedral but actually linear. This, most people in the field thought, was crap. But instead of them all just sitting around and saying online it was crap – which they did – they also went back and re-measured and retested old data on liquid water. And as a much respected colleague of mine said at the time – this probably wasn’t such a bad thing for our field it shook it up and got people really explaining in better detail than before WHY the data showed what it WAS.

tetrahedral ice

But this isn’t actually why I am writing this blog post.

One thing that seems to come up in this debate often is something that is advocated by Cameron Neylon quite strongly which is:

‘it makes much more sense to publish everything and filter after the fact’

And this is where I’d like to archly raise an eyebrow myself.

I really cannot see, nor have I heard a very good argument as to why this makes more sense?

Its hard enough trying to filter through all of the work IN peer review journals if people just publish everything, imagine the volume. And imagine the volume of crap – as I pointed out in a blog post about a year ago – the internet now, and just the newspapers in the past already sometimes serve this function – look at the cold fusion story – where the science never even made it to peer review.

Also what if a ton of people think its crap but it really isn’t crap – which would be sort of opposite of the recent arsenic story.

This sort of implies and it has been implied that peer-review as it stands now is a bad thing
Why do so many people think everything is wrong with peer review? Perhaps it is these articles we see on Twitter – the ones where we see where the peer review process has missed something, but there are far more stories about where this doesn’t happen, I would suspect, you just never hear about them.

Does peer review go wrong? Yes
Do we hear about where peer review failed in the news? Yes – think of the recent example of arsenic.

Are there problems with the peer-review process? Of course.. look at Jan-Hendrick Shoon and recently Amil Potti

Do most peer-reviews articles NEVER make it to the news? Yes

But will publish first filter later make this better? I really doubt it.

Peer review is a team of people who are ‘experts’ in your field who look at your work and assess it – why is this altogether a bad thing?

In fact it can be, and largely is, a good thing. I have written papers – especially my first papers as corresponding author, where I was so focused on the forest I didn’t just miss the trees I actually, unknowingly, cut down the trees. After these manuscripts were peer reviewed – it was clear to me, from the reviewer’s comments – that they had no idea what I was talking about. Why? Because I wasn’t at all clear – so I had to fix it and as a result had a better paper. I have also had reviewers suggest work to me by other that I wasn’t aware of, often in agreement of what I was saying. That not only strengthened what I was saying in a particular paper but also my research in general.

If everything was published first – would be people take the time to even look at my piece of research to tear it apart? I am not an FRS or a Nobel prize winner, nor am I likely to be, but when I submit my paper to a suitable journal I have the same equality as anyone else in getting it read (this isn’t true for all journals, but specialists journals, certainly) – and I want it to be ‘torn apart’ to see if my science withstands the test because THIS is the point, whether it scares me or not.

March 28, 2010

Should scientific peer review be thrown on the scrap heap?

Filed under: science — sylviamclain @ 10:40 pm
Tags: ,

Should peer-review be done away with in the sciences?

The assurance of quality through the peer-review process is the sacred cow of modern scientific practice.

Richard Smith thinks its time the cow is taken to the abattoir – that scientists should publish first and then let the scientific community decide afterwards – or as he puts it ‘publish then filter’.

Despite Smith’s alternative view, the peer-review process is essential to all of the sciences.

Peer-review is like democracy, just because it isn’t a perfect process doesn’t mean it should be tossed aside – it’s important. It’s important because it allows someone else to assess, in principle independently, the validity of scientific work. Just because it is not always done well, doesn’t mean it should not be done at all.

Smith began this debate in the British Medical Journal after Cathie Sudlow called into question the practice of peer-review in the same journal .

Her criticism centers on a recent ‘publication’ in Science, the flagship journal of the American Association for the Advancement of Science (AAAS). This study apparently presents evidence suggesting a viral link to the elusive disease Chronic Fatigue syndrome, perhaps giving false hope for many Chronic fatigue sufferers.

Sudlow has criticized the methods used in the Science paper and cited several other studies which have found no causal link between the virus and Chronic Fatigue in her support. As a result, she questions the practices of peer-reviewed scientific journals, especially top-journals such as Science – a reasonable use of babies and bathwater for population reduction – and believes there should be more responsibility from editors and reviewers: that journals shouldn’t rush to publish (even though she wanted them to rush to publish her E-letter which expressed doubt concerning the methods used in the Science article) – and authors themselves to not overstate their results.

The peer-review process is essential to publication in any scientific discipline.  Even though physicists, mathematicians and astrophysicists as Smith claims do, sometimes, pre-publish on the web for comments from the community (such as arXiv) – they ultimately publish in the same way as the other sciences – e.g. through peer-reviewed journalism.

Which is a good thing.

The process is undoubtedly imperfect but it still remains sound practice.  Scientific work, once published, is not set in stone but rather opens up a discourse between scientists – simply put, if you don’t agree with the science published, publish work which calls this into question yourself.  This too is an essential part of the scientific process, which Sudlow herself is actively and appropriately engaging in.

‘Publish then filter’ is not a good answer, at least as Smith depicts it.  In fact this mechanism is already in wide (though perhaps not widely accepted) practice.  Lots of people do already ‘publish then filter’ outside of the scientific literature – and lots of silly non-scientific ‘science’ makes it into the news as both Ben Goldacre and Bob Park have often pointed out. The mad also have fun with random ‘press-science’ such as the law suits to prevent the LHC from running, because it will end the world, and put ninja-seagulls from the future, armed with baguettes, out of business (but then they knew that anyway – they’re from the future (but they are seagulls…..)).

In this way, the ‘publish then filter’ mechanism is already in place, but doesn’t need to be included in the scientific literature itself.  Just because it’s in the news, doesn’t mean its good, or real for that matter, science.

‘Publish-then-filter’ kills one of the main features of peer review in science – the removal of obvious nonsense from the wider community’s scientific discussion.

Setting aside the ‘slash-and-burn’ model of changing the system of scientific evaluation, there are undoubtedly some ways that peer-review and scientific dissemination in general could be improved.

For example, in opposition to the ‘full disclosure’ of authors and referees Sudlow recommends, double-blind peer-review could be implemented, where both the authors (and their institutions) as well as the referees are anonymous.

There are several studies which indicate that this could be effective. For instance, see a studies in the The American Economic Review and in Behavioral Ecology.

Further, as a tool to improve dissemination, institutions such as the AAAS and the Royal Society could have websites or blog posts allocated to certain topics, with a short review, such as the science-writer Philip Ball has on ‘Water in Biology’. This would allow for easier access to work spread throughout many journals on a particular topic.

The peer-review question notwithstanding, there is a large underlying issue to this debate which addresses the culture and practice of modern science. Institutional practices as well as the ethos of the scientific community itself needs to be reassessed. But that is for another day….

Blog at

%d bloggers like this: