In the presentation to the public of the findings of the Cook survey in The Consensus Project, we already saw the consensus being inflated well beyond what was actually surveyed. This post will focus in on the small piece of text in which the peer-review process is explained to the public.
As text (emphasis by the authors):
What is peer-reviewed, and why is it important? When a paper has been peer-reviewed, that means it has been evaluated by a number of qualified scientists and found to have followed legitimate scientific methods.
When seeing it explained like this, the peer-review process seems to be quite overrated. As far as I know the peer-review process is more like a quality control. Some peers looked at the paper and they found it okay to publish. That a paper has been peer-reviewed doesn’t mean it is necessarily true.
Although I have no reason to believe that peer-review isn’t a good tool in most cases, I dare to doubt that peer-review somehow proves that a paper followed legitimate scientific methods. There are examples of tests that have been done with bogus papers or papers with errors in it, not or sparsely picked up in the peer review. There is no guarantee whatsoever that a paper is better because it is peer-reviewed.
Brighter minds than me also seems to think so. Just look at the wiki page of peer-review (my emphasis):
Richard Horton, editor of the British medical journal The Lancet, has said that
The mistake, of course, is to have thought that peer review was any more than a crude means of discovering the acceptability -not the validity- of a new finding. Editors and scientists alike insist on the pivotal importance of peer review. We portray peer review to the public as a quasi-sacred process that helps to make science our most objective truth teller. But we know that the system of peer review is biased, unjust, unaccountable, incomplete, easily fixed, often insulting, usually ignorant, occasionally foolish, and frequently wrong.
This is not hard to understand. The peer who reviews the paper could be an expert on the topic or it could be someone from a different discipline. It could be someone who says it is good enough if there are no obvious errors in it, or it could be someone who checks the paper thoroughly. It could be a neutral person, but could also be a competitor or a sympathizer. This could lead to things like a long time from submission until publication, a reviewer who may demand a futile and time consuming extra analysis (like the O’Donnell et al paper in which the methodology of another paper was disputed and where the author of the criticized paper himself was an anonymous reviewer – in that position he then asked changes to be made to the paper, which he then publicly ridiculed later),…
Wouldn’t that be an issue in groups with only few experts and polarized subjects, as I think is climate change? Then the process is biased towards the majority.
I have seen this inflated focus on peer-review many times before. Could it have something to do with the things we saw in the climategate emails, in which a small group of climate scientists were working together to exclude non-majority views via the peer-review process? If this is true, it could explain that focus on peer-review, because this is how the majority view is in the advantage. Not necessarily because of the science, but rather because of the process.
Don’t understand me wrong. I am not saying peer-review is necessarily a bad process. The point is that with the glorified way the peer-review is brought to the public the peer-review process is overrated.
The peer-review explanation ended with the funny quote at the end that most of the claims that are made by global warming skeptics on TV, in print and online are not based on legitimate science (emphasis by the authors):
Most of the claims that are made by global warming skeptics on TV, in print, and online are not based on legitimate science.
Have I got a surprise: most of the claims that are made by global warming alarmist on TV, in print and online are not based on legitimate science either.