Does peer-review need a peer-review? Or should skeptics be more proactive with scientific journals?
For those not in the know, Frenkel et al is an apparently peer-reviewed scientific paper from the MD Anderson cancer research centre, Texas, purporting to demonstrate that homeopathic ultra-diluted remedies (i.e. those with any active ingredient diluted to the point where the chances of finding 1 molecule of active ingredient is negligible.) can kill off certain cancer cell lines in the context of cell culture. Unsurprisingly, this paper has been widely toted by various pro-homeopathy websites, with cringe-worthy tag lines such as:
Sadly for homeopaths, perhaps rubbing their hands with glee at the prospect of an epic skeptic climb-down, the paper turns out to be garbage. As soon as the paper was obtained it was clear that the experiments and the analysis thereof were fatally flawed. In a nutshell, the homeopathic remedies were made up in high concentrations of extra neutral ethanol (as is often done). The ethanol concentrations were not controlled for, which essentially makes all conclusions invalid because ethanol alone, even at low concentrations, can have profound effects on cells. For more information see take-downs by Dr Rachie and Orac. It is also worthy of note that it appears that an unwitting author on the paper has come forward to also cast doubt about the conclusions in the paper:
“Therefore, I believe this study demonstrated changes in alcohol percentages on cells rather than the efficacy of homeopathic medicine.”
So – we have a journal article that even some of the authors express doubt in the findings of.
The website of IOCs has all manner of woo within, from “superfoods” to “homeopathy for cancer”. It also indulges in a bit of bait-and-switch – it says stuff like:
“The main benefit that these treatments provide is improvement in quality of life and enhanced wellbeing. Rather than competing with conventional cancer care, complementary therapies are complementing conventional care as part of comprehensive cancer care.”
— and elsewhere directs you to the aforementioned Frenkel et al paper – which seems to be suggesting that Homeopathy can indeed compete with conventional cancer care.
However, given the fact that Frenkel et al has been published in a relatively obscure journal, and has really only been a debating point on skeptical and homeopathy/altmed websites, what’s the harm?
The most asinine man in UK politics.
“and the university of Texas has shown the positive effect of homeopathic treatments in killing cancer cells while maintaining good cells.”
Following this, Treddinnick proposed a slew of pro-homeopathy EDMs, which are once again, attracting signatures from the more credulous member of parliament. One of these EDMs deals specifically with the Frenkel et al paper.
Followers of woo in the UK parliament will be aware that it was Tredinnick who proposed Early Day Motion 908 which expressed concern with the scientifically robust conclusions of the Science and Technology select committee evidence check into homeopathy – EDM 908 was deliciously relabelled by Prof David Colquhoun as “A handy list of dimwitted members of parliament.
Evidence based Policy?
But can we really blame Tredinnick for using this paper in parliament? Personally, I am all for MPs referring to peer-reviewed papers in parliamentary debates. “Evidence based policy” was an idea floated by various parties in the run up to the election, which I ( and I think pretty much all skeptics) were broadly supportive of.
So can we castigate Tredinnick for quoting a paper, which he may have taken on good faith as being scientifically credible? Tredinnick has no scientific qualifications and, assuming he has read the paper, may not have been in a position to question its accuracy, methodology or conclusions. Assuming he has read the paper.
If he has not read the paper, then should he be touting it in parliament? Probably not, although I imagine an awful lot of papers, books and articles get mentioned in parliament, without actually being read by the mentionner.
So sadly, whilst it would be easy to dismiss this as Tredinnick rattling his woo-sabre, whilst this paper has not been retracted, we cannot really fault an MP for quoting it in parliament. However much we might like to.
The continued existence of this paper is a failure of peer-review.
Dr Michael Brooks, writing in the New Scientist suggests that (because of this incident) peer-review needs an overhaul. Whilst I agree that this paper needs retracting, I think that an overhaul of peer-review may be a little premature. Let us not forget that according to PubMed 848,345 papers were published in 2009 — it is inconceivable that a few duff ones wouldn’t slip through the net.
Also, the formal peer-review process is just the first rung of the peer-review ladder, albeit the most critical. Every time a scientist reads a paper, that is another instance of peer-review. A paper’s conclusions and methods should be critically appraised by every single reader — this is why the “methods and materials” section is perhaps the most critical section of any paper. Flipping straight to the conclusions section without understanding how the authors reached those conclusions is folly.
In a case such as this, where:
- a paper is being touted to push unproven homeopathic therapies on desperate cancer patients,
- two scientists (Dr Rachie and Orac) have carefully demonstrated that the paper is fatally flawed,
- an author on the paper publically express concerns about the validity of the conclusions.
should we skeptics not take the fight to the journal editor?
After all, there is considerable overlap in the Venn Diagram of “research scientists” and “skeptics” sets. A letter to the editor of this journal, with suitable signatories, should produce the desired effect.
So… who’s up for writing that first draft?
EDIT: Minutes after posting this blog post 2 things happened:
- @JDMoffatt (himself a published scientist and academic) told me that he had, in fact, written to the editor of the journal in which Frenkel et al was published, but he hadn’t had a reply as yet.
- @JDMoffatt received a reply from the editor of the journal in which Frenkel et al was published.
The reply was rather unhelpful, and hardly the full and proper response that one might expect from a journal editor:
“Thank you for your comments. We are aware that the subject is controversial. We would be pleased to evaluate data (my emphasis) supporting your views if submitted”.
This is NOT how peer-review works. If someone has issues with a published paper, they write a letter to the journal (for publication) and this letter is given to the original authors to respond to (if they wish) and both letters are published side-by-side. The onus is not on the person raising issues to provide new data to refute it (although they can). They are not ‘disproving’ the original article – they are raising issues with the work within it. Grr.