Does peer-review need a peer-review? Or should skeptics be more proactive with scientific journals?
For those not in the know, Frenkel et al is an apparently peer-reviewed scientific paper from the MD Anderson cancer research centre, Texas, purporting to demonstrate that homeopathic ultra-diluted remedies (i.e. those with any active ingredient diluted to the point where the chances of finding 1 molecule of active ingredient is negligible.) can kill off certain cancer cell lines in the context of cell culture. Unsurprisingly, this paper has been widely toted by various pro-homeopathy websites, with cringe-worthy tag lines such as:
Skeptics May Have to Rethink Their Opinions–Homeopathy May Really Work
Sadly for homeopaths, perhaps rubbing their hands with glee at the prospect of an epic skeptic climb-down, the paper turns out to be garbage. As soon as the paper was obtained it was clear that the experiments and the analysis thereof were fatally flawed. In a nutshell, the homeopathic remedies were made up in high concentrations of extra neutral ethanol (as is often done). The ethanol concentrations were not controlled for, which essentially makes all conclusions invalid because ethanol alone, even at low concentrations, can have profound effects on cells. For more information see take-downs by Dr Rachie and Orac. It is also worthy of note that it appears that an unwitting author on the paper has come forward to also cast doubt about the conclusions in the paper:
“Therefore, I believe this study demonstrated changes in alcohol percentages on cells rather than the efficacy of homeopathic medicine.”
So – we have a journal article that even some of the authors express doubt in the findings of.
Frenkel himself has apparently left the MD Anderson cancer centre (Hat tip to @medtek off of Twitter for that one) , and has set up his own company integrated oncology consultants
The website of IOCs has all manner of woo within, from “superfoods” to “homeopathy for cancer”. It also indulges in a bit of bait-and-switch – it says stuff like:
“The main benefit that these treatments provide is improvement in quality of life and enhanced wellbeing. Rather than competing with conventional cancer care, complementary therapies are complementing conventional care as part of comprehensive cancer care.”
— and elsewhere directs you to the aforementioned Frenkel et al paper – which seems to be suggesting that Homeopathy can indeed compete with conventional cancer care.
However, given the fact that Frenkel et al has been published in a relatively obscure journal, and has really only been a debating point on skeptical and homeopathy/altmed websites, what’s the harm?
The most asinine man in UK politics.
Well – recently the paper resurfaced again, as it was mentioned in UK Parliament by conservative MP for Bosworth, David Tredinnick, as part of his proposals for an integrated healthcare bill.
“and the university of Texas has shown the positive effect of homeopathic treatments in killing cancer cells while maintaining good cells.”
Following this, Treddinnick proposed a slew of pro-homeopathy EDMs, which are once again, attracting signatures from the more credulous member of parliament. One of these EDMs deals specifically with the Frenkel et al paper.
Followers of woo in the UK parliament will be aware that it was Tredinnick who proposed Early Day Motion 908 which expressed concern with the scientifically robust conclusions of the Science and Technology select committee evidence check into homeopathy – EDM 908 was deliciously relabelled by Prof David Colquhoun as “A handy list of dimwitted members of parliament.
Evidence based Policy?
But can we really blame Tredinnick for using this paper in parliament? Personally, I am all for MPs referring to peer-reviewed papers in parliamentary debates. “Evidence based policy” was an idea floated by various parties in the run up to the election, which I ( and I think pretty much all skeptics) were broadly supportive of.
So can we castigate Tredinnick for quoting a paper, which he may have taken on good faith as being scientifically credible? Tredinnick has no scientific qualifications and, assuming he has read the paper, may not have been in a position to question its accuracy, methodology or conclusions. Assuming he has read the paper.
If he has not read the paper, then should he be touting it in parliament? Probably not, although I imagine an awful lot of papers, books and articles get mentioned in parliament, without actually being read by the mentionner.
So sadly, whilst it would be easy to dismiss this as Tredinnick rattling his woo-sabre, whilst this paper has not been retracted, we cannot really fault an MP for quoting it in parliament. However much we might like to.
The continued existence of this paper is a failure of peer-review.
Dr Michael Brooks, writing in the New Scientist suggests that (because of this incident) peer-review needs an overhaul. Whilst I agree that this paper needs retracting, I think that an overhaul of peer-review may be a little premature. Let us not forget that according to PubMed 848,345 papers were published in 2009 — it is inconceivable that a few duff ones wouldn’t slip through the net.
Also, the formal peer-review process is just the first rung of the peer-review ladder, albeit the most critical. Every time a scientist reads a paper, that is another instance of peer-review. A paper’s conclusions and methods should be critically appraised by every single reader — this is why the “methods and materials” section is perhaps the most critical section of any paper. Flipping straight to the conclusions section without understanding how the authors reached those conclusions is folly.
In a case such as this, where:
- a paper is being touted to push unproven homeopathic therapies on desperate cancer patients,
- two scientists (Dr Rachie and Orac) have carefully demonstrated that the paper is fatally flawed,
- an author on the paper publically express concerns about the validity of the conclusions.
should we skeptics not take the fight to the journal editor?
After all, there is considerable overlap in the Venn Diagram of “research scientists” and “skeptics” sets. A letter to the editor of this journal, with suitable signatories, should produce the desired effect.
So… who’s up for writing that first draft?
EDIT: Minutes after posting this blog post 2 things happened:
- @JDMoffatt (himself a published scientist and academic) told me that he had, in fact, written to the editor of the journal in which Frenkel et al was published, but he hadn’t had a reply as yet.
- @JDMoffatt received a reply from the editor of the journal in which Frenkel et al was published.
The reply was rather unhelpful, and hardly the full and proper response that one might expect from a journal editor:
“Thank you for your comments. We are aware that the subject is controversial. We would be pleased to evaluate data (my emphasis) supporting your views if submitted”.
This is NOT how peer-review works. If someone has issues with a published paper, they write a letter to the journal (for publication) and this letter is given to the original authors to respond to (if they wish) and both letters are published side-by-side. The onus is not on the person raising issues to provide new data to refute it (although they can). They are not ‘disproving’ the original article – they are raising issues with the work within it. Grr.
Anomalous Distraction 
Nice post but I think I agree with the end of Michael Brooks’s piece – where he concludes that a second layer of public evaluation should not be necessary.
It’s easy for scientist to filter the dross out for themselves and they’re probably sufficiently motivated to seek out other opinions on papers but for lay people this is a bit of a tall order.
Personally I see poorly designed, executed and written studies every day, even in well regarded journals. I think the greatest problem facing peer review is perhaps a lack of consistency. The assumption implicit in the peer review system is that all reviewers have the same standards of rigour but this just isn’t true. A revision of the peer review to provide greater consistency is necessary.
But for the record, David Tredinnick is still a bell-end.
Obvious question – how do you design in better consistency into peer-review? Training courses for reviewers?
Fair point and the million dollar question as far as I’m concerned…
Perhaps a clear set of guidelines from the journal itself? For instance there are some journals with rules regarding the reporting of research involving transgenic animals (can’t remember which ones exactly but will update with a link when I do). Reporters of studies involving transgenics seem to be particularly guilty of citing other authors, rather than describing their own methods of breeding and maintaining colonies of transgenics, something which can result in considerable variation in phenotype from lab to lab.
Of course, these are just individual instances and if you’re determined to submit without a full description you can always submit somewhere else but it the NAture group journals for instance were to adopt a set of guidelines other journals would quickly follow suit.
Link to Genes, Brain and Behaviour article outlining standards for reporting experiments in transgenic mice.
http://www.hubmed.org/display.cgi?uids=18778401
Sorry it’s behind a paywall but I can provide a .pdf to anyone who’s interested.
I’m getting more and more driven to the conclusion that peer review has reached the end of its useful life. It’s function has become not so much to maintain the quality of published papers, but rather to maintain a hierarchy of journals. Just about any old crap will get published in some journal or another and claim to be ‘peer reviewed’.
The hierarchy of journals is unhelpful too. We have got to the silly situation where everyone is wasting time trying to publish in the same few journals. Worse still, publication on a “top journal” isn’t always a guarantee of quality. Top journals are very prone to make judgements on the whims of fashion and in trying to scoop other “top journals”.
Worse still. the problem arises from scientists themselves or at least ex-scientist apparatchiks who are obsessed with quantity rather than quality -for example see http://www.dcscience.net/?p=182 and http://www.dcscience.net/?p=186
Soon I expect we’ll be publishing on the web and opening comments for reviewing. There will be no more rubbish than now and the amount saved on journal subscriptions will be astronomical. Bye bye Elsevier.
Good post.
Incidentally, if anyone says something like “the university of Texas has shown” you can be sure they are scientifically illiterate.
I completely agree with DC on peer review. It made sense 50 years ago when there was limited space in print journals, and so it would have been financially impossible to print everything. And it would have also been difficult to publish debates and criticisms. So, just to save space and money, it was fine to get the criticism out of the way before publication so readers could “take it as read” that it had been scrutinized, and found acceptable.
But with the internet, there’s no longer a barrier to publishing absolutely anything and any number of comments. Doing so would, I think, destroy the false mystique of “peer reviewed science” (because frankly, this is no longer an accolade) and would also force scientists to engage in debate (many people take the attitude that “once it’s published, I don’t need to defend it” which is understandable, but damaging in the long run.
Interesting points.. but if we move away from peer-review, how do we as scientists prevent non-scientists from accepting crappy science as fact? Whilst you are right to say that crap does get published now – a system with even laxer controls would surely lead to an increase in the amount of crap?
My fundamental worry with the internet route of publishing is that any old shite would get published, and without at least a baseline level of prior scrutiny people such as Tredinnick would start quoting ‘scientific’ reports that made ludicrous and unsupported claims willy-nilly.
Whilst this is obviously already happening (albeit at a fairly low level – at least in the political arena) – wouldn’t moving away from a journal/publishing-house based system allow pseudoscience and crap-science to get bigger and better exposure?
How do people envisage tackling this sort of problem?
Is an online peer-reviewed journal like PlosOne a happy medium?
I do not believe that peer review has outlived it’s usefulness. I do believe that we have reached a point in time where we could consider changes to the way that peer review operates, including a role for non-peers opinions to be considered as part of assessing the overall utility of the paper. This second half is problematic due to the technical complexity of many papers and also the reason why I see David’s idea having little chance of success. Essentially there is a need for a certain level of competence in some aspect of the paper in order to genuinely be able to detect errors in data, methodology or reasoning on behalf of the researchers.
Once you find a person with that level of competence they are essentially a peer in any case, this definition is somewhat broader than currently used but may be functional. Also, are we seeing an overreaction against peer review. It is true that we have seen some recent high profile failures, but it would take a considered review of of the peer review process itself before I would be convinced that we have a major problem on our hands and not just a few examples of bad process that we should treat as experiential learning to improve the existing system.
Certainly there are unsolved problems about self-publication. One of them is that, when it has been tried, people have been quite reluctant to make critical comment publicly.
Nonetheless, it seems to be time to start thinking about the present unsatisfactory situation. Pubmed (the database of the U.S. National Library of Medicine
and the National Institutes of Health) lists no fewer than 36 peer-reviewed journals about ‘complementary therapies’, including even Homeopathy.
Of course the problems are less severe in the ‘harder’ subjects, like, for example, biophysics and crystallography. But they are very serious particularly bad in clinical subjects.
A recent editorial in the European Journal of Clinical Investigation said
.
If we go on like this, science itself will be discredited.
What worries me is the inclusion of what appear to be vanity publications into pubmed, as DC more politely points out. When “editors” (or editorial numbskull staff?) think that any old bullshit “controversy” is equivalent to the rigor of proper intellectual debate, I think we need a red card system, or something similar.
While many have pointed out the faults in the Frenkel paper, no one has done a similar appraisal of other papers in the same number. I wonder (but am not an oncologist) whether they are all equally flawed and bullshit-laden. I was careful to point this problem out the editor in my letter to him, but he (or his minions) don’t seem the least worried about the reputation of the journal. Indeed, I suspect that they are rather enjoying the attention that they are getting. It is sad that they appear to be happy to see science discredited, and yet are scientists themselves.
I would welcome the opinion of any oncologists who know and have evaluated the work of Spandidos.
I speak fro a position of uncertainty here, but it seems to me that once not so long ago, editors rotated in and out of professional journals from the membership of those professions. Are we now seeing professional editors who have no connection to the professions who’s journals they edit? Could this be contributing to a decline in standards?
The problem is that it is hard to see how you could change the review process to improve it. Already the most competent and conscientious reviewrs are grossly overloaded with work which does little more than maintain a hierarchy among journals.
Short of limiting the number of publications (highly desirable but probably not achievable while we are ruled by bean counters) it is an unsolvable problem.
The defensive reaction of the editor of the journal to the letter from James Moffatt is par for the course, sad to say,
Why not send another letter, rhis time signed by as many people as possible, demanding retraction A letter for publication in the journal too, and perhaps a complaint to the publishers of the journal about the standard of editing, could add to the pressure too.
The publication of this sort of thing isn’t trivial. It could kill people.
A mass email or petition might have some impact. Indeed, my original idea was to contact every senior author who had a paper published in the same number of the journal and encourage them to request that the dodgy paper in question was withdrawn.
I changed my mind after considering how these authors might feel about such a request. And by “these authors” I mean people who possibly think that publication in this particular journal was a high point in their career.
After receiving the terse and vain reply from the editor, my inclination is to let sleeping dogs die. I have to agree with DC that in the current climate this sort of problem is par for the course. While bullshit scientists are rewarded for publishing bullshit science in as many bullshit journals as possible, this is going to continue.
(That said, a really *huge* petition might shame the journal and it’s editor. Virtually all academics have a computer with internet access, many read blogs, and some even twitter. Does any social media guru feel like getting the ball rolling? It might save some lives.)