This is not good. Not good at all.
On Friday, Paul Goldberg of The Cancer Letter reported on an investigation into Duke cancer researcher, Anil Potti, MD, and claims made that he was a Rhodes Scholar - in Australia. The misrepresentation was made on grant applications to NIH and the American Cancer Society.
The Cancer Letter, a $375/year go-to newsletter on cancer research, funding, and drug development, has made this issue free at this PDF link.
News & Observer higher education reporter, Eric Ferreri, has a nice overview of the situation. Potti has been placed on administrative leave by Duke and the American Cancer Society has suspended payments on his grant and initiated their own investigation.
This news follows on questions regarding Potti's highly-promoted research conducted in the lab of Joe Nevins at Duke. From The Cancer Letter PDF on page 6:
The Nevins and Potti team emerged as pioneers of personalized medicine in 2006, when Nature Medicine published their paper claiming that microarray analysis of patient tumors could be used to predict response to chemotherapy.
However, two biostatisticians at the MD Anderson Cancer Center attempted to verify this work when oncologists asked whether microarray analysis could be used in the clinic. Keith Baggerly and Kevin Coombes, the statisticians, found a series of errors, including mislabeling and an "off-by-one" error, where gene probe identifiers were mismatched with the names of genes.
Baggerly and Coombes said they devoted about 1,500 hours to checking Potti's and Nevins's work. These efforts--dubbed "forensic bioinformatics"--resulted in a paper in the November 2009, issue of the Annals of Applied Statistics.
The Nature Medicine paper lists two corrigenda on sample duplication and mistakes in the gene list that appear to have preceded the Baggerly and Coombes analysis.
According to current issue of The Cancer Letter and several previous issues, Duke terminated three clinical trials of microarray-based individualized chemotherapy but the trials were then restarted after evaluation of the methodology by outside experts. The report was considered confidential by a Duke official but Goldberg noted that following its submission to NCI, the report became subject to the Freedom of Information Act.
A link to the report is provided among the Special Reports list at The Cancer Letter website. The report details some of the problems with the data and the responses by the research team that included several corrections to other papers. However, Goldberg reports that Duke officials "were inaccurate in their description of the document's substance and conclusions when they announced completion of the investigation and resumption of the clinical trials earlier this year."
The personalized genomics work of Potti, now an associate professor of medicine at Duke, was widely publicized by the university and included television advertisements that aired locally as late as last year. When I attempted to access The Cancer Letter link to one of the announcements at Duke's YouTube page, the URL returned with the message, "This video is private."
This latest development reported by Goldberg also includes several other inconsistencies in Potti's biosketches regarding residency sites and awards from the American Society of Clinical Oncology.
Addendum (19 July, 11:45 am) after thinking more about this overnight: What I don't understand from a grant reviewer's standpoint is why Potti thought that being a Rhodes Scholar would make a damn bit of difference in how he was evaluated as an investigator.
- Log in to post comments
The Rhodes stuff is easy to understand, but for some of us, misses what is most interesting.
Baggerly and gang have been following this group's work and giving talks at many universities for several years, of which statisticians in bleeding edge medical research with high-dimensional data are most aware. Try both the recent Annals of Applied statistics article (it should have appeared in a more bio-info place perhaps) and some previous stuff:
http://www.nature.com/nm/journal/v13/n11/full/nm1107-1276b.html
where Coombes, Wang, and Baggerly show more details about their work on the 2006 Nature Medicine paper of Potti,...,Nevins. (That is, this doesn't "appear to have preceded the Baggerly and Coombes analysis", but perhaps I am reading that wrong.) Amazing work. I'm afraid to really get on top of this situation will likely take many hours.
It is correct that there are two corrigenda associated with the Nature Medicine article. The first was posted in November 2007 in response to a letter to the editor by Coombes, Wang, and Baggerly (Microarrays: retracing steps; Nature Medicine 2007; 13(11):1276-7.) The second was posted in August 2008 and followed the submission by Coombes and Baggerly of another letter to Nature Medicine (which was not published, although some of the key points eventually appeared in the Annals of Applied Statistics article).
Who is hiring people for academia who doesn't know that Rhodes scholars only go to Oxford.
It's just ignorant. It's also ignorant of the fraudster.
"Addendum (19 July, 11:45 am) after thinking more about this overnight: What I don't understand from a grant reviewer's standpoint is why Potti thought that being a Rhodes Scholar would make a damn bit of difference in how he was evaluated as an investigator."
Because a Rhodes in most of the english speaking world is the highest award you can get under the age of 25. Even professors put it front and centre. It's that prestigious.
What's more disturbing, if the Cancer Letter article is right, is that his institution may be knowingly dismissing the criticisms of this dude's line of work. They provide a link to the internal study document but it is so heavily redacted that it is impossible to tell what the panel's real conclusions are. And I don't know why it is redacted, since it is a reanalysis, supposedly, of existing data, which is supposed to be publicly available once a study is published. The thing about allegedly lying in his CV, if true, then should put into question the whole integrity of this guy's work.
At least he was sensible enough not to claim a Nobel Prize...
Sorry to comment again, but it occurred to me after looking at this guy's lab website is what happens when a PI is found out to have established himself through misrepresentation of his credentials but has been running a successful lab for several years? Do you simply forgive old transgressions because, after all, all the recent work was done by his trainees who likely did not do anything wrong? Or does the university fire him and shut down the lab and the field discredits all of the recent work, despite the fact that the offending PI was unlikely to have done any of the experiments and analysis?
The claim isn't quite so obviously bunk, from what's stated in the linked article. Claiming to win a Rhodes scholarship in Australia is quite plausible. Plenty of Aussies have done just that, and accordingly taken off to Oxford to continue their studies. One famous winner is a former Prime Minister. You can't study on one here, but you can win one here.
It is still bunk because the Rhodes foundation denies it, of course.
In my one of my very favorite parts, I hope you can follow these sentences in "Microarrays: retracing steps" where our brave hero at #2 writes:
"4. For docetaxel, their software yields only 31 of their 50 reported genes. Of the remaining 19 (Supplementary Report 9), Chang et al.2 name 14 as useful discriminators in the paper that described the test set used by Potti et al.1. We do not know how these 19 can be obtained from the training data, and we suspect that they were included by mistake. The model may more easily predict test classes with these genes."
"31-nerds" letter to Varmus at:
http://news.sciencemag.org/scienceinsider/2010/07/duke-suspends-clinica…
@rork: Totally, that paragraph is classic. I also laughed out loud when I first read it 2 years ago.
Being very familiar with the paper and following criticisms, Baggerly et.al. were right to draw serious questions about how the gene expression signatures were created. Potti et.al. used an extremely elaborate method to derive them that involved "manual curation" of drug sensitivity values and many people couldn't recreate it.
BUT........it needs to be said that other criticisms by Baggerly et.al. were not widely accepted as being serious or substantial. In particular, the charge that Principal Components Analysis was used to create signatures using both the gene-expression training set and gene-expression test-set, and therefore caused a "leakage" of test data into training data was a not a credible criticism. Potti. et.al responded by saying that since drug response data was not used in this signature generation, this technique was appropriate. I would agree. This technique is just identifying common features in the gene expression data between training and test. Also, criticisms of "off by one errors" were correct but did not affect the results or interpretations of the paper.
For me to say whether using principal components from the combined data was acceptable or not would require that I understood the methods in detail - I do not. My compliments to any who can fully discern them.
That the criticisms were not widely accepted as being deadly may be true, judging from subsequent history of journal and funding agencies, but that might merely represent popular opinion. The influence of people best able to judge is limited, since the ox on their (and others) tongues is mighty heavy, they are few, and they may be perceived as outsiders. This is why I give so much credit to the Coombes criticism, and their general suggestions that it would be good if we can see your data and tell what you did. That problem certainly persists in the literature today.
Amusement: I did not drop the names of any of the 31 letter signers, since choosing was too difficult, and every nerdish reader would find your choice suboptimal ("they probably don't even know who Efron is, the fools" etc), but thegreatbeyond bravely did.
I also looked at the names of the 31 letter signers. Some pretty major names in the field of cancer genomics/bioinformatics are on it. What's so fascinating about this incident is that the Potti&Nevins publications/clinical trials based on that original work came so far. And yet there seemed to be this huge undercurrent of opinion against them in the scientific community. Much of this can probably be attributed to the reputation of Nevins and Duke. My prediction: the junior member of the Nevins&Potti team will be made the fall guy.