When an article was published in *Science *last week reporting that DNA samples from exceptionally long-lived individuals differed detectably from those of normal individuals, it got plenty of positive attention from the mainstream media. However, the buzz from experts was rapid and telling: my colleagues in the statistical genetics community weren't excited about the results, but immediately, profoundly skeptical.
For people who've spent years doing genome-wide association studies (GWAS), several things stand out as unusual from this paper: the very large effect sizes of the identified SNPs (as noted by colleague Jeff Barrett in the Guardian), the extraordinary claim that the identified variants were able to correctly classify individuals as potential centenarians with 77% accuracy (a totally unprecedented level of accuracy for a complex trait), the fact that the associated variants haven't previously been associated with protection against any other common diseases (as you might expect them to be, given longevity is effectively a matter of avoiding or surviving every common disease), and several subtle technical issues, such as a rather strange-looking Manhattan plot (shown at the end of this post).
If the paper's claims were true they would be truly remarkable. However, the general feeling from the GWAS community right now seems to be that the identified associations are likely to be largely or even entirely artefactual, the result of failing to fully control for differences in the genotyping methods used in the cases and controls. The study used a mixture of two different genotyping platforms (albeit both made by Illumina) for their centenarians, while the control data was taken from an online database containing samples examined using multiple platforms. Disturbingly, similar potential genotyping bias also affects their replication cohort.
In a great article in Newsweek today Mary Carmichael has a series of damning quotes from big-name geneticists casting doubt on the study's findings. deCODE Genetics CEO Kári Stefánsson is (unsurprisingly) the most vociferous: he notes that there are consistent and previously known genotyping problems on the SNP chip used in the study for the two most strongly associated SNPs, and then goes further to argue that technical problems probably underlie nearly all of the reported associations in the paper:
Carmichael goes on to note a major methodological flaw in the paper: the failure to even attempt to validate *any *of the associated SNPs on an independent platform. This is absolutely standard practice in normal GWAS, and should have been demanded by referees - especially given the extraordinary claims being made in the paper.
What needs to happen next? For a start, the authors should release the raw intensity data for their genotyping experiments, which would allow independent investigators to spot obvious problems. Doing so immediately on a public database would go a long way towards showing they're not trying to cover up any methodological flaws. Ideally, they should also validate their putative associated SNPs using an independent platform and release those raw data as well.
More broadly, this is an important lesson for the increasing number of investigators wandering into the GWAS arena: they need to be aware that the genotype data they're working with aren't just clean, digital data points, but best-guess estimates (typically very reliable, but sometimes badly flawed) based on an noisy fluorescent intensity signal. There's a reason why researchers working on GWAS spend so much of their time on a regimented series of upstream "data cleaning" steps and careful downstream validation of new associations - it's all too easy for noisy data to introduce bias that produces a false association signal. So, kids, don't end up in Newsweek for all the wrong reasons: talk to someone who really knows what they're doing when it comes to GWAS data.
Finally, major journals need to stop letting sexiness push aside scientific rigor. Carmichael says it nicely:
If the key results from this paper do turn out to be based on easily-detected experimental artefacts, Science deserves to be embarrassed.
Anyway, here's the image that really made me go "whoah" - the Manhattan plot from the paper, tucked away in the Supplementary Data, which shouts "artefact" to anyone who's seen even a few GWAS papers. For the uninitiated, each dot in the plot represents a different SNP, with the alternating bands of colour showing different chromosomes. The y axis indicates the strength of the association between that SNP and longevity. The plot is unusual for a GWAS in that all of the highest-ranked SNPs are hanging out there by themselves, rather than being flanked by a column of other associated variants - a pattern characteristic of genotyping error rather than true association.
In contrast, here are the Manhattan plots from a "good" GWAS - the Wellcome Trust Case Control Consortium's analysis of 7 different common diseases (I've trimmed out two uninteresting ones), with the statistically significant SNPs highlighted in green. You can see that basically all of the most strongly significant SNPs are found in a "tower", the result of nearby SNPs being correlated with one another and thus all marking the same association signal:
Spot the difference?
