a blog about genomes, DNA, evolution, open science, baseball and other important things

Yesterday I got incensed about a quote in a story in the NYT from a prominent autism researcher about the significance of findings in their recent paper (which described the sequencing of protein-coding genes from autistic individuals, their parents and siblings).

The statement that so offended me, from the lead author of the paper, was that he was 99.9999% sure that a gene identified in his study plays a role in causing in autism. It’s a ridiculous assertion, completely at odds with what his group says in the paper. It  needs to be corrected.

However, my original post also included some statistical analyses that were based on a cursory reading their work, and, as a result, didn’t directly speak to the central claims of the paper. My basic critique is unchanged – the 99.9999% claim is insupportable – but this got me interested in the fine details of their results and analysis, and I thought it would be useful to post my thoughts here.

Let me also say at the outset that, while I didn’t like what was said in the NYT, and don’t agree with everything the authors say in the paper, I am not trying to take them to task here. Analyzing this kind of data is difficult, and there are all sorts of complexities to deal with.

First, a summary of what they did and found. The core data are sequences of the “exomes” (essentially the protein-coding portion of the genome) from 238 children with autism spectrum disorders (ASD), both of their unaffected parents, and (in 200 cases) an unaffected sibling.

The analyses they present focus on the families with data for unaffected siblings, enabling them to compare the transmission of inherited variants (those present in the parents) and de novo mutations (those not present in the parents) between affected and unaffected siblings. They observed no differences in the transmission of inherited variants between affected and unaffected individuals, but found a significant increase in the number of de novo mutations that change proteins in affected individuals. Specifically, there were 125 do novo, protein altering (mis-sense, nonsense or splicing) mutations in affected individuals compared to 87 in siblings, a significant difference (p=.01).

This observation provides reasonable support for the hypothesis that de novo mutations are associated with autism. But it does not indicate which – if any – of these specific mutations is involved. After all, that there were 87 de novo protein-altering mutations in unaffected siblings suggests that many of those identified in autistic individuals are not involved in the disease. There is also the possibility that the elevated mutations rate is a secondary consequence of some other factor that leads to autism, and that none of these specific mutations are actually directly involved in the disease.

Given that the the observation of a de novo mutation in one gene in one affected individual conveys limited information about its involvement in autism, the authors focus on cases where independent mutations were observed in the same gene in different affected individuals. The reasoning is that such an observation would be unlikely to occur by chance in a gene that is not involved in ASD.

This is where I initially mistook what they did. I assumed the quote in the NYT story was referring to the chances of observing the same gene twice amongst the 125 de novo mutations in affected individuals, and pointed out that we actually expect this to happen at least 30% of the time (I say at least because the 30% number comes from assuming random mutations are equally likely to occur in all genes – which is not correct for reasons such as differences in gene size, GC content, etc… – all things the authors factor into their calculations). Indeed, the observation in the paper that two genes are hit twice in the set of 125 is not a statistically significant finding, and, by itself would offer no evidence that these genes are involved in ASD – and the authors do not assert that it does.

Instead the authors focused on a small subset of mutations – those that introduce a premature stop codon into a gene (thereby generating a truncated protein, or, because of a process known as non-sense mediated decay, likely decrease the expression level) or alter a splice site (potentially affecting the structure of the gene). The numbers here are a lot smaller. In the affected individuals there were ten nonsense and five splicing mutations, while there were only five nonsense and no splicing mutations in the unaffected siblings. And, crucially, in the set of 15 such mutations one gene – SCN2A – appeared twice.

So now the question is, is this a significant observation? Under the simplest of models, if you picked genes 15 times randomly from a set of 21,000 you’d expect to hit at least once gene twice with a probability of around .005 – making it a reasonably significant observation.

However, this is actually an overestimate of the significance, as differences in gene size, base composition, etc… make it more likely that a random mutation will land on some genes than others, thereby increasing the probability of seeing the same gene twice. The authors did extensive simulations that take this into account, and, restricting their analysis to the 80% of genes expressed in the brain, they conclude the observation of two nonsense/splicing variants in brain expressed genes is significant, at a p-value of 0.008.

However, it is worth noting (from the authors Figure S8) that under conservative but reasonable estimates of the de novo mutation rate and number of genes involved in ASD, the degree to which the data implicate SCN2A specifically is weaker, with a q-value (probability that the gene is not involved in ASD under various models) of around 0.03. Again, this may seem a bit counterintuitive, given that their data say that it’s significant that they saw the same gene twice, and they found only one such gene, how could that gene not be involved? But one actually has more power to validate the general model that de novo nonsense/splicing mutations are contributing to ASD than you do to implicate specific genes. This is why State’s assertion in the NYT that SCN2A was 99.9999% likely involved in ASD was pretty egregious – it is simply not consistent with their own data.

There are a few other things to note here.

First, the p-values and q-values they report is not specific to an individual gene – it is the average probability of observing a double hit in non-ASD genes and the average probability that a double-hit gene is involved in ASD. But SCN2A is relatively large (2000 amino acids), and thus the observation of two mutations in this gene is somewhat weaker evidence for its involvement in ASD than it would be for a smaller gene. I haven’t done a full simulations, but given that SCN2A is 4-5x larger than average, it should be on the order of 20x more likely to be doubly hit by chance than a typical gene, and thus the average q-value reported is an underestimate. It would be easy, using the simulations the authors already have on hand, to ask what the false-discovery rate when the doubly hit gene is 2000 amino acids or longer. I suspect it would not longer be significant.

The model also fails to consider the possibility that such fairly significant mutations in many genes might be lethal, and thus would never be observed. Hard to get a great estimate of what fraction of genes this might be, and the number is probably small given that they’ll almost all be heterozygous, but, again, given that the observations are only marginally significant, this possibility seems worth considering.

Finally, the more I read the paper, the more uncomfortable I grew with the way that the paper moved back and forth from non-synonymous to nonsense/splicing mutations, depending on where they got statistical significance. They start out by arguing that the there is a significant increase in the number of de novo synonymous mutations in ASD affected individuals. They get statistical significance here because there are a relatively large number of such mutations. They then look for cases where the same gene was hit twice, and find two. But this is not a significant observations – failing to distinguish between the possibility that a subset of ASD-involved genes were being hit from the null model of genes being hit randomly. However, for one of these pairs they noticed that there were two nonsense mutations. There wasn’t a significant enrichment of de novo nonsense mutations in cases (10) vs controls (5), so they added in the five splicing mutations from cases (there were none in controls) and got a marginally significant enrichment (p=.02). Then they looked at how likely it would be to find the same gene hit twice by nonsense/splicing mutations, and got a marginally significant result.

It’s possible to justify this path of analysis from first principles, as nonsense/splicing mutations are difference from missense mutations – and maybe this was part of the analysis design from the beginning. But the way the paper was set out, it felt that they were hunting for significant p-values – which is a big no-no. What if they had observed that highly conserved amino acids in some gene had been hit by the same missense mutation in two families? Would they have pursued this result and evaluated its significance? This is a crucial question, because if they pursued the nonsense observation simply because it was what they observed, then their statistics are wrong, as they need to be corrected for all the other possible double-hit leads they would have pursued. This is not a subtle effect either – such a correction would almost certainly render the results insignificant.

I don’t know the details of how this experiment was planned. Maybe they always intended to do this exact analysis in the first place, and thus it’s completely kosher. But the scientific literature is filled with post facto statistical analyses, in which people do an experiment, make an observation, and then evaluate the significance of this observation under the assumption that this was always what they were looking for in the first place.

It’s sort of like how, in baseball broadcasts, the announcers are always saying things like “This batter has gotten hits in his first at bat in 20 straight games played on Sunday afternoon”. They say this because it sounds so improbable – and in some sense it is, as this specific streak is, indeed, improbable. But if you consider all the possible ways you can slice and dice a player’s past performance, it is inevitable that there would be some kind of anomaly like this – rendering it statistically uninteresting.

I’m not saying that something this extreme happened in this autism paper. But the way the data were presented in the paper definitely made it seem like they were looking for a statistically significant observation on which to sell their paper (to Nature and to the public).

And it’s a shame – the data in the paper are cool. But does it really make sense to make such a big deal out of what is, at best, a single marginally significant observation? What if they hadn’t chosen one of those two families for their study? Would the result be uninteresting? Of course not.

In the end, what this paper should have said was, we generated a lot of cool data, we found some evidence that de novo mutations are enriched in kids with ASD relative to their siblings, but we need more data – a lot of it – to really figure out what’s going on here. Unfortunately, in the world we live in, this would have been dismissed as kind of boring, and likely not worthy of a Nature paper (although far less interesting genome papers are published there all the time).

So the authors made a big deal out of an interesting single observation, when they should have waited for more data. And then, probably for the same reasons, they oversold the result to the press – and ended up expressing an  indefensible 99.9999% confidence in SNC2A’s involvement in ASD to a reporter.

And I hope you understand now why it pissed me off.

[UPDATE: There is a followup to this post here.]

Last week Nature published the results of three studies (1,2,3) looking at the sequences of protein-coding genes from hundreds of individuals with autism and their parents. The main results are that there is a higher rate of de novo mutations in affected individuals, that these primarily come from fathers, and that the affected genes are enriched for those involved in brain development and activity.

I think a bit too much is being made of these studies – they’re generally technically sound, but there remains no definitive link between any single mutation or groups of mutations and the disease. However, the authors of one of the papers have made a big deal about having found a mutation in the same gene in two unrelated individuals. This is described in a piece last week by Benedict Carey in the New York Times:

In one of the new studies, Dr. Matthew W. State, a professor of genetics and child psychiatry at Yale, led a team that looked for de novo mutations in 200 people who had been given an autism diagnosis, as well as in parents and siblings who showed no signs of the disorder. The team found that two unrelated children with autism in the study had de novo mutations in the same gene — and nothing similar in those without a diagnosis.

“That is like throwing a dart at a dart board with 21,000 spots and hitting the same one twice,” Dr. State said. “The chances that this gene is related to autism risk is something like 99.9999 percent.”

Wow. 99.9999 percent. That’s impressive. But I have no idea where it came from.

If the study had looked at exactly two families, and they had found a single de novo mutation in the affected individual in each family, and these had been in the same gene, then yes, it would have been like throwing a dart at a dart board with 21,000 spots (roughly the number of genes examined) and hitting the same one twice – or roughly 1 in 21,000. But this is not what they did.

The study actually examine 200 families with an affected and unaffected siblings, and identified 125 variants with the potential to alter protein function. So the question is not how likely it is to hit the same spot if you throw two darts, but rather how likely it is to hit the same spot if you throw 125 darts at a dart board with 21,000 spots. The answer is that you would expect to have two dots hit the same spot 30.9% of the time. That is roughly one in three times. In fact, the 30.9% number is a conservative estimate that assumes that the odds of hitting any given gene are the same – this is undoubtedly not the case, as some genes are bigger than others – so the real odds that that the authors would have found the same gene twice purely by chance are even greater. Either way, it’s a very far cry from 99.9999 percent odds against.

UPDATE: Now that I’ve had a chance to look at the paper in more detail, I realize the authors were making a more subtle point about the nature of the mutations involved – highlighting the fact that they found two non-sense or splice mutations in the same gene. The authors did some fairly sophisticated simulations of the chances of this occurring and found, if they restrict their analysis to genes expressed in the brain, that the chances of this occurring by chance are ~0.8%.

This is not the same as throwing darts at a dartboard with 21,000 genes as there are only 14,000 brain expressed genes. But I agree with the authors that this is not a trivially expected result. Though I still have no idea where the 99.9999% part of the quote came from. Four orders of magnitude is a big difference.

What’s annoying here is not so much that the NYT used this quote (though they really need someone around to check these kind of things), but rather that the quote came from the lead author of the paper – Matthew State – a clinical geneticist at Yale.

I can not believe State said this this way. I hope he was simply misquoted. But if he really said this, and assuming that then he understands the basic statistics involved (which, given his position, I find highly likely), then he must have oversimplified and somewhat misrepresented the significance of his findings in order to make it sound more impressive in the popular press.

——————-

For those interested in how I came up with the 30.9% number, even if it might not be relevant, the question we want to ask is how likely is it that if we picked a random gene 125 times from a set of 21,000 we would never pick the same gene twice. Think of it this way. The first gene we pick can not overlap another gene. When we pick the second gene, 20999 times out of 21000 (probability .99995238) it will not be the gene we picked first. When we pick the third gene, we assume the first two went into different boxes (otherwise we’d be done already) so the odds go down slightly, to 20998 times out of 21000 (.99990476) and they keep going down slightly each time until we get to gene 125 when the odds are 20876 out of 21000 (.99409524).

The counterintuitive part of this is that even though at each step the odds are low, in order to end up with all of the genes in different bins you have to be on the right side of that random probability at each of 124 different steps. And to calculate the odds of this, you have to multiple all of these numbers together: .99995238 * .99990476 * …. * .99409524 which equals .69088693. That means that there is only a 69.1 percent chance that all 125 randomly chosen genes will be different – or 1 30.9 percent chance that you’ll see at least one gene twice.

It’s the same logic as the classic probability question of how many people you have to have in a room for the odds to be greater than 50% that two of them share the same birthday – the answer being 22.

UPDATE: Several people here and on twitter complained that my analysis did not take into account the controls in the paper – and implied that the results would be very different if I did.

The controls have a completely negligible effect. The critique the commenters raised that the authors didn’t just observe two hits to the same gene in the autism cases, they observed no hits to that gene in the controls. The papers states that there were 87 relevant mutations in the controls. So, conditioned on the observation that some gene was hit twice in the cases, we want to know how likely it would be that you would not hit that gene in 87 controls. The answer is 99.6%.

So, whereas I stated originally that the probability of hitting the same gene twice by chance in 125 random samples from a pool of 21,000 genes was 30.9%, if we now ask what is the probability of hitting the same gene twice by chance in 125 random samples from a pool of 21,000 genes AND not hitting the same gene in a set of 87 controls, the answer is 30.8%.