Friday, September 14, 2012

A genetic test for autism?

This week's big autism story was a genetic test able to predict with 70% accuracy [1] whether or not a child had autism. Rather than looking for a specific gene that might differentiate autistic from non-autistic people, Stan Skafidas and colleagues from Melbourne University developed the test by combining information about many different genetic variations. Critically, having developed the test based on one set of genetic data, they then tested the test on genetic data from a completely new set of people.

I don't want to take anything away from the basic science. And it would, of course, be incredibly useful to know early on whether or not a child is likely to develop autism. But the headlines are misleading. The unfortunate truth is that we're still a long way from a genetic test for autism. A screening measure with 70% accuracy would only be slightly better than completely useless.

Here's why.

Say you had 1000 kids and you ran the genetic test to see which ones would become autistic.

If we assume that the rate of autism in the population is around 1% then we'd expect 10 of the 1000 kids to be autistic. Given 70% accuracy, we'd expect 7 to show up on the genetic test as autistic.

The problem is the other 990 who aren't autistic. 70% accuracy means that 30% would be incorrectly diagnosed as autistic. 30% of 990 is 297.

Putting those together, our genetic test thinks that 304 of the 1000 children are autistic. Of those, only 7 really are autistic, the other 297 have been mis-diagnosed.

In other words, if your child took the test and the test came out positive, there would still be a 98% chance that your child was not autistic.

[Edit 15/9]: The researchers describe the test as a test of autism risk rather than a diagnostic test. In those terms, what it tells you is whether you have a  2% risk, as opposed to the usual 1% risk.

This screener's base rate fallacy is a well known problem. Dorothy Bishop covered it when discussing screening for autism based on speech recordings and MRI scans. Ben Goldacre talked about it in relation to screening for terrorists. It's a problem any time you're looking for something that's relatively rare. Even a really good test gets torpedoed by a disasterous rate of false positives.

To be fair, the researchers did suggest that the test would be most useful for parents who already had one autistic child. The latest research suggests that, if you have an autistic child, the chances of a second autistic child may be as high as 19%. Even in that situation, you'd still expect roughly two thirds of the kids picked up by the test to actually be non-autistic. [Edit 17/9]: And that's assuming 70% accuracy. In fact, the test was much poorer at discrimination between siblings with and without autism, suggesting that it's predictive value in the real world will be much lower.

Update (15/09/12)

On Twitter, Emily asked whether such a test could ever be useful.
As a population screening test, I think the answer would have to be no. Let's assume that the sensitivity (ability to detect true cases) goes up hand in hand with the specificity (ability to reject non-cases). Then for each level of accuracy, we can plot your autism risk if you test positive for autism. Even if the test was 90% accurate, you've still only got an 8% risk of autism if you test positive.



Update (17/09/12): Stan Skafidas has responded (at length!) in the comments.


Update (06/06/13): Dan Geschwind and colleagues have written a critical commentary of the paper in Molecular Psychiatry. See Neuroskeptic for coverage.


Update (23/10/13):
Robinson et al have published another critical letter, in which they describe a failure to replicate Skafidis et al.'s results. See Questioning Answers for coverage.


Notes

[1] "Positive and negative predictive accuracies were 70.8 and 71.8% respectively"

Reference

Skafidas E, Testa R, Zantomio D, Chana G, Everall IP, & Pantelis C (2012). Predicting the diagnosis of autism spectrum disorder using gene pathway analysis. Molecular psychiatry PMID: 22965006 Full Text

Related post

26 comments:

  1. I'd read the articles on this as the test being to identify potential 'risk of autism' as opposed to autism absolutely. Does that make sense? So you take a family with a prior ASD diagnosis and using this genetic information assess the potential risk for any other children? But I certainly take your points above.

    Having said that, I think if there is already a child with autism in the family careful observation of any further children will be the best guide. And certainly is what parents do already.

    ReplyDelete
    Replies
    1. I guess the point is that "identifying potential risk of autism" in this case means identifying a 2% risk as opposed to the usual 1% risk.

      As you say, if you've already got an autistic child in the family, I'm not sure how reassured you would be by a negative test result that missed 30% of cases!

      Delete
    2. Even worse, it cannot distinguish an unaffected sib from a proband with ASD (see figure 3b)

      Delete
  2. Hi Jon,

    This is an important issue, beyond the context of the specific paper. Usually there is a trade off between specificity and sensitivity. So in many cases detecting more true positives result in detecting more false positives. Recalling that autism diagnosis became much more sensitive imply that many autism diagnoses might be false .

    ReplyDelete
    Replies
    1. For a given test and set of data, there is indeed a tradeoff between sensitivity and specificity. If you increased the threshold for saying that someone was autistic, you'd increase the specificity but reduced the sensitivity.

      However, it might be possible to have a better test that was more sensitive and more specific at the same time.

      Delete
  3. Interesting paper. The validation in ethnically distinct populations is a strong point and it is interesting, but I'm skeptical, if these SNPs in 146 genes were that important, why didn't they show stronger signals in SNP GWAS? I mean the classifier means that multi-gene effects can be detected, while a GWAS is only univariate, but still, seems a bit odd, anyway the good thing about these kinds of studies is they're easy to replicate, someone will come along with their own dataset and try to check this soon I hope...

    ReplyDelete
    Replies
    1. That, and that's an awful lot of markers to predict a risk off.

      Delete
  4. If a test could be 85% accurate, and it is true that one in every twelve people must care about autism for society to be able to cope with autistic people, then having a test which causes enough people to care, but not too many, could be useful.

    ReplyDelete
  5. Hi Jon,

    Thanks your invitation to comment.

    The incidence rate of ASD is 1:150.

    Hence for any expectant mother, from the general population, that asks herself "Does the child I am going to give birth have or will have autism?" she will be on average 99.3% correct is she simply reassured herself and said "no". This is much better than the the 70% accuracy of our test.

    Where this test has utility is when the prior distributions/proportions change. What do I mean you ask? Well consider the situation where a 6 or 12 mth old presents to a pediatrician because they are missing developmental milestones or exhbiting traits that are consistent with Autism.

    There, like your analysis should have done, needs to consider the conditional probability of Autism given this other information manifest as traits or not meeting developmental milestones.

    I don't know how familiar everyone is with conditional probabilities, it is essentially the probability of event A given that event B has occured.

    In essence these other events are such that the clinician has probable cause to suspect autism and then administer the risk test.

    Under those circumstance you could argue that the prior probability that a child has ASD is (50/50 or higher) or whatever but is it is significantly better than 1 in 150.

    This is when this test has utility in helping the clinician make an early diagnosis.

    The test will misdiagnose 30% of all children. However the 70% that it diagnoses correctly will get critical behavioural and speech interventions that will help their long term diagnosis. Clinical evidence does support the sooner the better.

    We can start to discuss the 30% misdiagnosed but I don't know if I am suitably qualified to answer that. Maybe we can argue that the children that didn't have the condition now get some extra help and that isn't that bad but that needs to be weighed up against the heartache it causes the parents. All issues that I unfortunately cannot address.

    I think the biggest problem is for children that have the condition and the test diagnosis as not having the condition. They have missed a critical point in their development.

    Let me please re-iterate that this test has an enourmous amount of work before it finds itself into the clinic.

    (part II in the next post) I have run out of space on this

    ReplyDelete
    Replies
    1. You say "The incidence rate of ASD is 1:150." Incidence and autism don't go together. Autism, as far as I know, is not something that one can acquire. The prevalence or autism in the US was last measured at 1:88 and, in my opinion, is much higher than that.

      I don't use "spectrum" either, given the known heterogeneity of autism and related characteristics. And, as an autistic person, I find the term "disorder" to be highly insulting, as well as inaccurate.

      I also question some of the early intervention strategies I have seen. Yes, every kid (and adult) should get the help they need, but in many cases it appears to me that kids are being pressured to "develop" faster than their brains permit. The autistic brain develops more slowly, so the developmental "milestones" that are apt for neurotypical kids may not be naturally reached until a year or two later, and that my be okay.

      I can tell you from personal experience that there is a huge amount of trauma that comes from being told every day of your life that you are doing things wrong. Nature is often smarter than we are.

      Delete
  6. Part 2

    Other important points about the paper
    (1) It needs independant validation. (Don't believe, like any other so called discovery, until there has been other independant validation) We have tested it on a large cohort from two groups but it really needs testing across sites etc...
    (2) The test was developed and tested on a Central European population (this was the largest cohort). As clearly indicated in the paper, when this was tested on a Han Chinese population (not unexpectably) it achieved 56% accuracy. Not much better than tossing a coin. For such a test to have clinical utility it needs to be able to predict autism in the general population. Unfortunately genetics are very ethnic background specific. As we have mentioned in our paper, the SNP rates are markedly different between populations. We are working on test for other ethnic backgrounds.

    Our results indicate that cellular pathways are similar between the Han and the CEU, less so the Genes, and almost not at all the SNPs. This makes it impossible to build one classifier for all ethinic groups. Need a seperate one for each ethnic group and the problem remain for children of mixed ethnic backgrounds.

    (3) We haven't verified the specificity of the test. We have only tested ASD versus controls. As clearly indicated in the paper we do not know how specific the test developed is aginst differentiating against over conditions. Namely is it going to confuse ADHD, Childhood onset of Schnizophrenia or other neurodevelopmental or neuropshychiatric disease for ASD? We don't know.

    I guess why are we excited about the results?
    (1) It shows that Genetics play an important role in ASD. I guess we suspected as such given the high concordance amongst monozygotic twins, one of the highest amongst any neurodevelopmental or neuropsychiatric disease.
    (2) We have identified genes that both contribute and protect against ASD. This I think is particularly interesting because it has opened up new avenues for investigation. Our results indicate that SNPs in genes KCNMB4,GNAO1 and GRM5 can both increase and decrease the risk of ASD. What was interesting was SNPs in genes associated with innate immune system such as TNFa,NFkB, IL1 and NOS reduce the risk. I think that this is very interesting. This information allows us with some ways to move forward.

    Hope this helps put our work into perspective.
    Stan

    ReplyDelete
    Replies
    1. Hi Stan

      Thanks for the detailed response. I very much like the idea of a Bayesian approach to autism diagnosis. So far as I know, diagnosticians don't mathematically combine information from different sources in this way, but perhaps they should!

      It seems I may have slightly misunderstood your intentions with this test, although, to be fair you did say the following in the discussion of the paper:

      "A predictive classifier as described here may provide a tool for screening at birth or during infancy to provide an index of ‘at-risk status’, including probability estimates of ASD-likelihood."

      As you say, the first challenge will be to replicate the result. My initial worry was that the test might really be telling us which populations the individuals came from rather than whether they were autistic or not. The replication with the independent samples is pretty compelling evidence. But it's a shame there's no data available where controls and probands are drawn from the same population.

      You're right I think that the real challenge is going to be differential diagnosis. Obviously many of the genes that have been linked to autism have also been linked to schizophrenia, language impairment, epilepsy, and intellectual disability, so it's an open question whether you're picking up on something that's autism specific or not.

      Delete
  7. As an autistic person, I object to autism being described as a "risk" rather than the gift that it can be. It is a difference, like being left-handed. Both conditions are challenging because the world was not designed for people with those features, and there is some stigma attached.

    I also doubt that you can "become" autistic, any more than you can become blue-eyed. You are either born that way or you're not.

    I say these things not to detract from the very useful points being made.

    I do question, however, the 1% assumption (used for illustration only, I realize). The actual prevalence of autism is probably closer to 3%, and it could be much higher if other closely-related conditions are considered in the same "neuroexceptional" category; dyslexia, bipolar, schizophrenia, and the like. All of these things may be indistinguishable to any screening tests that current science is able to produce.

    The difference between autism and bipolar, for example, is a very subjective one, from what I can tell, and I wonder if these really are two different things.

    ReplyDelete
    Replies
    1. The difference between autism and bipolar disorder is very significant. In both case, they can have problem managing their emotions but autism show a different pattern of answer in IQ testing as compared with bipolar disorder.

      Delete
  8. While Skafidas makes a fair point that this test works best when the conditional probabilities have been raised, such as among children presenting already having met some of the diagnostic criteria. There remain two problems.
    1/.Skafidas participated in a media event that stipulated this was a screening test for Autism, which as Brock shows above, it is not. When you lend your name and image to the media, you are endorsing their report. If you disagree, you withdraw your name from the report. This usually stops the message. Skafidas has concurred with this perspective.

    2/. The change in conditional probability is only useful if the group presenting have a 35% or more probability of being on the Autistic Spectrum (see table below). This proportion seems unlikely, but should be assessed by evidence.
    For any given proportion of children presenting that truly have a Dx, the number erroneously diagnosed will be:
    1%, 42.43
    5%, 8.14
    10%, 3.86
    15%, 2.43
    20%, 1.71
    25%, 1.29
    30%, 1.00
    35%, 0.80
    40%, 0.64
    45%, 0.52
    50%, 0.43

    Mark Stokes

    ReplyDelete
  9. Hi – Will you please post a link to your important Blog at The Autism Community at vorts.com? Our members will really appreciate it.
    Members include: Those living with Autism, parents of children with autism, their families, friends, support groups, etc.
    It's easy to do, just cut and paste the link and it automatically links back to your website. You can also add Articles, News, Photos, and Videos if you like.
    Email me if you need any help or would like me to do it for you. I hope you consider sharing with us.
    Please feel free to share as often and as much as you like.
    The Autism Community: http://www.vorts.com/autism/
    Thanks,
    James Kaufman, Editor

    ReplyDelete
  10. thanks for sharing..

    ReplyDelete
  11. Dear Jon,

    Your post contains incorrect calculations, if I may explain below:

    Whilst Stan Skafidas' test isn't 100% accurate, and not diagnostic, it's not as useless as you imply it to be.

    You made the assumption that 293 were incorrectly diagnosed, with the assumption that the 30% (100-30%) is the %age of NTs that get screened as positive, which is extremely erroneous unless they explicitly states so.

    Stan Skafidas' genetic test had 70% SENSITIVITY ("Accuracy").

    The 30% you're referring to is actually the 30% of genuine ASD people who don't get screened as positive (ie FALSE NEGATIVES, 3 people).

    Just because it has 70% sensitivity doesn't mean that 30% of the NTs will be screened as positive (unless their false positive rate happens to be 30% as well).

    Assumption: 1000 people total.
    Assumption: 1% genuinely have ASD (ie 10 people)
    Assumption: 99% are NT (ie 990 people)
    Assumption: Test has 70% 'accuracy' (70% sensitivity)

    SENSITIVITY = Number of True Positives (ASD) / (Number of True Positives + False Negatives, ie total number of genuine ASD people).

    THEREFORE
    Of the 1000 people,

    7 ASD people are true positive.
    3 ASD people are false negative.

    The other 990 NTs you can't comment on, as their proportions of TRUE NEGATIVE, FALSE POSITIVE and hence SPECIFICITY haven't been mentioned (unless the study states so).

    Therefore it is wrong for you to say 297 have been falsely diagnosed, as you were assuming a SPECIFICITY of 70% (70% TRUE NEGATIVES and 30% FALSE POSITIVES).

    SENSITIVITY =/= TRUE NEGATIVES

    Therefore you can't comment on the SPECIFICITY purely based on the SENSITIVITY.

    Eg:
    It could've been 7 ASDs being screened 'positive' and 100 NTs being 'false positive'.

    This would also result in 70% Sensitivity, but 80.8% Specificity.

    Ken

    ReplyDelete
  12. Edit, change to :

    You made the assumption that 293 were incorrectly diagnosed, with the assumption that the 30% (***100-70%***) is the %age of NTs that get screened as positive, which is extremely erroneous unless they explicitly states so.

    ReplyDelete
  13. Add:

    Should've been more specific, by TRUE NEGATIVE and FALSE POSITIVE %age, I was talking as a proportion of all the ones that are negative (NTs only), and not the total population.

    ReplyDelete
    Replies
    1. Thanks Ken.

      You're right, of course, that positive and negative predictive accuracy aren't necessarily the same. But in this case they were (see Footnote 1).

      Delete
    2. Aha, that makes more sense now, what a coincidence... Apologies for not reading more clearly...

      Perhaps the media should've been more specific when mentioning "70% accuracy", because usually people in statistics attribute that to 1 value (sensitivity or specificity etc), and not to both.

      Delete
  14. Is it more accurate than the thousand ways to be autistic? Or the 60% concordance of practitioners? You know I'm kind of pissy. Ben was never diagnosed as having learning disabilities, nor given accommodations, until college. The best the k-12 could do for him was "behaviorally disordered" which is a misnomer for "hates the place that makes him feel stupid".

    ReplyDelete
  15. A letter detailing concerns about experimental design & validity:
    http://www.nature.com/mp/journal/vaop/ncurrent/full/mp201334a.html

    ReplyDelete
    Replies
    1. Thanks Anon! If anyone has access to Molecular Psychiatry, I'd love a copy...

      Delete