Taal Levi
@taaltree

Apr 9  
Several companies have tests of varying quality, but Cellex makes a FDAapproved antibody test. The goal is to test for immunity due to prior exposure rather than an active infection. Cellex's test has a "sensitivity" of 93.8% and a "specificity" of 95.6%. Lets interpret this


Taal Levi
@taaltree

Apr 9  
The Cellex test appears to have higher rates than others. Sensitivity: Probability person tests positive if they are infected. Specificity: Probability person tests negative given that they are negative. 93.8% and 95.6% sound good, they mean that only 6.2% of + people test , and


Taal Levi
@taaltree

Apr 9  
only 4.4% of  people test +. But what does it mean when you test positive? Since the sensitivity is high, you'd assume that you're positive and immune to covid19. But sensitivity measures prob testing + given that you are +. We want prob of BEING + GIVEN that you TEST +


Taal Levi
@taaltree

Apr 9  
The conditionality is reversed. Probability you are + given that you test positive can be written as Pr(are+test+). Let's just derive Bayes theorem real quick in this context so you don't have to just trust me. Pr(are+test+) equals the sensitivity Pr(test+are+) multiplied by.. pic.twitter.com/npW0FJsO5s


Taal Levi
@taaltree

Apr 9  
the probability you are+ irrespective of how you test, which is the same as the proportion of positives in the population. Let's assume that in the USA this is ~1% or ~3.3 million people are really infected. The final term in the denominator is the probability you test positive.


Taal Levi
@taaltree

Apr 9  
The probability you test positive must be decomposed into the people who test positive and are positive plus the people who test positive and aren't. We have all the necessary numbers for this probability expression and get a denominator of 0.053. Plugging it all in to Bayes rule pic.twitter.com/WzmPe883hu


Taal Levi
@taaltree

Apr 9  
If we plug in sensitivity, Pr(test+are+), the probability a random person is positive, and the probability a random person tests positive, then a randomly tested person who TESTS positive, only has a 17.7% chance of actually BEING positive. More that 82% of people will be wrong! pic.twitter.com/9OKTT0wwuR


Taal Levi
@taaltree

Apr 9  
If you get a positive antibody test result and think you're immune and quit socially distancing, you are far more likely than not to make a big mistake. When millions of people make the same mistake rather that appreciate conditional probability and Bayes rule, many people die.


Taal Levi
@taaltree

Apr 9  
But it gets worse. If you never had symptoms but test positive and conclude you're one of the lucky people who had an asymptomatic infection, you're even more likely to make a big mistake. Of course, as a higher proportion of population becomes infected, this mistake diminishes.


Taal Levi
@taaltree

Apr 9  
The take home is that a high sensitivity, Pr(test+are+)=0.938, does NOT mean that Pr(are+test+) is high. 82 of 100 people who get a positive test will still be susceptible to the virus. Once 10% of population has been infected, this drops to 30%, but that is a LOT of people!


Taal Levi
@taaltree

Apr 9  
I foresee that someone will say that more than 1% of people have now been infected. At a prevalence of 5%, 47 of 100 people who test positive will still be wrong. I found an online tool that lets you explore this for various quantities.
vassarstats.net/clin2.html


Taal Levi
@taaltree

Apr 10  
Since this tweet is being spread around, some addendums and an attempt at communicating with a general audience. 1. Context is testing of random population. If you had severe symptoms in the past, the probability of false positive is lower because Pr(are+) is higher.


Taal Levi
@taaltree

Apr 10  
2. Prevalence varies locally, so testing random New Yorkers for antibodies will produce fewer errors than random Oregonians.
3. Point of this tweet was to teach some math and an important qualitative message about being careful with a single + test.


Taal Levi
@taaltree

Apr 10  
4. Now an attempt at explaining this idea for general audience. Say you test 1,000,000 random people and prevalence is low so only ~10,000 had been infected. Those 10,000 will be picked up with sensitivity 93.8%, so 9380 will get positive test. Only 4.4% of negative people test +


Taal Levi
@taaltree

Apr 10  
But that is 4.4%*1,000,000=44,000 false positives. 44,000/(44,000 + 9380) = 0.82. So in this example, 82% of the people with positive tests were never infected and are still at risk. There it is w/out math lesson. If you didn't have severe symptoms, a second + test really helps.


Taal Levi
@taaltree

Apr 10  
My bad, it's 4.4% of negatives. So 4.4%* (1,000,00010,000)=43560. 43560/(43560+9380)= 0.82. So still 82% but the math was off slightly. This is how you get in trouble with quick tweets while morning coffee hasn't sunk in!


Taal Levi
@taaltree

Apr 10  
5. Finally (maybe), RTPCR, which is used to test for current infections is much less susceptible to this issue because false positives are much more rare. Someone "PCRsplained" me about this (Our genetics lab does many thousands of PCRs every year and designs assays)


Taal Levi
@taaltree

Apr 10  
6. I lied, one more thing. This is not some great insight that I had or a fancy model. This is one of the first things you learn in Bayesian statistics. Every statistician knows this, but many MDs do not (at least that's what statisticians say). I just took the time to tweet it.


Taal Levi
@taaltree

Apr 10  
People much better at science communication are explaining this better and far more succinctly.
mobile.twitter.com/amykatherine/s…


Taal Levi
@taaltree

Apr 10  
And this visual summary is great. I like getting into the details since typically there are lots of examples without the math worked out, but somehow I've seen nothing about this false positive problem.
mobile.twitter.com/LCWheeler9000/…


Taal Levi
@taaltree

Apr 11  
7. This is an educational thread. I'm in no way stating that effective serology to test for prior exposure can't be done. For instance, people testing positive for HIV with an ELISA would then follow up with a Western Blot. I have seen no such proposal. Point is...think carefully


Taal Levi
@taaltree

Apr 12  
If you enjoy learning math in this context, here is an old thread about the trajectory we were on before distancing and could stll get on. Exponential is a good approximation only while prevalence is low and almost everyone is susceptible.
mobile.twitter.com/taaltree/statu…


Taal Levi
@taaltree

Apr 12  
And if you want to learn how basic epidemic models work...
mobile.twitter.com/taaltree/statu…


Taal Levi
@taaltree

Apr 12  
It appears WHO wants to restrict these tests to labs for exactly this reason
who.int/newsroom/comm…


Taal Levi
@taaltree

Apr 12  
My day job is teaching/research and I thought teaching on Twitter could be useful. Here is an older post about the OTHER kind of testing used for active infection.
mobile.twitter.com/taaltree/statu…


Taal Levi
@taaltree

Apr 15  
And....Yesterday Pence said the U.S. would create more than 20 million new antibody tests a month. There are going to be a lot of people with a false sense of security.


Taal Levi
@taaltree

Apr 16  
The FDA recently approved (emergency use) tests from Chembio and Ortho. Haven't found the specs yet. Abbott labs also has a new antibody test. Very possible one will have much higher specificity demonstrated with decent sample size. That'd be great
centerforhealthsecurity.org/resources/COVI…


Taal Levi
@taaltree

Apr 23  
NY doing a good job telling people how not to interpret even their much better test.
www1.nyc.gov/assets/doh/dow…

