Twitter | Search | |
Vinay Prasad
The Google AI mammogram paper is FLAWED. Want to learn why? AND why cancer screening is the LAST thing you should pick FIRST to work on with AI? Read this This is educational, I promise
Reply Retweet Like More
Vinay Prasad Jan 2
Replying to @VPrasadMDMPH
Ok, so the entire google AI paper says can we get AI to look at mammograms to better predict who ends up having bx proven breast cancer. But that's the very mistake they don't see...
Reply Retweet Like
Vinay Prasad Jan 2
Replying to @VPrasadMDMPH
It turns out the goal of cancer screening is NOT to find biopsy proven cancer. That is a poor surrogate for what you want to find. Side note: they run the problem of: the biopsies that would have been taken if their algorithm existed do not exist. Will return to this...
Reply Retweet Like
Vinay Prasad Jan 2
Replying to @VPrasadMDMPH
Cancer screening can find one of several things 1 something that is not cancer 2 a cancer that isn't going to bother you in your natural life (harmless) 3 a cancer that was going to harm you, but we found it, and can cut it out, and now it isn't going to harm u (curable) AND...
Reply Retweet Like
Vinay Prasad Jan 2
Replying to @VPrasadMDMPH
4. A cancer that has already spread and is going to harm you even though we found it (spread-already) We want a cancer screening test that finds more curable cancer. We don't want to find more 'not cancers' (#1), but here is the tricky bit
Reply Retweet Like
Vinay Prasad Jan 2
Replying to @VPrasadMDMPH
we also do not want to find more harmless cancers (#2), AND we don't want to find more spread-already cancers (#4). We want to find more curable cancers, but less benign lesions, harmless cancers, and spread already/ damage done cancers!
Reply Retweet Like
Vinay Prasad Jan 2
Replying to @VPrasadMDMPH
What are the features that distinguish harmless cancers from curable cancers from spread already cancers on biopsy? Go on, you can cheat and ask a pathologist
Reply Retweet Like
Vinay Prasad Jan 2
Replying to @VPrasadMDMPH
THERE AREN'T ANY.... No one knows. So if you unleash AI on a problem and prove you are better at finding biopsy proven cancer you have no idea if you are changing the ratio of harmless to curable to spread-already And without knowing that you don't know you are helping
Reply Retweet Like
Vinay Prasad Jan 2
Replying to @cragcrest
You may paradoxically be making it worse! BTW, as I was reading this I see more evidence that is awesome...
Reply Retweet Like
Vinay Prasad Jan 2
Replying to @VPrasadMDMPH
Now back to this idea of biopsies that don't exist. The other big problem with AI of diagnostic imaging is retrospective validation does not account for the fact that prospective deployment may change the way data is collected
Reply Retweet Like
Vinay Prasad Jan 2
Replying to @VPrasadMDMPH
There may be biopsies that AI would have encouraged that do not exist, and we don't know the results of tests that were not done. Anyway, back to my bigger point. Cancer screening is the LAST thing you should ask AI to do FIRST
Reply Retweet Like
Vinay Prasad Jan 2
Replying to @VPrasadMDMPH
Cancer screening is too hard. It is not even clear that mammography improves net outcomes for healthy women who participate (i am talking OM people
Reply Retweet Like
Vinay Prasad Jan 2
Replying to @VPrasadMDMPH
Whatever gains we think (10-15% RRR on cause specific mortality in cochrane meta-a) are contingent on therapies at the time (bad, and new drugs erode screening gains), and the specific modalities used.
Reply Retweet Like
Vinay Prasad Jan 2
Replying to @VPrasadMDMPH
When you change the rules around how the study is interpreted, you cannot be sure that the net result is better EVER if you find more cancer and EVEN if you find less non-cancer. Because you don't know: harmless from curable from spread-already ratios in what you find
Reply Retweet Like
Vinay Prasad Jan 2
Replying to @VPrasadMDMPH
And you will not know that unless you pony up and conduct a 15 year multicenter RCT. There are so many better diagnostic tests with short term mortality outcomes that AI should be applied to FIRST. Don't make it harder than you need too.
Reply Retweet Like
Vinay Prasad Jan 2
Replying to @VPrasadMDMPH
See also last season of Silicon Valley.
Reply Retweet Like
Vinay Prasad Jan 2
Replying to @VPrasadMDMPH
In response to several comments that see this as not improving outcomes but labor saving: There is no way on earth that if you deploy this prospectively you will only lower #1, and keep 2-4 perfectly identical That is an artifact of retrospective studies
Reply Retweet Like
Vinay Prasad Jan 2
Replying to @VPrasadMDMPH
You will do something to #2-4, and that will have health effects. Thus it will not simply be a labor-shifting algorithm
Reply Retweet Like
Vinay Prasad Jan 3
Replying to @VPrasadMDMPH
This was more popular than I expected, so will make a plug for my next book coming in April 2020 Everything you need to know about cancer medicine to understand issues like this in < 300 pages
Reply Retweet Like