Twitter | Search | |
Spooky Health Nerd
Ok, this has been bugging me since yesterday, so I thought I'd actually go through this new webpage that claims Hydroxychloroquine (HCQ) is effective for COVID-19 and review it Here goes nothing 1/n
Reply Retweet Like More
Spooky Health Nerd Aug 8
Replying to @GidMK
2/n So, first things first - what do the authors say they did? They describe the site as a "country-randomized controlled trial" looking at HCQ vs a control
Reply Retweet Like
Spooky Health Nerd Aug 8
Replying to @GidMK
3/n The findings, such as they are, say that the 'treatment group' - countries that supposedly used HCQ more - had fewer deaths/million than the 'control group'
Reply Retweet Like
Spooky Health Nerd Aug 8
Replying to @GidMK
4/n Now, this site claims that these countries had 79.1% fewer deaths, which, if true, would make HCQ the most effective medical intervention ever, literally the balm of the gods This seems somewhat unlikely, given that big RCTs have found no benefit whatsoever for the drug
Reply Retweet Like
Spooky Health Nerd Aug 8
Replying to @GidMK
5/n What did the authors actually do? Well, they use the words "randomized controlled trial", but the site actually reports NONE OF THESE THINGS The procedure was not randomized, there was no control, and this was not a trial 🤷‍♂️
Reply Retweet Like
Spooky Health Nerd Aug 8
Replying to @GidMK
6/n What the authors ACTUALLY did was retrospectively divide countries into 'HCQ' or 'no HCQ' and compare death rates of COVID-19/million between those groups This is what's known as an observational ecological trial
Reply Retweet Like
Spooky Health Nerd Aug 8
Replying to @GidMK
7/n Ecological trials are, as you may remember from my blog, prone to all sorts of biases They are extremely hard to do well, and often can tell you very little about whether a treatment is effective or not
Reply Retweet Like
Spooky Health Nerd Aug 8
Replying to @GidMK
8/n It is hard to know why the authors chose to use the terminology they did. It is entirely incorrect, but I'm not here to litigate intentions, only facts, and the fact is that this website is wrong Many, many times
Reply Retweet Like
Spooky Health Nerd Aug 8
Replying to @GidMK
9/n Moving on, the authors classified countries into either HCQ or no HCQ How did they choose these groups? Well, long story short, it appears almost entirely arbitrary
Reply Retweet Like
Spooky Health Nerd Aug 8
Replying to @GidMK
10/n Pretty much every single country was assessed in the same way - the authors cite a single (or maybe 2) news articles, and/or a tweet, to decide whether a country was in the intervention or control This is manifestly inadequate to actually define HCQ usage
Reply Retweet Like
Spooky Health Nerd Aug 8
Replying to @GidMK
11/n There is no attempt to actually assess HCQ usage, despite this being SOMETHING YOU COULD DO You could look at HCQ doses given/purchases made in countries by date, for example Instead, the authors reference tweets
Reply Retweet Like
Spooky Health Nerd Aug 8
Replying to @GidMK
12/n So right off the bat, any analysis is meaningless. We have no idea whatsoever whether people in the 'intervention' group ACTUALLY RECEIVED HCQ, making the calculations largely a waste of time But that's just the start of the errors here
Reply Retweet Like
Spooky Health Nerd Aug 8
Replying to @GidMK
13/n Next, we can look at the analysis itself Or, rather, we can't, because the authors don't describe what they did, only giving relative risks and p-values
Reply Retweet Like
Spooky Health Nerd Aug 8
Replying to @GidMK
14/n Without any description of the analysis whatsoever, we cannot trust a single number that is portrayed on this site, because the stats could be simply incorrect Impossible to know!
Reply Retweet Like
Spooky Health Nerd Aug 8
Replying to @GidMK
15/n The authors then report using "controls" to adjust the estimates, but these controls are again incredibly misleading For example, country-level obesity rates
Reply Retweet Like
Spooky Health Nerd Aug 8
Replying to @GidMK
16/n The obesity rates are referenced to the CIA world factbook Problem is, this data was last entered in 2016, and often is cited to sources up to a decade old So it's really very out of date!
Reply Retweet Like
Spooky Health Nerd Aug 8
Replying to @GidMK
17/n But forgetting that, this is a classic example of the ecological fallacy You CANNOT assume that country-level obesity rates apply to the people who got COVID-19 - if you don't check that this is true, whatever you produce is basically nonsensical
Reply Retweet Like
Spooky Health Nerd Aug 8
Replying to @GidMK
18/n On top of this, the outcome measure is terrible. There is no attempt to clarify whether the reported deaths from Our World In Data are correct for the country, simply the assumption that Algeria and France have comparable death reporting systems
Reply Retweet Like
Spooky Health Nerd Aug 8
Replying to @GidMK
19/n The ecological fallacy is at play in terms of death rates as well - deaths/million is a meaningless measure if you don't take infections into account!
Reply Retweet Like
Spooky Health Nerd Aug 8
Replying to @GidMK
20/n In short, the website fails in every possible way: - not a trial - entirely observational - countless errors - numerically meaningless - arbitrary decisions
Reply Retweet Like
Spooky Health Nerd Aug 8
Replying to @GidMK
21/n Again, I am not here to litigate people's intentions - I'll leave that to others - but I will say that this is remarkably similar to the trash science produced by tech bros in the first few weeks of March No understanding of the underlying issues at all
Reply Retweet Like
Spooky Health Nerd Aug 8
Replying to @GidMK
22/n Whatever your opinion on HCQ - whether you are pro or for - you cannot conclude anything of value from this website It is misleading and wrong
Reply Retweet Like
Spooky Health Nerd Aug 8
Replying to @GidMK
23/n The discussion section is also, it must be said, a sight to behold They defend their decision to use the term "country-randomized controlled trial" despite CONTRADICTING THEMSELVES (countries deciding is by definition NOT RANDOM)
Reply Retweet Like
Spooky Health Nerd Aug 8
Replying to @GidMK
24/n We already have a term for this kind of research - observational ecological study - so as I said, inventing a new term that is wrong is simply misleading. Just use the correct terminology if you're going to do something like this!
Reply Retweet Like
Spooky Health Nerd Aug 8
Replying to @GidMK
25/n If nothing else, the most basic additions that would make the website slightly less worthless would be - details of statistical analysis - measure of HCQ CONSUMPTION by country - deaths BY INFECTION as the outcome - control for other govt measures to prevent COVID-19
Reply Retweet Like
Spooky Health Nerd Aug 8
Replying to @GidMK
26/n Another point that's worth making - the authors say that HCQ is preventive of COVID-19 The methodology is not even vaguely close to what you'd need to know if that's true
Reply Retweet Like
Spooky Health Nerd Aug 8
Replying to @GidMK
27/n You'd want to see case numbers by day, along with every intervention (i.e. social distancing, school closures etc) and the day they were implemented for every country on the list Then, number of HCQ doses given by day AT AN ABSOLUTE MINIMUM
Reply Retweet Like
Spooky Health Nerd Aug 8
Replying to @GidMK
28/n Even if the authors make the suggested changes up until 25/n, they'd just end up with a meaningless correlation without measures like these, which are an enormous amount of work The thing about ecological studies is that the good ones take A LOT of time
Reply Retweet Like
Spooky Health Nerd Aug 9
Replying to @GidMK
29/n Oh, another thing The authors keep maintaining on Twitter that this study was "random" because patients didn't choose what treatment they got, countries did This is absolute nonsense
Reply Retweet Like
Spooky Health Nerd Aug 9
Replying to @GidMK
30/n Firstly, it is misdefining random If ANYONE chooses the treatment, then it's not random BY DEFINITION Random means no one chooses, it's that simple
Reply Retweet Like
Spooky Health Nerd Aug 9
Replying to @GidMK
31/n But also - PATIENTS DID CHOOSE This is where we get back to the ecological fallacy - it is absurd to suggest that individuals within countries didn't choose to take HCQ. Even countries that authorised it had adopters and non-adopters
Reply Retweet Like