Thursday, August 27, 2020

Lies, damn lies and statistics

This was going to be a post about THC, but the recent dust-up over the FDA’s emergency approval of convalescent plasma to treat Covid-19 has encouraged me to deal with this subject as a more pressing topic. We are going to discuss the use of statistics in medicine. While I know that sounds dull, trust me, it is important to all of us, not just to doctors.

Picture this. 100 people come to my clinic complaining of fever and a cough. All hundred test positive for Covid-19. I give all of them a secret potion made with ground up dried newt, sunflower seeds and some CBD. A month later, 95 have recovered completely, 3 are still in hospital but recovering and two have died. I call a press conference and announce that my remedy has a 98% cure rate and should be widely used.

Do you accept my claim? I hope not! As Groucho Marx said when asked “How’s your wife:” “compared to what?” If you have followed this evolving story, the death rate among people with Covid-19 who have symptoms is estimated to be somewhere between 1 and 2%, with a huge variation dependent on age and ethnicity. Young Caucasians have a death rate well under 1% while octogenarians have a mortality well over 10%. Thus, to make any sense about my claimed “cure,” you must first ask for a breakdown of the ages and ethnicities of those I treated. If they were all white college students, chances are my remedy killed rather than cured. If they were all elderly Blacks, there may be something that warrants further study. Finally, no matter what the demographic breakdown, the most important question of all, is how my remedy compared to other available treatments.

This brings up the idea of the controlled clinical trial. There is a well-known aphorism in science: the plural of anecdote is not data. Medicine is full of “accepted” treatments that were proven worthless, and the fact that a patient improved after a treatment does not always mean they recovered because of the treatment. They may have recovered despite the treatment, which actually made some patients worse, or would have recovered with no treatment.

The current gold standard in deciding whether one treatment is better than another is the controlled trial. A large group of patients are randomly given treatment A or B; neither the patients nor the doctors know which they are given. After an appropriate amount of time, the pre-specified outcome is compared between the two groups. The outcome chosen is crucial: ideally, it is both important and clear. I always look first at death rate – whether one is alive is obviously important, and it is also very clear; you don’t need a committee to decide if a patient is alive (as you do in many reported outcomes).

When the trial is reported, the researchers will describe the difference and will usually indicate whether it is “statistically significant,” using a P value. This is simply the odds that the outcome was due to a real difference between the treatments or simply by chance. If you flip a coin and it comes up heads three times in a row, this does not mean the coin is unbalanced. Every time you flip a balanced coin, there is a 50% chance it will be heads, so getting heads three times in a row is not surprising. If you get heads 20 times in a row, you should be suspicious that there is something unusual about the coin. Hence, when a study reports a difference, they indicate the likelihood the account was due to chance. A “P less than .05” simply means that there is less than a 5% chance the difference was due to chance. Note that this is not a guarantee the results were valid.

Also important, and particularly relevant to Covid treatments these days, is whether the results are presented as relative or absolute differences. Drugs companies, not surprisingly, tend to emphasize relative differences, which are usually larger. Let’s say that 40% of patients with a very nasty disease are dead in year without treatment, while with treatment A, 25% die and with treatment B, 22% die. The honest way of presenting this would be to say that 3 out 100 more patients lived with B than A. A marketer would rather say that the death rate was reduced by 12% (22 compared to 25).

The Mayo Covid study had several issues limiting its value for making life-and-death decisions. Most important, the study was observational, not controlled. There was no group given an alternative (or only supportive care). No attempt was made to select who got serum with different amounts of antibody. They followed a large group of patients who were given plasma and compared those who received transfusions within three days of the diagnosis with those transfused four or more days after. They also compared those who received higher, medium or lower amounts of antibody in the plasma they happened to receive.

My focus was on the death rate at 30 days (a “hard” end-point – good). Those transfused earlier had a 21.6% death rate; those who got the plasma later had a 26.7% death rate. Thus, the absolute difference was 5% - possibly important if verified by better studies, but not the “35% reduced death rate” put out to the media. The latter figure came from comparing death rates at 7 days between those who received very high dose of antibodies (8.9%) and those who received very low levels (13.7%), a difference that was less at 30 days. There was no way to prove the groups were the same.

Does convalescent plasma help patients with severe Covid-19 survive? I think the only honest conclusion one can reach is “Maybe.” It is biologically plausible. The observations reported are consistent with a possible benefit, but better designed trials are clearly needed before this can be considered of proven benefit.

What I do know is that the FDA, which is supposed to be our defense against allowing ineffective and/or dangerous medications to be marketed, has increasingly made decisions based on political pressure rather than science.

This goes hand in hand with the Trump administration’s directive to the CDC to change its testing guidelines to discourage testing of asymptomatic Covid contacts, a decision that is opposed by almost every expert in the field. Fewer tests may lead to fewer reported cases but will lead to wider spread and more deaths.

Prescription for Bankruptcy. Buy the book on Amazon


  1. Thank you, Ed! Reposting on my FB page.

  2. Thank you for stating clearly how data can be used properly or improperly. We need clarity and honesty during this time when the objectives for reporting have more to do with politics rather than health. Any political objective which puts more individuals at risk is immoral and unconscionable.

  3. Pleasant perusing, I love your substance. This is actually a phenomenal and enlightening post. Containing all information about damn lies. Hemp flower