PulmCrit (EMCrit)
PulmCrit - July 5, 2016 - By Josh Farkas
"It was interesting to see a neurologist defending TPA for stroke on Emergency Medicine Literature of Note this week. As always, the discussion rapidly focused on the NINDS trial, and whether it requires replication.
Medicine continues to be plagued by poorly reproducible studies. The storyline is familiar. First, a very positive study is released in a major medical journal, with great fanfare. This leads to widespread changes in practice. Decades later, it becomes clear that the study was incorrect.
The reason for these mistakes may relate in part to misconceptions regarding the p-value. It is often assumed that a low p-value indicates that the study is reproducible (i.e. a “real” result, not a fluke). Unfortunately this isn’t true..."
Recently a new tool was developed to help understand the reproducibility of clinical studies: the fragility index. This post will analyze the NINDS trial from the perspective of its fragility index...
- The NINDS trial has a fragility index of three, meaning that if three additional patients in the control group had a good outcome then the study would have been negative.
- A low fragility index suggests that the study may be poorly reproducible.
- Baseline differences between groups and data lost upon follow-up could easily have caused a shift in the outcome of three or more patients. Since this imprecision exceeds the fragility index, it is plausible that the results of this study were falsely positive due to a statistical fluke."
Related:
- Demystifying the p-value (see #4, reproducibility) (PulmCrit)
- tPA debate between Dr. Swaminathan and Dr. Jagoda (EMCrit)
- Ischemic stroke treatment archive (REBEL EM)
- tPA policy with Dr. Jerome Hoffman (FOAMcast)
- Thrombolysis for stroke (theNNT)
- The secret of NINDS (Skeptics Guide to EM)