I don’t know about you, but up until I wrote the blogs on More About AI and Real World Evidence , I had never heard of George Box. The latter contains a reference to a Wikipedia article that I wound up re-reading after a colleague of mine suggested there was a lot more to think about.
The phrase that caught my attention in those discussions, attributed to Box, was “All models are wrong, but some are useful”.
What turns out to be important for us, I think, are some of the examples that Box and others have cited about models we tend to treat as if they were factual representations of reality.
- PV = RT (where P = pressure, V = volume, T = temperature and R is a constant) is an equation most of us learned in Physics class as a statement of fact. Box points out that, to the extent that R is based on something called an “Ideal gas” and that no gas is ideal, and R for most gases is not known, so this is a model, albeit one that has shown itself to be generally useful.
- Subsequently, in the book Statistical Control by Monitoring and Adjustment, Box asserted “… there never was, or ever will be, an exactly normal distribution or an exact linear relationship. Nevertheless, enormous progress has been made entertaining such fictions and using them as approximations.”
To those of us who rely on statistical methodologies, this feels a bit like heresy. But I have seen places where the math was compelling, but the results were not. One that comes immediately to mind was a clinical study I reviewed (and rejected) that measured, among other things, serum Potassium levels and reported a statistically significant reduction of serum Potassium in a large patient population from 4.9 to 4.8 (P<0.001) claiming significant reduction in serum Potassium.
What’s wrong with that you might ask? Primarily, the problem is physical reality:
- Serum Potassium laboratory results are subject to a certain amount of error; the difference between the two values is 2%, and it seems unlikely that a clinical laboratory analysis is accurate to ±2%. By the way, do you know what the permitted margin of error is for any particular analysis in your clinical laboratory?
- Even if such a measurement is accurate, so small a difference in serum Potassium is unlikely in the extreme to have clinical significance.
- The difference was only “significant” because of the mathematical model. Clinically speaking, this application of a model was, to Box’s point, useless.
So why is this important? Because models, especially statistical models, are seductive; they can become surrogates for actual understanding that are useful often enough that we no longer question them. But, as a host of statisticians have described, life is messy, and defies exact replication in a model. As a result, we must always approach the results of such models with some healthy skepticism. And, to do that, we must understand enough about reality (in our cases, about human physiology, and pharmacology) to become suspicious when the model appears to have gone astray rather than immediately yielding to the model.
Our understanding of pathophysiology and pharmacology must therefore be grounded in long-understood basics of human physiology, which have not changed for centuries, and allow ourselves to question studies, irrespective of their alluring statistics, if they lead us to disbelieve those basic facts. Not disregard, but definitely question.
On the other hand, there are times when we must learn from the model that it is our understanding of reality that is faulty.
As a result, we cannot permit ourselves the easy checklist path of just believing any model, or just disregarding it. Rather, we must ask ourselves the hard questions and struggle for those hard answers where there is a question. And that is always a lot more work than we would like it to be.
What do you think?
As always, this blog contains opinions that are mine alone, and not necessarily those of ASHP or my employer, BD.
Dennis Tribble, PharmD, FASHP
Ormond Beach, FL