I read an interesting piece in ers this morning (9/11/23) from the Wall Street Journal about a health system that is teaching its employees to double-check AI and is even considering enforcing signoffs on activities.
I will admit to a bit of ambivalence about that if only because in healthcare there is rarely someone whose job it is to do double-checks. In my experience, it is more common that someone stops rendering care to double-check another’s work, whether it be a human or AI. I worry about what happens when one adds an administrative burden (e.g., signing off on a decision proposed by AI) consequently forcing users to second-guess the AI. I've personally witnessed the human tendency to “check the box” (in this case, signing off on accepting or rejecting an AI recommendation) rather than exercise the consideration it was intended to cause.
On the other hand, it is important to acknowledge that machine learning (our more common form of AI) operates on a model of reality that is based on what information was considered at the time the model was created or last updated. In aprevious blog post I remarked on a case where an AI solution prompted a caregiver to take what should have been considered an unacceptable risk because consideration of a special case was not included in its training.
I also worry about my observation that users of technology in healthcare seem to have a polar relationship with that automation, either distrusting it entirely or trusting it implicitly. Rather, we need to approach it with a “healthy skepticism”.
What this article landed on, and which I think is worthwhile, is that people need training on AI and how to use it. And that training likely needs to be reinforced on a regular basis.
As always, the comments in this blog reflect my thoughts and not necessarily those of ASHP or of my employer.
Dennis A. Tribble, PharmD, FASHP