A Federal Regulator Just Drew the Automation-Bias Line. It Runs Straight Through Your Worst Call.
Robert Grand · Battalion Chief who still runs calls
The FDA finalized its Clinical Decision Support Software guidance this year, with the final version landing March 11. On its face it’s the kind of document that lives in compliance departments and vendor legal reviews, dry language about which recommending software gets the lighter regulatory path and which doesn’t. Buried in it is a plain description of a human failure called automation bias, the tendency to over-rely on a machine’s suggestion. And then the guidance does something almost nobody in our service noticed. It names the single most dangerous place to lean on that kind of tool.
It pointed straight at the call you ran last night.
What the FDA Actually Put in Writing
Strip out the regulatory language and the finding is sharp. Automation bias, the FDA says, produces two kinds of error: errors of commission, where you follow the machine’s bad advice, and errors of omission, where you fail to act because the system never prompted you. Both are ways a trained professional gets quietly steered wrong by a screen.
Then comes the part that should stop every EMS leader cold. The guidance singles out emergency, time-sensitive settings as the highest-risk environment for this failure. It says software meant for critical, time-pressured decisions generally can’t qualify for the lighter regulatory path, because the clinician doesn’t have time to independently review the basis of the recommendation (Automation Bias and Clinical Practice: FDA Makes Incremental Updates to CDS Software Guidance (https://www.jdsupra.com/legalnews/automation-bias-and-clinical-practice-3711489/)). A federal agency looked at the whole landscape of clinical decision support and decided the riskiest place to use it is the place where the clock is running and there’s no time to check the work (Clinical Decision Support Software, Final Guidance March 11, 2026 (https://www.fda.gov/media/191560/download)).
That’s our entire job description. The clock is always running. There’s never time to check the work.
Presence Is Not the Same as Oversight
For years “human in the loop” has been the phrase that makes a tool sound safe in a procurement meeting. A person sits between the machine and the outcome, so the machine can’t really hurt anyone. I’ve made the case before that this isn’t good enough, that human in the loop only counts when it’s an actual spec: a trained operator with real authority, the standing to override, and a record of what they decided. The FDA just handed us the hard evidence for why that distinction is the whole ballgame.
Because presence and oversight are not the same thing.
If there’s no time to check the reasoning, a person sitting next to the screen isn’t exercising judgment. They’re providing a signature. That is not an argument for taking the human out of the loop. It’s an argument that the loop has to be built, not assumed, because the unbuilt version, a tired operator and a running clock, collapses into a rubber stamp exactly when the stakes run highest. Oversight that happens after a calm review of the tool’s logic is real oversight. Oversight that happens in three seconds with a sick patient in front of you is a reflex, and reflexes follow the path of least resistance. The path of least resistance is to believe the screen. The argument used to be abstract, the kind of thing you’d debate at a conference. Now it’s a finding in a federal guidance document, and it landed squarely on the environment we work every shift.
The Monitor at 0300
Make it concrete, because abstractions don’t kill anybody and missed calls do. A cardiac arrest, and the monitor throws up a rhythm interpretation. The medic has seconds, a compressing partner, and a family watching from the doorway. An AI triage assist on the box nudges a transport-or-treat decision based on the patient’s presentation. A size-up support tool feeds the incident commander a recommended course of action while the first line is still stretching. In every one of those, the FDA’s finding applies word for word: that’s where over-reliance is most likely and least catchable, because nobody has time to interrogate the tool mid-call.
I’ve stood at that monitor. When the screen hands you a clean interpretation, the clock is running, and the family is in the doorway, the honest pull is to take what the machine says and move. I’d like to tell you I always built my own read first. I didn’t always. Twenty-four years in, my own judgment account is deep enough to catch most of what a screen gets wrong. The medic who pins on the patch next month doesn’t have that cushion yet, and the tool will be in front of them from the first shift.
That gap is the whole problem. The tool is most seductive exactly when the operator is least equipped to second-guess it.
Why This Is a Protocol Problem, Not a Purchase
Here’s where most of our service will instinctively file this wrong. This is not a procurement question. You don’t solve automation bias by buying a better tool, because the bias lives in the human under time pressure, not in the software. You solve it in the protocol.
That means medical directors and protocol committees deciding, on purpose and in writing, which AI outputs are advisory-only on the highest-stakes calls. It means building the workflow so the medic forms an independent read before the tool speaks, or installing a deliberate verify step where the cost of a wrong answer is highest. It means QA and documentation capturing which version of the tool ran and what it recommended, because the incident review is going to ask, and “the algorithm said so” is not an answer that survives a deposition. The decision about where the human stays genuinely in charge has to be made deliberately, before the tool ever ships to the field, not reconstructed afterward from memory and a vendor’s marketing.
None of that is a capital expense. All of it is governance, and governance is the part we keep deferring because nobody’s job description currently includes it.
The Two Things Standing in the Way
I won’t pretend this is easy, because the two real constraints are stubborn. The first is medical direction bandwidth. Most departments run a part-time medical director who’s already splitting time with a clinical job, and writing automation-bias awareness into protocol is one more fight on a desk that’s full. The second is vendor opacity. You can’t write a protocol around reasoning the vendor won’t show you, and plenty of these tools are black boxes by design.
Put those together and you get the honest bind: the person who’d have to govern the tool can’t see inside it, and barely has the hours to try. That’s not a reason to skip the work. It’s a reason to start it before a call forces the question, while it’s still a planning exercise instead of an after-action.
The Warning Came Addressed to Someone Else
This is the part worth sitting with. The strongest signal about AI risk in emergency medicine this year didn’t come from NREMT, or a state EMS office, or any body that answers to us. It came from a federal regulator describing clinical software in general, and it happened to draw the most important line in our profession before our own institutions picked up a pen.
The services that read someone else’s regulation as their own early warning get to design the safeguard on their own terms, calmly, in writing, before anything goes wrong. The ones that wait for an EMS-stamped version will be retrofitting it under deadline, after an incident, with a lawyer in the room. The warning already arrived. It just came addressed to someone else, and the only question left is whether we open the envelope.
Robert Grand is a Battalion Chief at Eugene Springfield Fire with 24 years of service. He writes Frontline Intelligence, a newsletter on operational doctrine, technology, and leadership in Fire & EMS.
Subscribe
From the floor, not the vendor booth. Two times a week.