Validated Series: AI, Assurance & the Operational Reality of Safer Digital Health - Part 3

In the first article in this series, I wrote that AI in healthcare needs more than governance. Once AI is embedded into clinical software, governance has to become operational: structured testing, evidence capture, issue tracking, retesting and approval in the workflows where the technology will actually be used.

In the second article, I looked at the digital assurance gap AI is about to expose. A recent JMIR study found that documented assurance for digital health technologies is already inconsistent, even before AI becomes more deeply embedded into everyday clinical systems.

So where does that leave us?

For me, it points to a practical conclusion: AI validation has to happen where care happens. Not only in a vendor's development environment. Not only in a regulatory submission. Not only in a governance committee. And not only against generalized datasets that may not reflect the patients, staffing models, workflows, thresholds and operational constraints of a specific healthcare organization.

AI has to be validated locally, because that is where its promise becomes real and where its risks become visible.

We should want more AI in healthcare

Let's be clear: this is not an anti-AI argument. AI has enormous potential in healthcare, and we should want more of the right kinds of AI, used in the right ways, with the right controls around it.

AI can reduce documentation burden, help clinicians find relevant information faster, support earlier intervention, improve throughput and identify patterns that might otherwise be missed. For healthcare teams that are already stretched thin, those benefits matter. If AI can give time back to clinicians, reduce repetitive work or help teams see risk earlier, that is worth pursuing.

But wanting more AI does not mean pretending deployment is simple. In fact, the more AI becomes embedded into EHRs, EPRs, LIMS, middleware, diagnostic systems, patient portals and clinical documentation tools, the more important validation becomes. More AI means more software change. More software change means more workflows affected. And more workflows affected means more local testing, not less.

That can be a good thing if healthcare organizations have a practical way to manage it. It becomes a problem when validation still depends on spreadsheets, shared folders, screenshots, email threads and the institutional memory of whoever happened to be involved in the project.

The same AI feature is not the same deployment everywhere

Earlier this year, HHS published a Request for Information asking industry leaders, healthcare organizations and technology companies to weigh in on how AI adoption should be accelerated and supported in clinical care. The responses covered familiar themes: ROI, infrastructure, workforce readiness, trust, regulation, safety and the practical barriers that still sit between promising AI tools and confident use at scale.

What I found most useful was not the broad agreement that AI has value. Most of us already believe that. The more important question is how healthcare organizations are supposed to move from encouraging AI adoption to actually proving that AI-enabled workflows are safe, effective and manageable in their own environments.

Epic's response is especially relevant because it is clearly pro-AI, but also direct about the validation challenge. Epic notes that generative AI is probabilistic, which means performance has to be validated before deployment and continuously over time. It also points out that generalized pre-deployment datasets have limited predictive value for real-world performance. In other words, it is not enough to know that an AI feature performed well somewhere else, under someone else's conditions, against someone else's data.

The proposed answer is local validation: healthcare organizations evaluating how AI performs on their own data, embedded in their own workflows, both before deployment and through ongoing monitoring after go-live.

That makes sense because the same AI-enabled feature can carry different risks depending on where and how it is used. A predictive model may perform differently based on patient population, acuity mix, data quality, local workflows and staffing resources. A clinical documentation assistant may behave differently depending on specialty, note style, terminology and how clinicians review or accept suggested text. A triage tool may be safe in one setting but create too much operational burden in another.

Epic gives a very practical example: a predictive model that estimates the likelihood of imminent medical deterioration. The model produces a probability, but the healthcare organization still has to decide what threshold should trigger an alert. Set the threshold too low and frontline staff may face too many false positives. Set it too high and the organization may miss patients who needed earlier attention.

That is not just a model-performance question. It is an operational validation question.

A hospital with a dedicated remote monitoring team may be able to tolerate more false positives. A smaller hospital relying on busy frontline nurses may not. The model may be the same, but the risk changes because the workflow changes.

That is exactly why AI validation has to happen where care happens.

Local validation is not only about the model

When people talk about AI validation, they often jump straight to the model: accuracy, sensitivity, specificity, bias, drift and performance against a dataset. Those things matter. But in healthcare, the model is only part of the story. The workflow around the model matters just as much.

Who sees the AI output? Where does it appear? Is it clearly labeled as AI-supported? What action is the user expected to take? Can the user override it? Is the override documented? What happens when the output is wrong? How are edge cases handled? Who reviews failures? What is retested after a fix, configuration change or model update?

Those questions are not abstract. They determine whether AI becomes a useful support tool or a hidden source of risk.

A model can be technically impressive and still fail operationally if the workflow around it is poorly understood. A summary can look polished and still omit something clinically important. An alert can be accurate and still create fatigue. A recommendation can be reasonable and still be over-trusted by a busy user under pressure.

This is why structured UAT and validation matter so much. They force the organization to test not only whether the software works, but whether the AI-enabled workflow is safe, understandable and controllable in the real environment where it will be used.

Ongoing monitoring still needs operational discipline

Another important point from Epic's response is that AI performance must be monitored continuously over time. That is especially true for probabilistic systems, where performance can shift as data, workflows, users and patient populations change.

Judy Faulkner recently made a related point when she warned that AI can be "gamed" and that healthcare needs to monitor how it evolves. I do not read that as anti-AI. Quite the opposite. The same discussion highlighted major opportunities for AI, including disease prediction, rare-patient matching and helping clinicians compare diagnostic decisions against large sets of similar patient data.

That is the right balance.

AI can be powerful, and AI has to be watched. Both things can be true.

But "watching" AI cannot be limited to a dashboard somewhere. Monitoring has to connect back to operational controls. When performance changes, the organization needs to know what changed, whether workflows are affected, whether users need retraining, whether thresholds need adjustment and whether validation needs to be repeated.

That is where many healthcare organizations may struggle. Ongoing monitoring may produce signals, but operational validation turns those signals into action. If the model drifts, what gets retested? If a prompt changes, who reviews the output? If the vendor releases a new version, which workflows are affected? If users develop workarounds, how are those captured? If a failure is identified, how is it documented, assigned, corrected and retested?

These questions are not glamorous, but they are the questions that make AI safer to use at scale.

Where the safety case meets the test case

This is the natural conclusion of the series.

Part 1 asked how AI governance becomes operational. Part 2 showed why the assurance gap is real. Part 3 is the practical answer: validate locally, document clearly and make the process repeatable.

A safety case needs evidence. That evidence does not appear magically at the end of a project. It comes from the work itself: test scenarios, expected results, actual results, screenshots, issues, mitigations, retesting, approvals and final documentation.

That is where the safety case meets the test case.

For AI-enabled healthcare software, this connection becomes even more important. It is not enough to say that a feature was reviewed, approved or enabled. Organizations need to show how it was tested in their environment, with their users, their workflows, their data, their configuration and their operational constraints.

They need to know what was tested, what failed, what changed, what was retested and who approved it. And they need to be able to do it again when the next update arrives.

That last part matters. AI validation is not a one-time project. It is a recurring operational function.

More AI means more validation

Healthcare organizations are not going to receive AI as one big transformation event. They are going to receive it in pieces: a documentation assistant here, a prioritization model there, a new decision-support feature in an existing system, a workflow recommendation in an EHR or a new AI capability inside a laboratory, imaging or operational platform.

Some of these changes will be obvious. Others will feel like ordinary software updates.

That is why validation has to become easier to repeat. If every AI-enabled change requires teams to start from scratch, organizations will either slow adoption or cut corners. Neither outcome is good.

The better path is structured, reusable validation. Test assets that can be adapted. Evidence captured as work happens. Issues linked directly to failures. Retesting documented clearly. Approvals built into the process. Reports generated from the testing record, not assembled manually after the fact.

This is not about creating more bureaucracy. It is about making good validation easier to do well.

If we want more AI in healthcare, we need to make local validation practical, repeatable and manageable for the teams responsible for patient care.

The practical question

The future of AI in healthcare will not be decided only by the quality of the model. It will also be decided by whether healthcare organizations can deploy AI-enabled software safely in the messy, high-pressure, deeply local reality of care delivery.

So the practical question is not simply: Is this AI good?

The better question is: Has this AI-enabled workflow been validated where care actually happens?

That is where confidence will come from. Not from hype. Not from fear. Not from generic assurances. From structured testing, clear evidence, issue resolution, retesting and approval in the local workflows where clinicians, staff and patients will actually experience the technology.

AI can help healthcare. I believe it will. But the more AI we bring into healthcare, the more important validation becomes.