Part 1: We Cannot Validate AI-Enabled Healthcare Software with Spreadsheets and Shared Folders

Dean Mawson, Founder and Clinical Director, DPM Digital Health Consultancy recently wrote a thoughtful blog post on AI in the NHS, making the point that governance, not hype, will determine its future.

His article is written from a UK perspective, but the issue he raises is not limited to the NHS. Every healthcare organization adopting AI-enabled software will eventually face the same practical question:

How do we prove this technology is safe to use in our real workflows?

That question matters whether the system is deployed in an NHS trust, a US health system, a private laboratory, a diagnostic network or a hospital group in any country where clinical care now depends on digital systems.

Because AI is not only arriving as a standalone product with a bold label on the front. It is being woven into the software healthcare teams already use: EHRs, EPRs, LIS/LIMS, middleware, diagnostic platforms, documentation tools, triage models, scheduling systems and clinical decision support. In many cases, users may not think of themselves as "using AI" at all. They may simply see a new summary, recommendation, alert, prioritization or suggested action inside a familiar workflow.

That makes the operational risk harder to see.

Healthcare teams do not just need to know that an AI feature exists. They need to understand where it appears in the workflow, what it is intended to do, what it should not be used for, how users are expected to respond, what happens when it is wrong and how those controls are tested before the feature becomes part of live care.

That is not only a governance question.

It is a validation question.

The gap between policy and practice

Most healthcare leaders now accept that AI needs oversight. Some countries have more formal guidance than others. Some organizations have clinical safety officers, AI committees or risk frameworks already taking shape. Others are still trying to understand where AI is entering their environment in the first place.

But no matter how mature the governance structure is, the operational questions remain the same once a software supplier releases a new AI-enabled feature:

• What changed?
• Which workflows are affected?
• What test scenarios are needed?
• Who needs to test them?
• What evidence must be captured?
• What counts as an acceptable result?
• What happens if the AI output is incomplete, biased, misleading or simply wrong?
• How will failures be documented, corrected and retested?
• Who has authority to approve the feature for use?
• And how will the organization prove later that this work was done?

Those questions will not be answered by an AI inventory alone. They will not be answered by a policy document sitting in a governance folder. And they certainly will not be answered by a spreadsheet tracker, a shared folder full of screenshots and a long email chain where critical decisions slowly disappear from view.

AI makes structured UAT more important, not less

There is a temptation to treat AI as if it belongs in a completely different category from ordinary software validation. In some ways, it does. AI introduces risks that are more difficult to see and harder to explain: automation bias, cognitive offloading, drift, population bias, reduced transparency and outputs that may sound confident even when they are wrong.

But in another way, AI brings us back to something very familiar. Healthcare software still has to work safely in the real world.

That means real users, real workflows, real patient scenarios, real exceptions, real handoffs and real operational pressures. It means testing not only whether the software technically functions, but whether the workflow around it is safe, understood and controlled.

If an AI-assisted tool summarizes clinical documentation, teams need to test whether it omits important information, introduces incorrect language or creates a misleading impression.

If an AI-supported triage model prioritizes patients, teams need to test edge cases, escalation pathways, override processes and whether users understand the limits of the recommendation.

If AI is embedded into a laboratory, diagnostic or EHR workflow, teams need to validate how that output moves through the system, how it influences decisions and how failures are detected before they reach patients.

That is operational validation. And it needs to be structured, repeatable and documented.

The spreadsheet problem just got bigger

Spreadsheets and shared folders have survived in healthcare validation because they are familiar. They are easy to start with, everyone knows how to open them and they give teams the illusion of control.

But they were already weak tools for complex software validation.

They separate test steps from evidence. They separate issues from screenshots. They make approvals hard to track. They create version control problems. They make it difficult to see progress in real time. And when the project is over, someone still has to pull everything together into a report that proves what happened.

With AI-enabled systems, those weaknesses become more serious.

It is one thing to manage a spreadsheet for a simple configuration change. It is another thing to use that same approach to prove that an AI-influenced workflow was tested against intended use, known hazards, clinical exceptions, user behavior risks and approval requirements.

AI does not eliminate the need for UAT. It raises the bar for it.

Healthcare organizations will need reusable test assets for AI-enabled workflows. They will need evidence captured as testing happens, not reconstructed later. They will need issue tracking that connects the failure, screenshot, discussion, resolution and retest. They will need approval records that show who reviewed the outcome and why it was considered acceptable.

And when a software vendor releases the next version, they will need to do it again without starting from scratch.

Governance only works if it reaches the work

Dean's article makes an important point: AI governance must move from concept to operational practice. I agree completely.

But operational practice is where healthcare teams often struggle, not because they do not care about safety, but because the work is detailed, resource-intensive and easy to underestimate. Clinical teams are already stretched. Digital teams are managing multiple projects. Safety and quality teams are being asked to assess increasingly complex systems. And the pace of software change is not slowing down.

That is why the operational layer matters.

Governance may define the expectations. Risk management may identify the hazards. Clinical, operational and quality leaders may determine what controls are required.

But testing is where those controls are proven.

Testing is where intended use becomes a set of scenarios. Testing is where workflow assumptions are challenged. Testing is where users reveal confusion, workarounds or overreliance. Testing is where issues surface early enough to be corrected. And testing is where an organization builds the evidence it may later need for safety, compliance, inspection or internal assurance.

In other words, the future of AI in healthcare will not be determined only by whether organizations write good governance policies.

It will also be determined by whether they can turn those policies into structured validation work that real teams can actually complete.

The practical question

As AI becomes more embedded in healthcare software, every organization will need to ask a very practical question:

Can we prove that this AI-enabled workflow was tested, reviewed and approved in the context where it will actually be used?

If the answer lives in a spreadsheet, a shared folder and someone's inbox, that may not be good enough.

Not for the clinical teams relying on the system.

Not for the safety, quality and compliance teams carrying responsibility.

Not for regulators, inspectors or internal review boards.

And certainly not for patients.

AI may be new, but the obligation is not.

Healthcare organisations still need to know what was tested, what failed, what was fixed, what was retested, what was approved and what evidence supports that decision.

That work needs structure.

It needs traceability.

It needs to be repeatable.

And it needs to move beyond spreadsheets and shared folders.

In the next issue of Validated, I'll look at a recent JMIR study on digital health technology compliance in the NHS and why its findings should concern healthcare organizations far beyond the UK.