Building a clinical validation harness for ML in healthcare

Clinical validation means proving, on data the model never saw, that it works for the patients and sites it will serve, overall and by subgroup, and that it keeps working after launch. Here is what FDA and NIST expect and how to turn that into a repeatable harness.

In short

  • Clinical validation of an AI model means testing it on sequestered data from different sites than development, against a documented reference standard, with a pre-specified analysis plan and subgroup results.
  • FDA's January 2025 draft guidance on AI-enabled device software functions spells out these expectations and adds performance monitoring after launch, while NIST's voluntary AI Risk Management Framework covers models that are not devices.
  • A validation harness turns this into code: hashed test sets, recorded model versions, automatic subgroup tables and the same run repeated on every model change.

Clinical validation of an AI model means showing, on data that was kept away from development, that the model performs well enough for its intended use in the population and sites where it will run, overall and in each relevant subgroup, and then showing that it keeps performing after deployment. A validation harness is the code and process that make this repeatable: a sequestered and versioned test set, a pre-specified analysis plan, subgroup reports, a record of which model version was tested, and monitoring once the model is live. FDA's January 2025 draft guidance on AI-enabled device software functions is the most detailed public description of what regulators look for, and its recommendations are useful even when your model is not a regulated device.

Who requires clinical validation of AI models in healthcare?

If your model is part of a medical device, FDA does. Its draft guidance “Artificial Intelligence-Enabled Device Software Functions: Lifecycle Management and Marketing Submission Recommendations,” issued in January 2025 and still marked not for implementation, describes what a marketing submission should contain: data management, model description, performance validation, monitoring and more. It builds on the Good Machine Learning Practice guiding principles that FDA, Health Canada and the UK's MHRA published in October 2021, which the International Medical Device Regulators Forum extended in a final document in January 2025.

Not every clinical model is a device. FDA's Clinical Decision Support Software guidance, in a version dated January 2026, explains which decision support functions fall outside the device definition under section 520(o)(1)(E) of the FD&C Act. Even then, the hospital buying your product will ask how you know it works. NIST's AI Risk Management Framework 1.0, released January 26, 2023 and intended for voluntary use, organizes that work into four functions: Govern, Map, Measure and Manage. In July 2024 NIST added a Generative AI Profile (NIST AI 600-1) for risks specific to generative models.

How should you split data so the validation is honest?

FDA's draft guidance is direct about independence. Reviewers want to know whether the test data are independent of the training data, for example sampled from completely different clinical sites, and whether they were sequestered from the model developers and the development stage. Data leakage between development and validation sets creates uncertainty about the model's true performance.

  • Describe how development data were split into training, tuning, tuning evaluation and any other subsets, and which development activity used each one.
  • Describe the controls that keep test data out of the development process.
  • Take test data from sites different from those used to develop the model, and justify why it is a robust external validation.
  • Avoid relying on a single site. FDA says a single site may be a useful starting point but is generally not appropriate for judging representativeness, and that at least three geographically diverse US clinical sites or health systems may be appropriate to clinically validate an AI-enabled device.
  • Be careful with data from outside the US. FDA notes that such data may not reflect US demographics, practice of medicine or standard of care, and that AI-enabled devices may need a higher proportion of US data in clinical validation.
  • Process test data only the way real-world inputs will be processed in the final product, and justify any cleaning, inclusion and exclusion criteria or missing-data handling against that.

In a harness, these requirements become storage and access rules. Store the test set with a content hash and access controls that keep it away from the people tuning the model. Record the model version used for every evaluation. FDA explicitly warns against adjusting a model opportunistically after looking at test results, and asks for the differences between the tested model version and the released one.

What is a reference standard, and how do you build one?

The reference standard is the best available representation of the truth for each case, such as the confirmed diagnosis or the expert-drawn contour. FDA asks how it was established, how much uncertainty it carries and how equivocal or missing cases were handled. When the reference standard comes from clinicians, FDA wants the grading protocol, the data the clinicians saw, how their readings were collected and adjudicated, the blinding protocol, the number of clinicians and their qualifications, and an assessment of intra- and inter-clinician variability.

For annotation, FDA recommends that each annotator work independently, without knowing the others' decisions, and asks for the annotators' expertise, their instructions, how disagreements were resolved and a plan for fixing incorrect labels. A harness should store labels with annotator IDs and timestamps so agreement can be recomputed whenever the label set changes.

Which metrics and subgroup analyses should the harness report?

Write the statistical analysis plan before you look at test results. FDA's draft guidance asks for primary endpoints, pre-specified success criteria that are clinically justified, the hypotheses, a sample size justification, the handling of multiplicity and a subgroup analysis plan.

Subgroups matter because, as FDA puts it, acceptable performance in some subgroups can mask lower performance in others when you only look at the total population. The subgroups depend on the intended use, but FDA says they should generally include patient sex, age, race and ethnicity, disease variables, clinical site, data acquisition equipment such as the camera brand, and where relevant the user's skill level and device configuration. If you make a performance claim about a specific subgroup, that analysis needs to be statistically powered. If you do not, each subgroup still needs enough patients for the result to mean something.

The right metrics depend on what the model does. FDA's examples: models that estimate a measurement may need a precision study of repeatability and reproducibility; models that monitor time-series data may need stability and change-tracking studies; prognostic decision support may need survival, calibration and discrimination analysis. FDA also asks for information on the uncertainty of model outputs, so report confidence intervals alongside point estimates.

Should you validate the model alone or the clinician using it?

Often both. One of the Good Machine Learning Practice principles is to focus on the performance of the human-AI team rather than the model in isolation when a human is in the loop. FDA's draft describes a spectrum. For some devices the main question is standalone performance: did the output match the expected output? For others it is whether the clinician working with the tool performs the same or better than without it. In medical imaging, reader studies that compare clinicians with and without the AI are typically the primary performance evaluation for tools that aid decisions. FDA also recommends usability evaluation, so you know users correctly receive, understand and act on the model's output.

What does a validation harness look like in practice?

A harness ties a model version, a test set and a report together. A run definition kept in version control might look like this:

run: sepsis-risk-v3.2-external
model:
  name: sepsis-risk
  version: 3.2.0
  artifact_sha256: <hash>
test_set:
  id: ext-2026-q3
  sha256: <hash>
  sites: [site-d, site-e, site-f]   # none used in development
  access: validation-team-only
reference_standard: adjudicated-chart-review-v2
analysis_plan: sap-2026-07.md       # written before unblinding
primary_endpoint: auroc
success_criteria:
  auroc_lower_ci: 0.80
calibration: [calibration_slope, brier]
subgroups: [sex, age_band, race, ethnicity, site, device_vendor]
outputs: [metrics.json, subgroup_table.csv, report.pdf]

The names and thresholds above are placeholders; your own come from the intended use and the analysis plan. Every harness needs the same parts:

  • Immutable inputs: hashed test sets and model artifacts, so a result can always be traced to exactly what produced it.
  • Separation of duties: the people who tune the model cannot read the test labels.
  • A pre-registered analysis plan, written before the run and referenced by it.
  • Automatic subgroup tables with sample counts next to every metric, so thin subgroups are visible.
  • Versioned reports stored with the model release, which also become the software version history a submission needs.
  • The same harness re-run on every model change, including a change of model provider or model version.

How do you keep validating after go-live?

FDA's draft guidance says the performance of deployed AI-enabled devices may change or degrade over time, for example through changes in patient populations, disease patterns or data drift, and that users may not notice when the device is part of a highly automated process. It recommends a performance monitoring plan covering how you will detect and assess performance changes and watch their causes:

  • Changes in patient demographics or disease prevalence.
  • Shifts in input data.
  • Input data corrupted by the pipeline, such as missing values, duplicate records and data type mismatches.
  • Changes in user behavior or user demographics.

The plan should also cover how you deploy updates and corrective actions in time and how you tell users about monitoring results. If you expect to retrain or adjust the model, FDA's final guidance on predetermined change control plans (PCCPs) lets a manufacturer describe planned modifications, the method for developing, validating and implementing them, and an assessment of their impact, all agreed in the original submission. Your harness is the natural place to run the validation steps that a PCCP promises.

Does this apply to LLM-based assistants?

The same logic applies, with different metrics. You still need a held-out set of real questions, a reference standard written by clinicians, a pre-agreed way to score answers, and subgroup views by specialty, site or patient type. NIST's Generative AI Profile is a starting point for the extra risks of generative models.

Our AI assistants cite their sources, so reviewers can check each answer against the document it came from, and they run on Claude via AWS Bedrock or Anthropic directly, with provider switching. Because the model behind an assistant can change, re-running the same validation set is how you know the new configuration is at least as good as the old one.

Onix Systems has built software since 2000 and has 370+ engineers. If you are planning clinical validation for a model, or need a harness around one that already exists, talk to one of our engineers. Our 5-day free technical audit ends with a written report on your data splits, reference standard and monitoring plan.