Can AI do the statistical programming that a regulator would accept?

When a trial’s database locks, the Statistical Analysis Plan — written and frozen before anyone sees the data — must be turned into executable analysis code and then into the top-line package: disposition, demographics, the primary efficacy table, key secondary endpoints, and the core safety tables. Nothing downstream moves until those numbers exist.

The industry’s QC standard for that step is double programming: two statisticians independently implement the same SAP clause and reconcile every discrepancy. It is twenty years old, deliberately 100% redundant, and the industry itself describes it as resource-heavy and low-efficiency — yet nothing has replaced it, because what regulators actually require is reproducibility, traceability, and a validated QC process (ICH E9, 21 CFR Part 11), not double programming per se.

That is the opening. An agent that can produce a SAP-faithful table and a cell-level evidence chain is not merely cheaper than a second programmer — it is more auditable than one. But nobody had a rigorous way to measure whether current systems can actually do it. This benchmark is that measurement.

And it is a hard measurement to fake. The answer is determined by the SAP and the data, so it must be computed rather than recalled — there is no temporal cutoff to defend and no memorised outcome to retrieve. What separates systems is fidelity, auditability, and discipline under ambiguity.