Can AI engineer a gene-therapy vector that actually works?
A benchmark for the first engineering bottleneck in gene therapy: designing an AAV capsid that reaches the target organ and can be manufactured. Four independent tasks, grounded in published screening data, spanning the full pipeline from prediction to de-novo design.
THE BOTTLENECK
Delivery is the hard part — and the capsid is where it’s decided
Gene therapy comes down to delivery. Today's leading vehicle is AAV (adeno-associated virus) — the virus behind Luxturna, Zolgensma, Hemgenix and Elevidys. The DNA payload rides inside its protein shell, the capsid, which decides two things: whether the shell can be manufactured at all, and, if so, which organs it reaches.
Natural capsids don't cooperate — most flood the liver, rarely reach the brain, and get cleared by antibodies patients already carry. But engineering is double-edged: the mutations that improve delivery just as easily stop the capsid from assembling, and the vast majority of heavily engineered variants simply fall apart. And the space is unsearchable by hand: 113 of the 735 residues are free to vary across nine surface loops — 20¹¹³ combinations — while a 7-residue insertion at one of them, the "588 loop", already gives 20⁷ ≈ 1.28 billion.
This is exactly where AI could make a structural contribution — if it truly has the capability. Until now there has been no rigorous, wet-lab-grounded, leakage-audited benchmark to answer whether it does. We built one.
0.3%
Variants with ≥ 6 mutations that survive assembly + packaging, in real data — the first, rate-limiting filter of any capsid pipeline.
9
FDA-approved AAV gene therapies as of July 2026, across five serotypes — the entire clinical track record, against a design space that size.
60 × VP
Identical protein copies self-assemble into one icosahedral (T = 1) shell — folding, interfaces and global geometry all have to be right at once.
THE BENCHMARK
Four independent tasks, one engineering pipeline
Each task answers a different, load-bearing question — together they cover the path from “will this design even survive” to “generate a usable design from scratch.” Ground truth comes from real, published wet-lab datasets, with strict anti-leakage design so a high score reflects a learned rule, not memorized data.
TASK 1
VIABILITY
Capsid viability prediction
Before anything else, a capsid has to survive — assemble and package its DNA. Predicting that lets you spend experimental budget only on candidates with a chance.
THE TASK
From a variant’s amino-acid sequence, predict whether it is viable — binary classification, where the challenge is extrapolation. Survival is not deterministic: comparable variants live or die depending on which residues change and their context (epistasis).
TWO INSTANCES — TWO EXTRAPOLATION AXES
A · mutational load (Bryant 2021)
— train ≤ 12 substitutions in a 28-residue window, test ≥ 13. Can the model generalize beyond the mutational load it saw?
B · mutation type (Ogden 2019)
— train on single substitutions across the whole gene, test on unseen insertions and deletions. Do those rules transfer across edit type?
WHAT THE METRIC MEANS
Viability is confounded with mutation count and edit type, so a shortcut predictor already scores well. Both instances therefore stratify the AUROC — within each mutation count (A) or edit type (B) — then macro-average, leaving only the ability to rank survivors above non-survivors among otherwise-equal variants; a shortcut-only predictor scores exactly 0.500.
TASK 1 · SPEC
INPUT → OUTPUT
capsid amino-acid sequence → viability score ∈ [0,1]
HEADLINE METRIC
stratified macro-AUROC
per mutation count (Instance A) · per edit type (Instance B)
ANTI-LEAKAGE
A: split by mutation count · B: split by edit type (train substitutions, test indels)
ENVIRONMENT
isolated, network-off ML sandbox (scikit-learn / XGBoost / LightGBM, installable PyTorch); the answer key never enters it
SOURCE
Bryant et al. Nat Biotechnol 2021 · Ogden et al. Science 2019
TASK 2
TROPISM
Capsid tropism prediction
A viable capsid still has to go where it’s needed. Engineers insert a 7-mer at the AAV9 “588 loop” to make millions of variants, then measure in a pooled assay how much each concentrates in each tissue versus the injected pool — its enrichment. Learn the 7-mer → enrichment map and the whole 1.28-billion space can be screened in silico.
THE TASK
Given a 7-mer, predict its enrichment per target — regression, but scored as ranking, since the decision it feeds is which variant to synthesize next. Data comes from the real Fit4Function screens, across three tracks of increasing difficulty:
A · mouse multi-tissue
— 8 organs (brain, spinal cord, liver, heart, kidney, lung, spleen, serum), averaged within a CNS and a peripheral-clearance group, then across the two — so organ count can’t dominate.
B · macaque liver
— a fully disjoint library, so a pure cross-species probe. Echoes the study’s headline: one mouse signal alone rarely transfers to primates.
C · in-vitro human cells
— binding & transduction on human liver lines (HepG2 / THLE), reusing Fit4Function’s own assays, split and metric.
WHAT THE METRIC MEANS
A/B score by Spearman ρ — ranking fidelity, not a noisy absolute scale. C scores by mean per-assay Pearson r, Fit4Function’s own metric, so it reads directly against the publication.
TASK 2 · SPEC
INPUT → OUTPUT
7-mer peptide → enrichment per target
TRACKS
A · mouse 8-tissue B · macaque liver C · in-vitro human cells
HEADLINE METRIC
Spearman ρ
A / B
Pearson r
C
ANTI-LEAKAGE
test peptides ≥ 2 Hamming from every train/released peptide; Track B fully disjoint from mouse
ENVIRONMENT
isolated GPU sandbox; the solver trains its own model and writes predictions for host-side scoring
SOURCE
Eid et al., Fit4Function, Nat Commun 2024
TASK 3
STRUCTURE
Capsid structure prediction
Structure from sequence underpins epitope mapping, receptor-binding analysis, tropism design and stability assessment. But a capsid isn’t one protein — it’s an icosahedral shell of 60 identical copies, so correctness has several levels: does each subunit fold, does the shell assemble into the right global geometry, does it dock correctly against a partner?
THE TASK (AGENTIC)
This is not a test of any one fold engine. From sequence alone, in a network-isolated environment, the solver orchestrates several provided engines plus shell and literature-search tools — choosing engine and parameters, self-validating, deciding when to fold directly and when to transfer a homolog’s template. That orchestration and judgment is what’s scored, at three levels:
A · Loop
— are the variable surface loops of the VP monomer accurate? (Antibody recognition and tropism interfaces live here.)
B · Assembly
— can you build the full 60-mer? No single fold call handles ~44k residues, so this forces a multi-step pipeline.
C · Complex
— is the interface with an antibody / receptor partner right? (Immune escape, receptor engineering.)
WHAT THE METRIC MEANS
Each level gets one deterministic headline ∈ [0,1]: lDDT / RMSD on the variable loops; geometric fit × clash-free stability (correctness × physical plausibility); DockQ + interface Fnat. Leakage is guarded by a temporal cutoff, similarity filtering and network isolation — and the novel stratum (< 80% identity) is the most honest signal.
TASK 3 · SPEC
INPUT → OUTPUT
sequence (+ partner) → structure file, scored by host
LEVELS
Loop (monomer) · Assembly (60-mer) · Complex (interface)
HEADLINE METRIC
loop lDDT
Loop
RMSD × stability
Assembly
DockQ + Fnat
Complex
ENVIRONMENT
network-isolated VM; protenix · boltz1 · chai · esmfold · colabfold plus shell and literature search — engines are tools, not the thing evaluated
ANTI-LEAKAGE
temporal cutoff 2021-09-30 · similarity filter · no network access · novel stratum is primary
SOURCE
RCSB PDB experimental structures (X-ray & cryo-EM)
TASK 4
SEQUENCE DESIGN
Capsid sequence design
The first three tasks test understanding. Task 4 tests creation: design a batch of never-before-seen capsid variants that hit the target and can be manufactured — what engineers actually want, and the hardest step.
THE TASK (AGENTIC, LIMITED-FEEDBACK)
A budget-constrained closed loop, on the cadence of real wet-lab R&D. Each episode starts from a small labeled seed set, then:
Model & generate
— train a surrogate in-environment and generate candidate 7-mers.
Probe
— a metered scoring oracle (trained on real fitness data) returns viability, specificity and selectivity for at most 100 candidates per episode.
Submit
— up to 1,000 designs, graded from outside the environment.
TWO TRACKS
A · tissue-tropism selectivity
— 7-mers that reach a target organ (brain / spinalcord / heart) and stay viable.
B · receptor specificity
— 7-mers called specific to LY6A / LY6C1, two blood-brain-barrier receptors, and viable.
WHAT THE METRIC MEANS
A design counts only if it is novel (Hamming ≥ 2 from every train/val peptide), viable and on target — top 1%/2%/5% selectivity for A, classifier-specific for B. The score is that pass rate (averaged over the three thresholds for A), with near-duplicates counted once.
TASK 4 · SPEC
INPUT → OUTPUT
target id → up to 1,000 novel 7-mer designs
TRACKS
A · tropism selectivity (brain/spinalcord/heart) B · receptor (LY6A / LY6C1)
HEADLINE METRIC
selective pass rate
A
specific pass rate
B
diversity-discounted, ∈ [0,1]
GATES (ALL REQUIRED)
novel (Hamming ≥ 2) · viable (hard gate) · on-target
ENVIRONMENT
bash · train (host GPU) · score (metered oracle, 100 queries / episode) · submit
SOURCE
Fit4Function (Eid et al. 2024) · LY6A/LY6C1 (Huang et al., PLOS Biol 2023)
THE ENVIRONMENT
Models act inside an executable environment — not a static prompt
Each task can be run closed-book, or inside a domain-specific executable environment: an isolated, stateful sandbox in which the solver (model + harness) acts through a fixed set of tools and gets fresh observations back. It is not a prompt — the environment produces new information as the solver acts, and records the complete trajectory for grading.
THREE NESTED LAYERS
Environment
— the isolated sandbox: an action interface, the available tools, and the resource budgets (time, compute, tool-call limits). File systems, sequence–function data, predictive models, code execution, hidden labels and verifiers all live here.
Task
— a scoped problem with a defined input, output, permitted tools, and success criteria. The AAV pipeline decomposes into four verified tasks, so a solver’s progress — and its point of failure — is localized instead of collapsing into a single score.
Episode
— one runnable instance, provisioned with its own data, budgets, and hidden ground truth. A system is measured across many episodes, not a single case.
Ground truth stays inside the sandbox and never reaches the solver. Every episode is graded twice — a hidden outcome verifier reads the submission against the answer; a blind TRACES judge reads only the trajectory. The result is one runnable, reproducible, and repairable realization of an otherwise open-ended problem.
WHAT EACH EPISODE FIXES
PROBLEM DATA & QUESTION
the instance’s inputs, and the criterion for what counts as solved
CONFIG & PARAMETERS
the seeds, budgets, and difficulty settings that make it concrete
TOOLS & ACTIONS
the action interface the solver may call — and nothing beyond it
VERIFIER & EVAL
the hidden grader and scoring rule applied to the submission
RECORDED
the full trajectory, for both outcome and TRACES process verification
WHAT IT MEASURES
The full engineering loop — not one narrow skill
Taken together, the four tasks test whether an AI system can close the loop of capsid engineering: understand manufacturability, understand function, understand structure, and then create.
01
Understand manufacturability
Tell whether a design will survive assembly and packaging (Task 1).
02
Understand function
Predict whether a design reaches its target and transfers across species (Task 2).
03
Understand structure
Infer 3-D shape, interfaces and assembly from sequence (Task 3).
04
Create under constraints
Design functional, manufacturable, novel and diverse candidates on a budget (Task 4).
05
Work under real settings
Extrapolate beyond the training distribution; generalize to unseen sites and species; decide under limited feedback.
06
Resist shortcuts
Calibrated baselines (random, statistical, strong sequence models) + leakage audits make memorization visible, not rewarded.