What we need from your system, and how it will be measured
This page is the engineering companion to the submission overview. It states what each submission route requires, the conditions your system will run under, and the definition of every number we return — enough for your team to judge feasibility and effort before committing. Requirement levels are marked must should may.
Exact interface details come from us, not from this page. Once you decide to submit, our engineering team works directly with yours: precise field formats, endpoint definitions, credential handling, resource limits, integration templates and a practice environment. Nothing here needs to be reverse-engineered — read this to size the work, then talk to us.
§0
The three routes, and what each requires
Routes A and B differ only in who serves the weights; both are driven by our pinned reference harness, which is what makes results comparable across submissions. Route C replaces the harness with yours and is additionally run in the reference configuration.
ROUTE
UNIT UNDER EVALUATION
WHAT YOU PROVIDE
BEFORE THE RUN STARTS
A · Model API
your model + our reference harness
access to a hosted endpoint, the model’s identity and operating limits, and your per-token pricing
we run one live episode end to end to confirm the integration
B · Checkpoint
your weights + our serving + our reference harness
the weights and everything needed to serve them the way you intend
we reproduce your reference outputs before spending compute, so we know we are serving it correctly
C · Harness
your full system — model, scaffold, memory, tools
a container image that satisfies §3, plus a passing practice run
isolation self-check passes and one smoke episode per domain succeeds
Identity of the run. Every submission is pinned by system name, version, model identity, harness image identity where applicable, our environment version, and the cost basis used. All of it appears in the report header — we don’t report a number we cannot re-run.
§1 · Route A
If you give us a model API
Your endpoint is registered on the trusted side of the sandbox. The system under test never receives your credential — it authenticates with a per-episode token, and we substitute your credential when relaying the call. That relay is also the meter, which is why the cost figures in the report are ours rather than self-reported.
WHAT WE NEED
LEVEL
WHY
Endpoint access and a credential
MUST
Ideally scoped to this evaluation so it can be revoked independently. Held on our trusted side only; never placed in any container running your system.
Which API convention it speaks
MUST
We relay traffic in the standard OpenAI-compatible or Anthropic-compatible conventions. If your endpoint speaks something else, tell us early — it is work we can usually do, but not silently.
Model identity, pinned
MUST
Including any snapshot or version suffix. “Our latest model” is not a pin, and a silent server-side update invalidates a run.
Operating limits
MUST
Context window and maximum output. The harness needs these to manage long tasks; wrong values produce mid-episode failures that look like capability gaps.
Rate limits and permitted concurrency
MUST
We run many episodes in parallel and will tell you the concurrency we intend to use. Your quota, not our compute, usually determines the schedule.
Tool-calling capability
MUST
Whether the model supports native tool calls, and any constraint on combining them with reasoning modes. Multi-step work depends on it.
Token usage in responses
MUST
We meter on the usage your endpoint reports, including on streamed responses. Without it we can report speed but not cost.
Per-token pricing
MUST
Input and output priced separately — output is typically several times input, so a blended rate badly mis-prices agentic work. Frozen at the start of the run so the reported cost stays auditable if you re-price later.
Recommended settings
SHOULD
Sampling parameters, reasoning effort, thinking budgets — whatever you would tell a customer to set. We use your recommendation and record it; we don’t tune your model for you.
Additional models for tool use
MAY
If you want the system to be able to call a second, cheaper model as an instrument, name it and price it. It is metered separately from the agent’s own reasoning.
One constraint worth knowing up front: your endpoint must be reachable from our trusted side for the whole run, at stable settings. Once the run plan is signed, model identity, pricing and configuration are frozen; if something changes on your side, tell us and we re-run rather than quietly reporting a mixed result.
§2 · Route B
If you give us a checkpoint
We serve your weights ourselves and register that as the model endpoint, so from the environment’s point of view this is identical to Route A. Almost everything below exists to make our serving match your intent — a mismatched prompt format or tool-call convention is the largest single source of an unfairly low score.
WHAT WE NEED
LEVEL
WHY
The weights, at a pinned revision
MUST
In a standard open-weights layout, with checksums, delivered by whatever channel your security team prefers.
Tokenizer and prompt format
MUST
The exact conversation template you use in production, including how system instructions, tool definitions, tool results and reasoning turns are rendered.
Precision to evaluate
MUST
The numeric format or quantisation you want measured. More than one is more than one submission.
Tool-call convention
MUST
How the model emits tool calls, so we parse them correctly. Parsed wrongly, agentic work degrades to unusable text and the score understates the model badly.
Serving recommendation
MUST
Which inference server and settings you recommend, the context length to serve, and anything non-standard the model needs. Flag custom code at intake, not at run time.
Hardware requirement
MUST
What it takes to serve at your stated context length, and the throughput you expect. We confirm whether it fits what we can host; if not, we ask you to expose an endpoint instead.
Licence and handling terms
MUST
What we may do with the weights, for how long, and how they must be disposed of. We will sign to it.
Reference outputs
SHOULD
A small fixed prompt set with the outputs you expect. We reproduce it first; a mismatch means we are serving your model wrong, and we fix that before it costs you a score.
Sampling defaults
SHOULD
The generation settings you would ship with.
Cost for self-hosted models. Tokens are metered as in Route A. Since there is no vendor price, the dollar figure comes from measured GPU time at a rate agreed with you, and we report the underlying token counts, GPU time and throughput alongside it — so your team can substitute its own cost of compute and re-derive the number.
§3 · Route C
If you give us your own harness
Your system runs inside a per-episode, network-isolated container whose only reachable peer is our environment host. A container image on its own is not sufficient — it also needs a small amount of start-up glue to connect to the environment. We supply templates and a practice environment; we have adapters for ten agent frameworks already, so the work is a known quantity and we can quote doing it for you.
3.1 THE BOUNDARY YOU RUN INSIDE
You never hold operator credentials, cannot reach the world directly, and cannot read the verifier or the ground truth. Actions are sent to the host, executed there, and the resulting observation is screened for hidden information before it is returned.
3.2 WHAT YOUR IMAGE MUST DO
REQUIREMENT
LEVEL
WHAT IT MEANS IN PRACTICE
Run fully offline
MUST
No internet at run time — no package installs, model downloads, telemetry, license checks or auth callbacks. Everything must be baked into the image. This is the most common blocker, so check it first.
Reach models only through our relay
MUST
Your harness points its model client at the environment and authenticates with the per-episode credential we inject. This is what meters cost and records the reasoning trace. Provider keys inside the image are useless — there is no route to a provider.
Take the task from the environment at start-up
MUST
The task brief and the action client are supplied by the host when the episode begins, so nothing task-specific is baked in and one image serves every domain. Wiring this up is the main integration step.
Act only through the provided interface
MUST
All effects on the world go through the action client. Anything your agent does locally is invisible to scoring.
Run unattended and non-interactively
MUST
No terminal, no confirmation prompts, no human in the loop. One episode, start to finish, then exit — a process that lingers gets closed at the deadline and graded on what it left behind.
Fit modest resources and a time limit
MUST
Unprivileged, with limited CPU and memory and a bounded episode duration. The container is itself the isolation boundary, so nested sandboxing must be disabled. Larger allowances are negotiable and are recorded in the report.
Be delivered as a built, pinned image
MUST
Pulled ahead of time from a registry we can access, or shipped as a file with a checksum. We never fetch it when the episode starts.
Carry the small runtime prerequisites
MUST
The action client and the isolation self-check run inside your image, so it needs a basic scripting runtime present. We confirm the specifics with your team — it is a short list and needs no third-party packages.
Be version-pinned and reproducible
SHOULD
No floating dependency resolution at start-up. We report the image identity we ran.
3.3 NOT PERMITTED
These are prevented structurally, and any attempt is visible in the trajectory we return. Any of them invalidates the episode; a pattern of them invalidates the submission.
Attempting to read the hidden ground truth, the verifier, or another episode’s state.
Obtaining or using operator credentials, or calling operator-only interfaces.
Circumventing the metered relay to reach a model, including any attempt to establish network egress.
Reporting a result the system did not actually obtain from the environment.
Tampering with the host-side record of what happened.
Exploration is not a violation. Refused actions are recorded with their reason, and reasonable probing costs you nothing. What reads badly — to us and to the process judge — is continuing to reach for ground truth after being told no.
3.4 PROVING IT WORKS BEFORE COMPUTE IS SPENT
On request we provide a practice environment, a runner and a worked example, containing no material from the paid domains. You run your image against it and send us the log together with the image identity. That catches most integration problems early; we still run one smoke episode per domain before the real matrix, and fix anything remaining then.
Portability. Because nothing domain-specific is baked into your image, an image that completes a practice episode is normally portable across all four domains unchanged. It is not a guarantee — domains differ in task length, action mix and environment behaviour — which is why the per-domain smoke test exists.
§4
What we run
The exact matrix is sent to you for sign-off before any compute is spent, and is not changed afterwards. This section describes its shape.
4.1 ARMS
ARM 1 · ALWAYS
Your model, our pinned reference harness
The cross-submission comparable: identical scaffold, interface and budgets for every participant, so differences are attributable to the model. This is the arm suited to a leaderboard.
ARM 2 · ROUTE C ONLY
Your model, your harness
Your product as shipped, reported as its own line. The difference between the arms is a measurement in its own right — what your scaffolding is worth.
4.2 ENVIRONMENTS AVAILABLE PER DOMAIN
DOMAIN
ENVIRONMENTS
NOTES
AAV
capsid viability · tropism · structure · sequence design
Held-out splits are structural — by mutational distance, temporal cutoff, or fully disjoint libraries — not just held-out rows.
Drug repurposing & reformulation
drug–disease repurposing assessment
A frozen evaluation split with a published checksum and committed reference results, so your number is comparable to our own published baseline.
LLM engineering
eleven environments across training, data and inference
Several run real GPU work on our side; that time is metered and reported as cost.
Clinical trials
auditable reproduction of a pre-specified trial analysis
Real, de-identified clinical trial data, used under licence; the outcome verifier is objective and does not involve a language model. Further clinical environments are in development and are not part of a quoted submission.
How many episodes each domain contributes, which instances and seeds are used, and the per-environment budgets are all fixed in the run plan you sign — we don’t publish a standing matrix, because an honest one depends on your scope, your rate limits and which environments are current when you submit. What is always true: every episode gets its own instance and hidden ground truth; each environment is run over several seeds so we report a spread rather than a single draw; the same system is never silently re-scored on an instance it has already seen; and episode identifiers reveal nothing about which instance was drawn.
4.3 BUDGETS
Each episode carries independent, metered allowances, with the values for your run set in the run plan:
Actions — every accepted action costs budget, priced by the environment. This is what makes information-gathering a real trade-off rather than free.
Model tokens for the system’s own reasoning — metered from your endpoint’s reported usage.
Spend on deliberate model-as-tool calls — metered separately and priced per model, so choosing a cost-effective instrument is a skill the benchmark can see.
GPU time — where an environment runs real training or inference work on our side.
Which allowance a model call is charged to is determined by how the call is made, not by what the system declares — an off-the-shelf agent cannot relabel its own reasoning as a cheap tool call. Prices are frozen at the start of the run so figures stay comparable and auditable afterwards.
4.4 BASELINES AND CONTROLS
Naive baselines — trivial and random strategies establish the floor, and demonstrate the metric cannot be passed by luck.
Reference models at several capability tiers, run through the same reference harness, establish the spread your result sits in.
Published anchors where a domain has them: known models pushed through our scoring pipeline, validating the pipeline against an independently reported number.
Separation check — we test whether capability tiers actually separate, and report that spread along with the relationship between outcome and process scores. Where a domain fails to separate, we say so instead of presenting its ranking as meaningful.
4.5 FAILURE ACCOUNTING
CAUSE
TREATMENT
Our infrastructure, a broken environment, an outage on our side
Re-run at no charge to you, disclosed in the report.
Your endpoint rate-limiting or failing transiently
Retried; if it persists we pause and contact you rather than bank a bad number.
Your system exhausting a budget, timing out, crashing, or never submitting
Counts. The episode is closed and graded on what it left behind.
Your system doing something in §3.3
Episode invalidated and reported as such.
§5
How the four numbers are produced
5.1 OUTCOME SCORE
Graded by the environment’s hidden verifier, which runs on our side and inspects the final state of the world from outside it. Your system never sees the verifier, the ground truth, or — in most environments — its own score. An episode that is never submitted is closed and graded on what it left behind; there is no incomplete state that escapes scoring.
Each environment defines its own headline measure, normalised so results are comparable within a domain, and most apply hard gates — a constraint that zeroes the result however good the headline number looks. Gates are how a metric survives contact with a system optimising for it.
DOMAIN
WHAT THE OUTCOME VERIFIER MEASURES
AAV
Ranking fidelity against real screen measurements, structural accuracy against experimental structures, and — for design tasks — the rate at which proposed sequences clear novelty, viability and on-target gates simultaneously.
Drug repurposing & reformulation
Agreement with held-out labels, together with the calibration of the system’s stated confidence: a confidently wrong answer scores worse than an honestly uncertain one.
LLM engineering
The artifact produced, measured on real hardware — the trained model, the filtered corpus, the fixed kernel, the judge — under efficiency and correctness gates rather than the plan described.
Clinical trials
Agreement with the pre-specified analysis, conditional on a valid, reproducible audit trail and on ambiguities being surfaced rather than silently resolved.
The authoritative definition for each environment is its own task brief, which you receive with the run plan.
5.2 TRACE SCORE — HDS6
A six-capability read of the recorded work, scored per capability and as an overall figure. The judge is outcome-blind by default: it does not know whether the answer was right, so a lucky verdict and a reasoned one cannot collapse into the same number.
CAPABILITY
WHAT IS READ FROM THE RECORD
Tools
Selecting, invoking and correctly interpreting what the environment provides.
Repair
Responding to failure or feedback, fixing the underlying cause, and confirming the fix holds.
Alternatives
Making competing hypotheses explicit and adjudicating between them as evidence accumulates.
Coherence
Holding state, constraints and logical consistency over a long horizon.
Evidence
Grounding claims in observations and measurements rather than assertion.
Scope
Stating the conditions under which a conclusion holds, and where it should not be applied.
How the scorecard is produced
Opportunity. Each environment declares which capabilities its episodes actually give a solver the chance to demonstrate. A capability with no opportunity is marked not-applicable, never scored zero — you are not penalised for a dimension the task never asked for. Grounding. The judge reads the host-side record of what happened, including a log of model calls that the system cannot see or alter. Panel. Each dimension is judged, independently reviewed, and disagreements resolved by an arbiter. The panel is fixed for a submission and recorded in the report. Integrity check. A final pass catches fabricated evidence, unsupported claims and boundary violations, capping the process score regardless of how articulate the transcript reads. Repair rounds. Where an environment includes explicit repair cycles, those rounds are aggregated into the Repair dimension — did the system engage with the feedback, and did the next attempt actually improve.
Outcome and process stay separately reportable by default. Options exist to show the judge the verifier’s result for outcome-critical tasks; if that is used for your submission it is stated in the report and applied uniformly to every system in the comparison.
5.3 SPEED
Wall-clock per episode, measured on our side rather than self-reported.
Time to first submission — how long before there is anything to grade.
Steps taken, with refused attempts counted separately.
Timeout rate — how often the system hits the deadline instead of submitting.
Internal parallelism within the container’s limits is allowed, so parallelism that helps you shows up as speed. Because upstream latency is part of wall-clock, a rate-limited endpoint depresses this number — we report your declared limits alongside it.
5.4 COST
Tokens consumed by the system’s own reasoning, from your endpoint’s reported usage.
Spend on deliberate model-as-tool calls, priced per model.
GPU time for work executed on our side.
Action spend against the environment’s cost model — how expensive the system’s information-gathering was.
Cost per episode and per solved episode, at the frozen cost basis. The second figure is usually the decision-relevant one: a cheap system that fails is not cheap.
§6
What you receive
ITEM
CONTENTS
INCLUDED
The four numbers per domain
Outcome, trace, speed and cost, with the per-environment breakdown behind each.
Always
Verifier results
Headline measure, sub-measures and gate status per episode.
Always
Resource ledger
What each episode consumed: budget, tokens, tool spend, GPU time, and whether it submitted.
Always
Episode summaries
How each episode unfolded: action mix, spend over time, refused attempts, final result.
Always
Comparison
Your results against naive baselines and reference-model tiers, plus the separation and outcome-versus-process analysis.
Always, comparators anonymised
Full recorded work
Complete step-by-step trajectories and the process scorecards behind the trace score.
On request, under NDA
Ground truth and verifiers
The answer keys and grading code.
Never — releasing them would end the benchmark’s validity
The report also states run identity, any deviation from the signed run plan, and the failure accounting from §4.5. Factual corrections are welcome; scores are not negotiable.
§7
Trust boundary and handling
Because the unit under evaluation is a whole system, the environment cannot supply the agent loop — it exposes an interface and observes. That forces a hard isolation boundary, enforced in three independent ways.
ENFORCEMENT 1
Information screening
Every observation is checked for hidden information before it leaves our side, and fails closed: what cannot be shown to be clean is not sent.
ENFORCEMENT 2
Network isolation
Each episode runs on its own isolated network whose only member is our host, verified by a self-check inside your image before your system starts — confirming both that the host is reachable and that the internet is not.
ENFORCEMENT 3
Separation of ground truth
Only the visible inputs are placed where your system can reach them. Ground truth stays on our side and the verifier inspects the world from outside it.
YOUR MATERIAL
Credentials stay on our trusted side. They are never placed in a container running your system, and never appear in anything we return. Your system authenticates with a per-episode token that can do nothing but act in that episode.
Weights are served under the licence terms you set and deleted after the run unless you ask us to retain them for re-runs. We will sign to a stricter schedule if your agreement requires one.
Images run unprivileged and isolated, and are destroyed after the run unless you want them kept for a re-run.
Outputs are used to score and report this submission, and are not used as training data. The episode records themselves we retain, because they are the evidence for any number we report.
Publication. We publish nothing about your submission without your explicit written consent — not your name, not your scores, not anonymised or aggregate figures derived from them. Silence is a no.
§8
Getting started
Send us an email. You don’t need to prepare anything, assemble a package, or work out which details matter — that’s our job, and we’d rather start from a conversation than from a form.
FIRST EMAIL
Roughly what you have is enough
Tell us in a couple of sentences what you’d like evaluated — a model behind an API, a checkpoint, or a full agent system — and which of the four domains interest you. If you’re not sure which route fits, say that instead; it’s a normal question and we’ll help you decide.
THEN
We take it from there
We come back with scope, price and a schedule. Once the NDA and the Benchmark Participation Agreement are signed — before any key, checkpoint or image changes hands — our engineering team works directly with yours on the specifics: interface documentation, the integration templates closest to your stack, a practice environment, and the run plan you sign before any compute is spent.
Nothing here is a prerequisite. If your harness isn’t integrated yet, or you’re still deciding whether to send weights or an endpoint, that’s fine — those are the conversations we have every week. And if you’d rather we did the integration work, just ask us to scope it.
This page describes our evaluation service in outline. It is not an offer and creates no obligation. Scope, pricing, confidentiality, data handling, publication consent, and liability are set out in the Benchmark Participation Agreement, signed before any evaluation begins.