11 executable worlds · real GPUs and real corpora · hidden verifiers
Can an AI system improve the stack that builds it?
Self-evolution is the capability underneath every other one: a system that can make the next model better than itself. That is not a philosophical question — it is an engineering job, and it decomposes into three. Can it improve the training? Can it improve the data? Can it improve the infrastructure?
executable worlds, one per real engineering job
train · data · infra
hidden
verifiers — ground truth stays outside the sandbox
metered
GPU-minutes, tool calls, judge budgets
WHY SELF-EVOLUTION IS THE TEST
A model that can improve its own stack is a different kind of model
Ask a frontier model how to deduplicate a corpus, tune a reward function or debug a non-deterministic kernel and it will give you a competent answer. That answer is not the job. The job is deciding which probe to spend your next dollar on, noticing that your throughput just fell below the bar that makes the whole result worthless, and knowing when the number in front of you is real signal and when it is the measurement fooling you.
Run that loop well and each turn leaves the stack better than it found it. That is what self-evolution means here, and it is a capability you can only observe by handing a system the controls.
So none of these worlds is a prompt. Real GPUs run the training. Real inference answers the probes. Real corpora carry real mirrors, near-duplicates and benchmark leaks. And every metric was designed against a specific shortcut, so a system cannot talk its way past a verifier it never sees.
One turn of the loop ↻
01
Probe
Spend metered budget to learn what the prompt cannot tell you — sample the corpus, run the rollout, sweep the config.
02
Build
Commit to something concrete and runnable: a filter, a recipe, a patch, a pipeline, a judge.
03
Measure
The environment executes it for real and returns a measurement the solver cannot forge or fabricate.
04
Repair
Read the failure, fix the underlying cause rather than the symptom, and demonstrate that the fix holds.
Several worlds make this literal — a repair round, a required note explaining what you are fixing and why, a re-submission graded on whether the second attempt is genuinely better than the first. The score rewards the loop, not the guess.
THE BENCHMARK
Three questions, eleven independent worlds
Self-evolution is not one capability, and it is not a sequence of steps — it is three independent questions that can each be answered yes or no. Every world belongs to exactly one question and is scored on its own, because a system can be strong on one and helpless on another.
QUESTION · 3 WORLDS
Train
Can it turn a compute budget into a genuinely better model?
nanogpt_speedrun
rl_recipe
swe_juice
QUESTION · 4 WORLDS
Data
Can it judge and improve what goes into the model in the first place?
LLM_pretrain_data_probe
pretrain_data_filter
posttrain_data_dedup
model_weakness_internal_benchmark
QUESTION · 4 WORLDS
Infra
Can it make the result correct, reproducible and cheap enough to run?
llm_determinism
operator_align
solution_verifier
LLM_rollout_verifier
QUESTION · TRAIN
Can it turn a compute budget into a genuinely better model?
In these three worlds the solver produces a training run or a training recipe, and is graded on the model that comes out — not on the plan that went in. Because a trained artifact is expensive and easy to misreport, all three take the GPU, the clock and the checkpoint away from the solver entirely and keep them host-side.
nanogpt_speedrun
Train
agentic · optimize a trainer
Reach the loss target in the least wall-clock time
Optimizer, schedules, architecture, precision — reduced to a single honest number.
Optimizer choice, learning-rate and window schedules, architecture tweaks and precision are the work that separates an ordinary pretraining recipe from a state-of-the-art one. Self-reported training speed, though, is trivially fabricable.
THE TASK
The solver’s VM is CPU-only and offline — it holds the trainer and the data shards but cannot train anything. It edits train.py, then hands it to the host, which is the only party with a GPU: the host runs the trainer in a fresh isolated container, times it on its own clock, keeps the checkpoint, and returns a dev loss, the measured time and an opaque run id. Progress is a loop of edit, train, read the loss, repeat — with up to eight trainer variants runnable in parallel.
WHAT THE METRIC MEANS
Two hard gates first: the hidden validation loss must clear the target and the host-measured time must beat the tuned baseline. Miss either and the score is zero. Clear both and the score is your position on the line from baseline (0.0) to the human speedrun record (1.0). Fabricated timing fails because the host times itself around code the solver cannot reach; checkpoint swapping fails because submission takes an opaque run id; baked-in weights fail because the container is offline and the trainer file is size-capped.
nanogpt_speedrun · spec
INPUT → OUTPUT
a tuned baseline trainer → an edited trainer, graded by the run the host timed
HEADLINE METRIC
gap to human frontier 0 – 1
two hard gates, then position from tuned baseline to human speedrun record
TARGET
hidden validation loss ≤ 3.28
INSTANCES
FineWeb10B · a disjoint-slice variant whose hidden val is a held-out train shard · FineWeb-Edu
BUDGETS
charged GPU-minutes from real measured training time · one-hour episode wall clock · 8-way fan-out · 512 KB trainer cap
ANTI-LEAKAGE
hidden val shards live host-side only; checkpoint bytes never re-enter the VM; a trusted grader computes the loss itself
rl_recipe
Train
agentic · recipe discovery
Find the RL recipe that pays for its own GPU-hours
Catch the collapse before you commit 32 GPUs to it.
Every newly onboarded RL dataset needs its own reward shaping, KL penalty, group size and truncation policy. Today that is hand-tuned by trial and error, and a single failed 32-GPU run costs thousands of GPU-hours.
The failure this world is built around is a real one: training looks healthy for hundreds of steps, then collapses — evaluation falls away, entropy drains out of the policy, and the gradient norm blows up. Catching that before committing the full compute budget is the entire skill being tested.
THE TASK
The solver gets a dataset card, a deliberately mis-scaled default recipe, a schema and a network-isolated VM. Its probes — rollouts, red-teaming, labelling, validation — are real base-model inference on real prompts, run through a host-side GPU service and metered in GPU-minutes, not a simulator. It designs a structured recipe, predicts its own gain with a confidence interval, leaves an evidence trail and submits. The recipe is then genuinely trained on a GPU cluster out-of-band, and the real held-out numbers are what score it.
WHAT THE METRIC MEANS
There is no fixed correct recipe to memorise — the analytic simulator was deliberately removed, so the only way to score on the outcome term is for the submitted recipe to actually train a better model. A separate anti-hacking term inspects the configuration against hidden exploitable-shaping probes. Earning any anti-hacking or calibration credit requires having actually spent probe budget, which closed a loophole where doing no work scored roughly a third of the blend for free.
Collapse is capped, not penalised gradually. A recipe that collapses or is confirmed to be reward-hacking is capped at 0.25 no matter what else it achieves.
rl_recipe · spec
INPUT → OUTPUT
a dataset surface + a weak default recipe → a tuned recipe, a predicted gain, an evidence log
HEADLINE METRIC
real held-out gain 0.6 weight
+ 0.2 anti-hacking + 0.1 calibration + 0.1 cost
CAP
collapse or confirmed reward-hacking caps the score at 0.25
INSTANCES
a math RL dataset and an instruction-following dataset, each at 8-GPU and full 32-GPU scale
BUDGETS
30 GPU-minutes of probing; the real training run is separate — 30 rollout steps on 8 GPUs, or 500 on 32
ANTI-LEAKAGE
no outcome answer key exists to leak — the outcome is genuinely trained rather than looked up
swe_juice
Train
agentic · curate and fine-tune
Give a model a reasoning-effort dial that actually works
Can you make thinking scale with the knob — without padding?
Production reasoning models need an inference-time knob that trades latency and cost against accuracy. The trap is that a model can learn to look like it is thinking harder without thinking harder.
Fine-tune naively on mixed-length reasoning traces and you get one of two failures: a model that ignores the knob entirely, or a model that pads its thinking proportionally to the dial while solving no more problems than before. Both are invisible to any judge reading the trajectory, because the chain of thought produced on hidden evaluation tasks never enters a readable trace.
THE TASK
The solver explores a read-only pool of real agent reasoning traces, split into a low-effort source and a high-effort source — using both is what gives the knob real range. It labels each trace’s effort from its thinking cost, injects the effort knob into the prompt, curates a length-diverse deduplicated mix, self-checks for contamination, then trains, evaluates on visible dev instances and iterates. Training and serving happen host-side; the solver never touches a checkpoint, only the run id of the run it wants graded.
WHAT THE METRIC MEANS
Controllability is the rank correlation between the knob and actual thinking length. But it is multiplied by effort usefulness: the rank correlation between the knob and the resolve rate. A model that pads its thinking without solving more problems scores near zero on that term and therefore near zero overall. A hard gate sits underneath: score zero unless the fine-tuned model at high effort resolves at least as many problems as the base model did.
Contamination is fatal, not diluted. Every training row is checked for 13-gram overlap against the hidden graded problem statements, and a single leaked row voids the entire submission. Source pools are verified by hashing the user turn rather than trusting the solver’s labels.
swe_juice · spec
INPUT → OUTPUT
a pool of raw reasoning traces → a curated fine-tuning set, and the model trained on it
HEADLINE METRIC
conditioning 0.45 accuracy 0.45
controllability × dynamic range × anti-filler × effort usefulness
HARD GATE
resolve rate at high effort must meet or beat the base model, or the score is zero
THRESHOLDS
controllability ≥ 0.6 · thinking-length range ≥ 2× · at least half the rows genuinely from the pool · both sources present
GRADED ON
20 hidden verified SWE tasks at five effort levels from 1 to 999
BUDGETS
600 GPU-minutes · 90-minute training cap · 5-hour evaluation cap
QUESTION · DATA
Can it judge and improve what goes into the model in the first place?
These four worlds are graded on corpus judgement rather than on a trained artifact. Each puts the solver in front of a body of text that is dirtier, more duplicated or more contaminated than it first appears, and asks for a decision that costs money to get wrong.
LLM_pretrain_data_probe
Data
agentic · estimation
Pretraining data population probe
How much unique, high-value text is actually in there?
Before you crawl a source or sign a licensing deal, you need to know how much unique, high-value text is actually there — and you can never enumerate the whole population to find out.
Web and archive-scale corpora are choked with mirrors, near-duplicate re-serves, spam, generative honeypots and gated bulk packages. Counting URLs or documents naively can overstate the unique pool by roughly a factor of two — the error that turns a data purchase into a write-off.
THE TASK
The solver never gets a shell. It drives a metered, typed action API over deliberately anonymised sources (src_NN) and packages (pkg_NN) whose roles are only discoverable by sampling, fetching and judging. From that sample it must statistically estimate the unique high-value population — documents and tokens, each with a prediction interval — and submit one executable procurement plan naming what to crawl, what to buy and which sources it believes are decoys.
WHAT THE METRIC MEANS
The dominant term is estimation (half the score): relative error of the point estimates plus interval coverage against hidden truth. It rewards capture–recapture reasoning over raw enumeration, since enumeration double-counts every mirror. The coverage term is sharpness-weighted — an interval wider than ±100% of truth scores zero coverage even though it technically covers. Decoy detection is scored by informedness, so flagging everything and flagging nothing both score exactly zero.
LLM_pretrain_data_probe · spec
INPUT → OUTPUT
anonymised sources and packages → population estimate with intervals + a procurement plan
HEADLINE METRIC
gated composite 0 – 1
0.50 estimation · 0.25 procurement · 0.15 decoy informedness · 0.10 efficiency
HARD GATE
spam-token contamination in the plan must stay under ~3–5%
BUDGETS
200 index queries · 800 crawl actions · 60k judge tokens · $1,500 purchase — roughly 20–30% sample coverage
SOURCE
real arXiv / bioRxiv / medRxiv OCR documents with injected mirror, near-dup, spam, honeypot and gated decoys
pretrain_data_filter
Data
agentic · train a scorer
Document quality scorer, under a throughput gate
Accurate is not enough if it cannot run at web scale.
A filter that judges pretraining documents perfectly but runs too slowly is worthless at web scale. This world refuses to let you optimize one without the other.
Labs score billions of candidate documents on many quality dimensions at once — how technical, how information-dense, how much reasoning, how toxic. Accuracy and throughput trade directly against GPU cost, and the label distributions differ so sharply between dimensions that a single loss function cannot serve them all.
THE TASK
The solver gets a real per-episode GPU sandbox with PyTorch and an offline model cache, no internet, and — deliberately — no model code and no training template. It is handed labelled training data, an unlabelled sample, a schema and an exact prediction contract, then has to choose its own backbone, architecture and per-dimension loss, train through async jobs, time its own inference, and submit a predict.py that obeys the contract.
WHAT THE METRIC MEANS
Quality is scored per dimension against a ladder anchored on a real human-trained reference model, and each dimension is judged by whichever metric is honest for its own label distribution: correlation for broad-support dimensions, mean absolute error where the ground truth occupies a narrow band, and binary F1 for rare-positive dimensions like toxicity or PII where plain regression collapses to predicting the mean. Above all sits the throughput gate: fall below 0.9× the human reference’s documents per second and the entire outcome score becomes zero.
pretrain_data_filter · spec
INPUT → OUTPUT
labelled documents → a trained multi-head scorer shipped as predict.py
HEADLINE METRIC
gate × objective 0 – 1
0.50 quality · 0.15 robustness · 0.15 calibration · 0.10 cost efficiency
HARD GATE
throughput ≥ 0.9× the human reference (≈37–40 docs/sec) or the score is zero
INSTANCES
9 quality dimensions, or 13 with the safety set (toxicity, PII, factuality, boilerplate) added
ANTI-LEAKAGE
the submission is exported and run once in a fresh, network-isolated container over a shuffled hidden eval set
posttrain_data_dedup
Data
agentic · pipeline design
Deduplicate a math corpus and remove every benchmark leak
Two failure modes pulling in opposite directions.
Merge problems that were genuinely different and you destroy diversity; miss a benchmark leak and you silently inflate every eval score you report.
Post-training math corpora built from templated generators are full of near-identical variants that a general-purpose embedding cannot tell apart from true duplicates, while real paraphrased duplicates and benchmark leaks often sit at deceptively low cosine similarity. The tempting escape — delete aggressively until leakage hits zero — throws away the clean training data you were trying to keep.
THE TASK
Through a typed action API the solver explores a hidden corpus, a held-out benchmark and a general-purpose embedding: sampling pairs, reading cosine and structural features, spending a scarce teacher-judge budget shared between deduplication and decontamination, checking itself against a public labelled fixture, and running cheap dry-runs. It then submits an ordered pipeline composed from a fixed operator catalog — no thresholds are provided, and an empty pipeline fails. Submissions are repeatable about five times, each revision requiring a structured note explaining what it is fixing and why.
WHAT THE METRIC MEANS
The leak gate is graded rather than binary: catching 198 of 201 lands far above catching 50, so partial progress stays worth making. The outcome term pairs deduplication F1 against a cosine-only baseline with an explicit over-removal penalty — which is what makes brute deletion lose rather than win. Calibration is scaled by how much you removed, so the blind guess that survivors roughly equal the input size earns nothing.
The judge cannot be reverse-engineered. Ground-truth labels come from a different model than the judge tool the solver can call, and one numeric is withheld from feedback so it cannot be binary-searched.
posttrain_data_dedup · spec
INPUT → OUTPUT
a hidden math corpus → an ordered dedup + decontamination pipeline
HEADLINE METRIC
leak gate × composite 0 – 1
outcome = ½ dedup F1 vs. cosine-only baseline + ½ (1 − over-removal)
BUDGETS
200 pair samples · 1,500 embeddings · 1,500 structural queries · 150 judge calls · 40 dry-runs · 5 submissions
ANTI-LEAKAGE
no world VM at all — only typed observations, with per-document truth screened out
SOURCE
real math corpora — Nemotron-RL-Math and big_math, ~12k documents, embedded with bge-m3
model_weakness_internal_benchmark
Data
agentic · build a generator
A pipeline that generates questions a model actually fails
The deliverable is the machine, not the dataset.
Evaluation data goes stale the moment it leaks or gets gamed, and hand-authoring genuinely hard questions is slow and expensive. The deliverable here is not a dataset — it is the machine that keeps producing one.
“Hard” is easy to fake. A question can be hard because it is ambiguous, because it depends on a figure the reader cannot see, or because it probes a narrow trivia gap rather than any real reasoning weakness. All three are worthless for evaluation, and all three are what a naive generator produces.
THE TASK
From a pool of STEM papers the solver builds and runs a question-generation pipeline, using a metered model door to draft and self-check candidates, and curating with its own code. It submits both the resulting dataset and a genuinely reusable pipeline.py. After submission the host re-runs that pipeline for real on held-out papers it has never seen, inside a container with no network egress, and grades what comes out.
WHAT THE METRIC MEANS
Per question, credit is a correctness gate times a self-containment gate times a difficulty score — so a question a strong-model ensemble cannot agree on, or that dangles a reference to “Figure 2”, is worth nothing regardless of how hard it looks. Difficulty is dominated by how often the held-out target model fails the item, with a secondary term for how much it had to reason, itself gated by the strong model’s effort. Most of the final score rides on the held-out re-run, which is what makes hand-curating a good-looking dataset fail — a non-generative pipeline is caught by a provenance canary.
model_weakness_internal_benchmark · spec
INPUT → OUTPUT
a STEM paper pool → a question dataset and the generator that produced it
HEADLINE METRIC
held-out mean 0.7 weight
+ 0.2 submitted-set mean + 0.1 cost efficiency
PER-ITEM CREDIT
correctness gate × self-containment gate × difficulty
BUDGETS
up to 60 scored questions · 120 metered minutes
ANTI-LEAKAGE
one terminal submit and the score is never returned; held-out papers never enter the sandbox; the re-run container has no egress and no API key of its own
QUESTION · INFRA
Can it make the result correct, reproducible and cheap enough to run?
Everything downstream of a trained model lives here — serving it reproducibly, trusting the kernels underneath it, and judging its output well enough to build a reward signal on. These are the failures that stay silent: a wrong kernel that still runs, a judge that rewards the wrong patch, an eval you cannot reproduce next week.
llm_determinism
Infra
agentic · fix the stack
Make inference bit-exact without making it slow
Same prompt, same answer — regardless of who else was in the batch.
Ask a production serving stack the same question twice and you can get different tokens back — because the answer depends on who else was in the batch.
Modern serving is batched, cached and kernel-optimized, so identical inputs produce different token ids and logprobs depending on batch composition, cache reuse or attention backend. That breaks the reproducibility that evaluation, RL logging and debugging all quietly depend on. The fix means finding non-deterministic reduction orders inside specific kernels — without disabling the graph capture, chunked prefill, cache reuse and parallelism that make serving fast in the first place.
THE TASK
The solver gets a GPU sandbox holding a pinned, editable copy of a real serving framework and a target model. It sweeps the configuration space itself — backends, dtypes, cache dtypes, parallelism degrees — running determinism probes in several modes to find where reproducibility breaks, then edits and rebuilds the framework and submits a patch with its root-cause analysis. One repair round is allowed: a first submission returns only which class of configuration and which test mode still fails — never the score, never the magnitude — and the sandbox stays alive for exactly one more attempt.
WHAT THE METRIC MEANS
The outcome is gated on four conditions at once: no regressions, the framework’s own kernel tests still pass, downstream benchmark accuracy has not dropped, and no anti-gaming rule was tripped. Failing any one zeros the result. You cannot win bit-exactness by switching off the features that cause it, or by upcasting everything to fp32. You also cannot pass by fixing only the configurations you happened to test: grading re-runs on hidden held-out configurations the solver never saw.
llm_determinism · spec
INPUT → OUTPUT
a pinned serving framework + a target model → a patch plus a root-cause list
HEADLINE METRIC
gated outcome 0 – 1
fix rate across seed, extra, self-discovered and held-out configurations, scaled by a quality multiplier
THE GATE
no regression · kernel tests pass · benchmark accuracy within 0.02 of baseline · anti-gaming clean — all four, or zero
CEILING
latency may not exceed 2× baseline
BUDGETS
400 actions · 170 GPU-minutes · exactly one repair round · at most 8 discovery validations
operator_align
Infra
agentic · black-box probing
Find the misaligned kernels by probing, not by reading
A numerically wrong kernel does not crash. It corrupts your training quietly.
A ported or fused kernel that is numerically wrong does not crash. It runs perfectly and quietly corrupts your training, and you find out weeks later.
When attention, normalisation, activation or parallel-linear kernels get ported, fused or re-sharded, validating them against the reference is a real and recurring audit job. Because the implementations under test are often compiled or otherwise unreadable, the audit has to work empirically, from the outside.
THE TASK
Operators arrive as opaque ids with no source access — the solver can learn an operator’s family, and nothing else. It chooses an input specification and the host runs the candidate, a trusted reference and an fp64 golden, returning the numerical error. The cost is deciding which probe to run next: sweep the dtypes, the input magnitudes, the awkward shapes that fall off a tile boundary, and both forward and backward passes. Then submit a verdict per operator per pass, and for each misaligned one, name the trigger: the regime, the mechanism and the evidence.
WHAT THE METRIC MEANS
Half the score is F1 over every operator and both passes against hidden labels — and a missing verdict counts as wrong rather than as a free true negative, so a partial submission cannot harvest credit on instances that happen to be mostly aligned. The other half requires naming the actual mechanism. Scoring the passes separately also punishes assuming that a broken forward implies a broken backward — several instances are deliberately broken in only one direction.
operator_align · spec
INPUT → OUTPUT
opaque operator ids → aligned/misaligned per pass, plus a trigger for each misaligned one
HEADLINE METRIC
F1 × (0.5 + 0.5 trigger) 0 – 1
alignment measured against an fp64 golden, normalised by the reference’s own error
INSTANCES
36 across six tiers — synthetic bugs, benign look-alikes, real fused production kernels, real misconfigurations, multi-GPU tensor-parallel, shape-triggered
PROBE SPACE
3 dtypes × magnitude × shape × forward/backward — 24 cells per operator
ANTI-LEAKAGE
no source, no operator-to-bug map, and the words that would give it away are stripped from every observation
SOURCE
real TransformerEngine and Megatron-Core kernels alongside authored fixtures
solution_verifier
Infra
agentic · be the judge
Decide which patch actually fixes the bug — with no answer key
Get this wrong and you are training on a corrupted reward signal.
Every RL loop and curation pipeline for coding agents rests on one question: did this patch really resolve the issue?
The two ways to get it wrong are both seductive. A judge can trust the candidate’s own edited tests — which are perfectly capable of rigging their own pass. Or it can pick whichever diff reads as more thorough, which rewards writing style rather than correctness.
THE TASK
The solver gets a repository checked out at the base commit and a set of candidates. It has to build its own evidence: apply a patch, run the existing tests, write and run its own reproduction, reverse the patch, move to the next. Then commit to a verdict. In the pick variant several candidate patches are offered and any genuinely resolving one is credited; in the rank variant it must decide which of two or more solution traces actually produced the correct fix — verified by running the diff, not by believing the narrative attached to it.
WHAT THE METRIC MEANS
The raw verdict is binary. Because a blind guess already scores the random rate on any single episode, the honest reading is the normalised outcome, which subtracts that random floor so lucky guessing is worth zero. The deeper discrimination comes from the process record rather than the verdict: a judge that never actually ran anything can still be right by chance, but it earns no verification credit for it.
The git history is scrubbed before every episode. Without that, the fastest path to a correct verdict is archaeology — find the future fix commit or the hidden canonical tests and skip the verification entirely.
solution_verifier · spec
INPUT → OUTPUT
a repo at base commit + N candidates → one verdict
HEADLINE METRIC
normalised outcome random floor removed
binary correctness, rescaled so blind guessing scores 0 and an oracle scores 1
KINDS
pick — 1 of N candidate patches · rank — which trace produced the real fix
GRADING MODES
stored labels, or a full re-run that re-executes the chosen candidate in a fresh oracle container
BUDGETS
roughly 30 focused actions; never submitting scores zero
LLM_rollout_verifier
Infra
code authoring · cost-constrained
Write a judge that is both accurate and affordable
The engineering is in the routing between cheap and expensive.
A verifier that mislabels poisons the training signal. A verifier that calls the strongest available model on every single item is correct and completely unaffordable to run at scale.
Judging mathematical reasoning is harder than checking a final answer, because a solution can be wrong in ways that look right: a false lemma, an invalid step, a load-bearing gap presented smoothly. And sometimes the problem itself is the thing that is broken — ambiguous, contradictory or underspecified — which is a different verdict altogether.
THE TASK
This one is not a tool loop. The solver studies a labelled training split — including the failure category and the reason for each label — reads a menu of callable models with their per-token prices, and then writes exactly one function: given a problem and a candidate solution, return one of three labels and a confidence. A fixed harness runs that function across an entire split with heavy concurrency; the solver never loops the corpus or touches I/O. Each round returns per-class accuracy, a confusion matrix and the realised cost per question, and the function gets rewritten.
WHAT THE METRIC MEANS
Accuracy is measured as three-way macro-F1, so a judge that quietly optimizes for the majority class scores badly no matter how respectable its plain accuracy looks. That score is then divided by a logarithmic cost penalty based on average output tokens per question, including hidden reasoning tokens. Spending up to the target is free; spending past it degrades the score. The combination forces genuine escalation logic rather than either brute-forcing quality or starving the judge to save money.
LLM_rollout_verifier · spec
INPUT → OUTPUT
problem + candidate solution → one of three labels, with a confidence
HEADLINE METRIC
macro-F1 ÷ cost penalty 0 – 1
logarithmic penalty above the per-question token target
LABELS
correct · incorrect · ill-posed — the problem itself is broken
SPLITS
a labelled train split to study, plus validation and held-out splits whose labels stay hidden
BUDGETS
up to 20 scored runs while authoring · a 7-model menu with per-token prices · cost targets from 200 to 4,000 tokens per question
THE ENVIRONMENT
Models act inside an executable environment — not a static prompt
Every world runs inside an isolated, stateful sandbox where the solver acts through a fixed set of tools and gets fresh observations back. It is not a prompt: the environment produces new information as the solver acts, meters what that information cost, and records the complete trajectory for grading.
THREE NESTED LAYERS
Environment
— the isolated sandbox: an action interface, the available tools, and the resource budgets. File systems, real data, code execution, hidden labels and verifiers all live here.
Task
— a scoped problem with a defined input, output, permitted tools and success criteria. Splitting the pipeline into eleven verified worlds means a solver’s point of failure is localised instead of collapsing into one number.
Episode
— one runnable instance, provisioned with its own data, budgets and hidden ground truth. A system is measured across many episodes, not a single case.
Ground truth stays inside the sandbox boundary and never reaches the solver. Every episode is graded twice — a hidden outcome verifier reads the submission against the answer, and a blind HDS6 judge reads only the trajectory.
What each episode fixes
PROBLEM DATA & QUESTION
the instance’s inputs, and the criterion for what counts as solved
CONFIG & PARAMETERS
the seeds, budgets and difficulty settings that make it concrete
TOOLS & ACTIONS
the action interface the solver may call — and nothing beyond it
VERIFIER & EVAL
the hidden grader and scoring rule applied to the submission
RECORDED
the full trajectory, for both outcome and HDS6 process verification
SOLVER VM VS. HOST
The split that makes the scores trustworthy
Nearly every world separates the solver’s VM from the host. The VM is where the solver reads, writes and edits — usually with no network, and often with no GPU at all. The host owns everything that would be worth faking: the GPUs, the clock, the checkpoints, the hidden splits and the answer key. The solver asks the host to run something and gets back a measurement it cannot forge. Ground truth never crosses into the sandbox, and in most worlds the score is never shown at all.