LLM Engineering Leaderboard

FULL RESULT MATRIX

All eleven boards, cell for cell

The complete harness × model matrix for the eleven LLM infrastructure and training-stack environments, copied from the result appendix with no rewriting. Rows are harnesses; columns are the six foundation models. Every environment keeps its own metric and its own table — nothing is merged and nothing is averaged across environments.

Claude Code only runs Opus-4.8 and Codex CLI only runs GPT-5.6-sol — by design, not a missing run. Row order is fixed to the appendix order and never sorted, so one harness can be read across environments.

TRACES is built and operated by Apodex, which also submits its own solvers. Apodex entries are marked.

LLM solver (harness × model)

Single-model harness (Claude Code / Codex CLI)

ApodexHarness (in-house)

published SOTA / method baseline (ref)

best in column

OUTCOME MARKERS

Refused

safety or content-policy refusal

Fault

harness bug, API, compatibility or quota fault

Gate not met

a hard gate was missed (loss, speed, throughput, no-regression) or training failed

Not submitted

nothing submitted, or not submitted to protocol

Timed out

budget exhausted; nothing submitted inside the limit

Note

a remark on the score, not a failure

Genuine

a real capability failure: the run completed and satisfied none of the required fixes

Harness artifact

no scorable submission ever reached the world

Grounding

stopped to ask the user instead of acting

Out of sweep

not in the sweep this metric was computed from

N/A

route not applicable

A number with a superscript letter (for example 0.000a) means the run happened and either scored a genuine zero or needs a note. — means the configuration was never run. A marker in place of a number means it ran and produced nothing scorable. Heat bars scale to the maximum of their own table and are comparable inside it only.

Every number above comes from the harness × model result-matrix appendix, copied cell for cell with no rewriting. — means the configuration was never run; a marker in place of a number means it ran and produced nothing scorable; a number with a superscript letter is a real score that needs a note. Scores from different environments are not comparable.

Pre-training corpus probe

LLM-pretrain-data-probe

Probe a pre-training corpus for its coverage and its poisoning sources using capture–recapture population estimation. Each cell is the mean of 6 episodes — arXiv / bioRxiv / medRxiv × two coverage levels. Score = population-estimate accuracy + canary-source detection + acquisition outcome + budget efficiency, behind a corpus-contamination gate.

harness	Opus-4.8	GPT-5.6-sol	DeepSeek-v4-pro	GLM-5.2	Kimi-K3	Apodex-1.0
Claude Codesingle-model	0.563	—	—	—	—	—
Codex CLIsingle-model	—	0.294^a	—	—	—	—
DeerFlow	0.491	0.601	0.447	0.454	0.351	0.395
OpenHands	0.442	0.534	0.444	0.432	0.379	0.459
A-Evolve	0.553	0.585	0.506	0.567	0.609	0.451
ApodexHarnessin-house	0.553	0.506	0.552	0.385	0.601	0.546

Reading: All 26 cells scored, none zeroed, and the whole field is pressed into a mid-range band of 0.294–0.609. Table high is 0.609 (A-Evolve · Kimi-K3). A-Evolve is both the strongest and the steadiest harness: all six models at or above 0.451, row mean 0.545 against 0.524 for ApodexHarness and 0.456 for DeerFlow. Swapping the model inside one harness moves the score more than swapping the harness does (DeerFlow 0.351–0.601). Codex CLI’s 0.294 is not a capability score: 3 episodes scored 0 after saturating the request rate limiter, and the other 3 average 0.589 — on that showing it belongs in the top tier.

Timed out aCodex scores 0 on three episodes: its tight shell-loop probing exhausts the environment’s per-episode request limiter, which also carries its model traffic. It averages 0.589 on the other three.

nanoGPT speedrun

nanogpt-speedrun

Train nanoGPT to a target validation loss as fast as possible. A submission scores only if it clears both the loss gate and the speed gate; otherwise it is 0.

harness	Opus-4.8	GPT-5.6-sol	DeepSeek-v4-pro	GLM-5.2	Kimi-K3	Apodex-1.0
Claude Codesingle-model	0.009	—	—	—	—	—
Codex CLIsingle-model	—	0.014	—	—	—	—
DeerFlow	0.062	0.000^b	0.109	0.000^ab	0.005	0.000^b
OpenHands	0.123	0.147	0.005	0.056^a	0.033	0.025^a
A-Evolve	0.025	0.005	0.004	0.018	0.000^b	0.037
ApodexHarnessin-house	0.016	0.000^ac	0.000^c	0.108	0.076	0.000^b

Reading: The lowest-scoring environment on the page. Of 26 cells, 7 score 0.000, and the ceiling is only 0.147 (OpenHands · GPT-5.6-sol) — no cell anywhere in the table clears 0.15. The two hard gates, loss and speed, zero out everything that merely almost worked, and note b shows how narrow that margin can be: where the loss gate was met, the speed gate was missed by under 13 s. What separates solvers here is not model quality but whether they converge inside the budget and submit on time. OpenHands leads on row mean (0.065) at roughly twice the next harness.

Note aMean over two episodes; the third returned no result.

Gate not met bZero from mixed per-episode causes; where the loss gate was met, the speed gate was missed by under 13 s.

Not submitted cNo episode produced a submittable training run.

Pre-training data filter

pretrain-data-filter

Train a multi-head quality scorer to filter pre-training data. Score = per-dimension Pearson correlation against the reference, behind a throughput gate that zeroes the cell if missed.

harness	Opus-4.8	GPT-5.6-sol	DeepSeek-v4-pro	GLM-5.2	Kimi-K3	Apodex-1.0
Claude Codesingle-model	0.485	—	—	—	—	—
Codex CLIsingle-model	—	0.497	—	—	—	—
DeerFlow	0.234^b	0.491	0.000^a	0.217^b	0.416	0.000^a
OpenHands	0.446	0.215^b	0.000^a	0.223^b	0.000^c	0.224^b
A-Evolve	0.457	0.532	0.244^b	0.000^a	0.254^b	0.000^a
ApodexHarnessin-house	0.462	0.429	0.000^a	0.241^b	0.154^b	0.354

Reading: All the real separation here is in whether the submission survives scoring at all. 7 of 26 cells are 0.000 and a further 9 lost one of their two episodes, which halves the mean — so most of the spread below the top is an artefact of the gate, not of filter quality. The dominant cause is a submission that is not self-contained, loading a backbone from a path the offline scoring sandbox does not mount. Table high is 0.532 (A-Evolve · GPT-5.6-sol), and the two single-model CLIs — Claude Code 0.485 and Codex CLI 0.497 — beat every multi-model harness on row mean, because they lost no episodes.

Gate not met aZero in both episodes: the submission missed a hard gate — most often it was not self-contained, loading a backbone from a path the offline scoring sandbox does not mount.

Gate not met bZero in one of the two episodes, so the mean is roughly half the working episode’s score.

Fault cOne episode only; the other returned no result (harness SDK fault) and is excluded.

Operator alignment detection

operator-align

Run TransformerEngine on GPU and find numerically misaligned or mismatched operators. Outcome score = pooled F1 × (0.5 + 0.5 · trigger_rate), computed jointly rather than averaged per instance, with a process score on top.

harness	Opus-4.8	GPT-5.6-sol	DeepSeek-v4-pro	GLM-5.2	Kimi-K3	Apodex-1.0
Claude Codesingle-model	0.722	—	—	—	—	—
Codex CLIsingle-model	—	0.491	—	—	—	—
DeerFlow	0.663	0.659	0.659	0.675	0.725	0.714
OpenHands	0.740	0.409	0.592	0.631	0.646	0.716
A-Evolve	0.722	0.662	0.628	0.712	0.720	0.641
ApodexHarnessin-house	0.727	0.734	0.657	0.653	0.743	0.607

Reading: All 26 cells scored, none zeroed, and the tightest field on the page: everything sits in 0.409–0.743 and every harness row mean lands in 0.622–0.722. Table high is 0.743 (ApodexHarness · Kimi-K3). The Opus-4.8 column is the steadiest (0.663–0.740 across five harnesses); the GPT-5.6-sol column is by far the most scattered (0.409–0.734), and its low end, OpenHands · GPT 0.409, is the weakest cell in the table. Pooling detections across all 33 instances before scoring is what keeps this board so compressed — no single instance can carry or sink a run.

Internal benchmark construction

internal-benchmark

Score a submitted evaluation harness against an internal reference suite. Item difficulty is judged by a fixed held-out model, Qwen-3.5-35B. Only this single-reference setting was scored; the alternative reference-model variant and its subtasks were not run.

harness	Opus-4.8	GPT-5.6-sol	DeepSeek-v4-pro	GLM-5.2	Kimi-K3	Apodex-1.0
Claude Codesingle-model	0.160	—	—	—	—	—
Codex CLIsingle-model	—	0.097	—	—	—	—
DeerFlow	0.000^a	0.138	0.088	0.085	0.000^b	0.020
OpenHands	0.089	0.195	0.044	0.044	0.000^b	0.057
A-Evolve	0.065	0.117	0.048	0.046	0.031	0.078
ApodexHarnessin-house	0.355	0.102	0.070	0.108	0.000^b	0.068

Reading: Nobody actually built a high-quality benchmark. The ceiling is 0.355 (ApodexHarness · Opus-4.8) and it is a lone outlier — every other cell in the table is at or below 0.195, and 4 are 0.000. A-Evolve is the only harness where all six models produced a non-zero score, but its band is just 0.031–0.117. All three protocol violations land on Kimi-K3, one each under DeerFlow, OpenHands and ApodexHarness, which makes it a model-side protocol-following problem rather than any one harness’s fault. The two single-model CLIs (0.160 / 0.097) are unremarkable here.

Timed out aTimed out due to a slow multi-round loop in one of the two settings.

Not submitted bDid not follow the submission protocol.

RL recipe

rl-recipe

Score a submitted reinforcement-learning recipe by its improvement over the default recipe on a held-out set. The outcome score is imputed via the HDS6 proxy (r = 0.979), then combined with anti-gaming and cost terms and normalised to 0–1.

harness	Opus-4.8	GPT-5.6-sol	DeepSeek-v4-pro	GLM-5.2	Kimi-K3	Apodex-1.0
Claude Codesingle-model	0.751	—	—	—	—	—
Codex CLIsingle-model	—	0.800	—	—	—	—
DeerFlow	0.821	0.791	0.748	0.724	0.789	0.731
OpenHands	0.747	0.797	0.776	0.805	0.732	0.732
A-Evolve	0.751	0.779	0.709	0.742	0.793	0.708
ApodexHarnessin-house	0.741	0.804	0.749	0.732	0.807	0.697

Reading: All 26 cells scored, none zeroed — as in 01, 04, 07, 09 and 10 — and all of them crowd into 0.697–0.821, a spread of 0.124 with every harness row mean inside 0.747–0.800. The peak is 0.821 (DeerFlow · Opus-4.8) and even the lowest cell, ApodexHarness · Apodex-1.0, reaches 0.697. In its current form this environment barely separates anything, and the caption says why: the true held-out outcome spans only 0.626–0.715, so what is plotted is dominated by the anti-hacking term rather than by recipe quality.

Post-training data dedup

posttrain-data-dedup

Deduplicate post-training data and strip contamination against the held-out test sets. Score combines three terms: dedup F1, leak recall on decontamination, and over-deletion rate.

harness	Opus-4.8	GPT-5.6-sol	DeepSeek-v4-pro	GLM-5.2	Kimi-K3	Apodex-1.0
Claude Codesingle-model	0.386	—	—	—	—	—
Codex CLIsingle-model	—	0.296	—	—	—	—
DeerFlow	0.304	0.269	0.310	0.360	0.266	0.319
OpenHands	0.233	0.272	0.249	0.273	0.309	0.273
A-Evolve	0.307	0.338	0.309	0.416	0.402	0.319
ApodexHarnessin-house	0.316	0.304	0.279	0.252	0.234	0.301

Reading: All 26 cells scored, none zeroed, but the whole field is pressed into a narrow 0.233–0.416 band with a visibly low ceiling. Best is 0.416 (A-Evolve · GLM-5.2), worst is OpenHands · Opus-4.8 at 0.233. A-Evolve has the strongest row mean (0.349) and OpenHands the weakest (0.268) — a gap of only 0.081, while swapping the model inside A-Evolve alone moves the score by 0.109. Model choice matters at least as much as harness choice here, and neither matters much.

Reasoning-length controllable fine-tune

swe-juice

Fine-tune Qwen3-8B so that its reasoning length responds to a reasoning-effort knob, without losing agentic SWE ability. Score = controllability of the effort→length mapping + solve rate on hidden GitHub issues.

harness	Opus-4.8	GPT-5.6-sol	DeepSeek-v4-pro	GLM-5.2	Kimi-K3	Apodex-1.0
Claude Codesingle-model	0.000^a	—	—	—	—	—
Codex CLIsingle-model	—	0.210	—	—	—	—
DeerFlow	0.362	0.215	0.000^b	0.462	0.399	0.000^c
OpenHands	0.096	0.090	0.244	Fault d	Fault d	0.000
A-Evolve	0.493	Refused e	0.243	0.095	0.000	0.243
ApodexHarnessin-house	0.245	0.246	0.000^f	0.000^g	0.244	0.248

Reading: The messiest failure profile on the page: of 26 configurations only 23 produced a score, and behind them sit 2 safety refusals, 4 harness/API/compatibility faults, 1 run where training never started, and 1 non-submission. Only A-Evolve · Opus-4.8 0.493 and DeerFlow · GLM-5.2 0.462 finished training and came away with a respectable score. Four cells in the ApodexHarness row cluster at 0.244–0.248, as if they hit the same ceiling.

Not submitted aNo submission.

Fault bHarness bug killed training.

Timed out cKept exploring, training never started.

Fault dModel–harness compatibility fault.

Refused eOpenAI refused to run (safety flag).

Fault fApodexHarness’s history truncation drops the reasoning-content field, causing a 400 from the model API.

Refused gRefused on content-policy grounds.

Patch correctness verdict

solution-verifier

Decide whether a candidate code-fix patch is correct, across pick instances (choose the correct candidate) and rank instances (order the candidates), drawn from OpenSWE and ACR. Score = binary accuracy of the pick and rank verdicts.

harness	Opus-4.8	GPT-5.6-sol	DeepSeek-v4-pro	GLM-5.2	Kimi-K3	Apodex-1.0
Claude Codesingle-model	0.80	—	—	—	—	—
Codex CLIsingle-model	—	0.90	—	—	—	—
DeerFlow	0.80	0.80	0.75	0.65	0.90	0.80
OpenHands	0.75	0.95	0.75	0.60	0.80	0.50
A-Evolve	0.85	0.90	0.75	0.80	0.75	0.50
ApodexHarnessin-house	0.85	0.95	0.70	0.85	0.90	0.60

Reading: The strongest environment on the page (0.50–0.95), with no gaps and no zeroes across 26 cells. OpenHands · GPT-5.6-sol and ApodexHarness · GPT-5.6-sol tie at 0.95, and GPT-5.6-sol is the best column on every harness that runs it. The Apodex-1.0 column is visibly the weakest: 0.50 under both OpenHands and A-Evolve, 0.60 under ApodexHarness, with only DeerFlow reaching 0.80. GLM-5.2 is the other soft spot (0.60–0.85). ApodexHarness has the best row mean (0.808), OpenHands the worst (0.725).

Inference determinism repair

llm-determinism

Locate and fix the sources of non-determinism in a pinned SGLang deployment. Scored by a hidden bit-exactness test suite, behind a no-regression hard gate that zeroes the cell the moment it is violated. Cells are not all scored the same way: an episode that reached submit gets one repair round, while one that exhausted its budget is scored once on its final state — and the two highest scores in this table are of the latter kind.

harness	Opus-4.8	GPT-5.6-sol	DeepSeek-v4-pro	GLM-5.2	Kimi-K3	Apodex-1.0
Claude Codesingle-model	0.375	—	—	—	—	—
Codex CLIsingle-model	—	0.636	—	—	—	—
DeerFlow	0.424	0.452	0.347	0.477	0.531	0.312
OpenHands	0.687	0.742	0.359	0.441	0.490	0.310
A-Evolve	0.341	0.740	0.354	0.577	0.547	0.313
ApodexHarnessin-house	0.625	0.416	0.506	0.550	0.625	0.343

Reading: All 26 cells scored, none zeroed, spread over 0.310–0.742. OpenHands is the strongest harness — Opus-4.8 0.687 and GPT-5.6-sol 0.742 are both best in column — with A-Evolve · GPT-5.6-sol 0.740 close behind. The two single-model CLIs do not lead here: Claude Code · Opus-4.8 reaches only 0.375, nearly half of what the same model does under OpenHands (0.687) and ApodexHarness (0.625). The Apodex-1.0 column is both the weakest and the flattest (0.310–0.343, almost identical across four harnesses) — on that side the model’s own ceiling binds first. Yet the Opus-4.8 column spans 0.341–0.687, so the harness can halve the score too. Both effects are strong in this table.

Rollout three-way verification

LLM-rollout-verify

Build a three-way verifier for mathematical solutions — correct, incorrect, or an ill-posed problem. The agent writes a judge_one function and the environment runs it over the whole corpus. Score = macro-F1 on a hidden held-out set, log-discounted by the output token cost per item.

harness	Opus-4.8	GPT-5.6-sol	DeepSeek-v4-pro	GLM-5.2	Kimi-K3	Apodex-1.0
Claude Codesingle-model	0.050	—	—	—	—	—
Codex CLIsingle-model	—	0.480^a	—	—	—	—
DeerFlow	0.000^b	0.000^b	0.000^b	0.000^b	0.000^b	0.000^b
OpenHands	0.000^a	0.856	0.000^b	0.000^a	0.000^b	0.000^b
A-Evolve	0.000^b	0.100	0.067	0.000^b	0.100	0.000^b
ApodexHarnessin-house	0.000^a	0.856	0.000^a	0.820	0.856	0.100

Reading: The most sharply bimodal environment on the page. Of 26 cells, 16 are 0.000; at the other end four cluster at the top — OpenHands · GPT-5.6-sol, ApodexHarness · GPT-5.6-sol and ApodexHarness · Kimi-K3 all at 0.856, with ApodexHarness · GLM-5.2 at 0.820 — and only 6 land in between (0.480, 0.100, 0.100, 0.100, 0.067, 0.050). Read both ends with care. Every cell is a best-of-n over that cell’s repeats, so the high end is a best case, not a typical one. At the low end an unannotated zero would not mean what it appears to: of the 16 zero cells, 4 are marked genuine — the run completed and satisfied none of the required fixes — and 12 are harness artefacts where no scorable submission ever reached the world, including the whole DeerFlow row. At episode level the appendix records 8 genuine among 23 zeros. No zero anywhere is caused by a confirmed regression: the no-regression gate fired five times across 72 episodes and no firing was ever confirmed. Effectively only GPT-5.6-sol clears this environment reliably, plus ApodexHarness on Kimi-K3 and GLM-5.2; ApodexHarness has by far the best row mean (0.439) and DeerFlow scores nothing at all.

real aGenuine: the run completed and satisfied none of the required fixes.

artifact bHarness artifact — no scorable submission ever reached the world: early stopping or a tool-frequency guard (DeerFlow), an iteration or wall-clock cap.