Coding-agent harness benchmarks

Compares coding-agent harnesses while holding the model and task fixed. Each board is one result-sealed bundle.

8 harnesses4 models5 result-sealed bundles773 valid result rows704 matched result rowsupdated 2026-07-21

Hide boards with disclosed caveats

Scores are not comparable across bundles: task sets, trial counts, and timeout caps differ.

OpenBench: gpt-5.6 across 7 harnesses

kind releasedate 2026-07-21denominators matched (task, trial)matched rows 294common task/trials 42results SHA 546f167576aatask-set SHA d949d46e8aae

gpt-5.6-sol

Harness↕	Solve rate · Wilson 95%↓ 050100%	Solved↕	Median wall↕	Fresh tokens/solve↕	Uncached input/solve↕	Output/solve↕	Cache-read/solve↕	Cache-write/solve↕	Telemetry source / basis	Telemetry coverage
cursor	85.7% 72.2%–93.3%	36/42	42.9s	5,519	72	5,446	407,067	45,776	nativeharness_reported	proxy 0/42 · native 42/42
devin	81.0% 66.7%–90.0%	34/42	68.4s	—	—	—	—	—	unavailable	proxy 0/42 · native 0/42
grokbuild	81.0% 66.7%–90.0%	34/42	86.3s	46,216	38,727	7,489	368,079	0	proxyproxy_measured	proxy 42/42 · native 0/42
opencode	81.0% 66.7%–90.0%	34/42	60.5s	42,634	36,708	5,927	313,916	0	nativevendor_split	proxy 0/42 · native 42/42
claude	76.2% 61.5%–86.5%	32/42	58.8s	40,255	34,455	5,800	153,488	0	proxyproxy_measured	proxy 42/42 · native 42/42
pi	76.2% 61.5%–86.5%	32/42	39.7s	37,602	32,916	4,686	111,408	0	proxyproxy_measured	proxy 42/42 · native 42/42
codex	73.8% 58.9%–84.7%	31/42	94.6s	117,107	102,174	14,933	1,164,511	0	proxyproxy_measured	proxy 42/42 · native 0/42

results.jsonlrelease page

OpenBench: deepseek across 5 harnesses

kind releasedate 2026-07-20denominators matched (task, trial)matched rows 190common task/trials 38results SHA ec6b9d58fd3atask-set SHA 1c4805e7fcd0

deepseek-v4-flashcaveats disclosed

4 caveat(s) from the release page

Archived first-party matrix published without rerunning models. The source contains 225 rows across 5 harnesses, 15 tasks, and 3 trials.
The bundled results SHA is sealed. Current-tree task checks report expected post-run definition drift; recorded task digests are retained as run-time fingerprints.
The unified board ranks only task/trial cells shared by every harness. The historical release page retains its older per-arm countable view, so its denominators differ.
Harness and host/container version stamps vary in the archive. Token columns use the complete counting-proxy lane for every displayed harness.

Harness↕	Solve rate · Wilson 95%↓ 050100%	Solved↕	Median wall↕	Fresh tokens/solve↕	Uncached input/solve↕	Output/solve↕	Cache-read/solve↕	Telemetry source / basis	Telemetry coverage
claude	78.9% 63.7%–88.9%	30/38	47.6s	83,323	35,234	48,089	2,053,623	proxyproxy_measured	proxy 38/38 · native 37/38
pi	73.7% 58.0%–85.0%	28/38	24.5s	69,624	37,987	31,638	1,729,696	proxyproxy_measured	proxy 38/38 · native 35/38
opencode	68.4% 52.5%–80.9%	26/38	40.3s	71,319	37,711	33,608	1,584,020	proxyproxy_measured	proxy 38/38 · native 37/38
grokbuild	65.8% 49.9%–78.8%	25/38	27.7s	131,327	81,795	49,532	1,745,746	proxyproxy_measured	proxy 38/38 · native 0/38
codex	63.2% 47.3%–76.6%	24/38	27.2s	92,723	40,352	52,372	1,212,576	proxyproxy_measured	proxy 38/38 · native 33/38

results.jsonlrelease page

OpenBench: grok-4.5 across 4 harnesses

kind releasedate 2026-07-20denominators matched (task, trial)matched rows 156common task/trials 39results SHA 3905ac339eeatask-set SHA 1c4805e7fcd0

grok-4.5caveats disclosed

4 caveat(s) from the release page

Archived first-party matrix published without rerunning models. The source contains 180 rows across 4 harnesses, 15 tasks, and 3 trials.
The bundled results SHA is sealed. Current-tree task checks report expected post-run definition drift; recorded task digests are retained as run-time fingerprints.
The unified board ranks only task/trial cells shared by every harness. The historical release page retains its older per-arm countable view, so its denominators differ.
No counting-proxy telemetry was retained for this run. Complete native split telemetry is available only where the board reports 100% native coverage.

Harness↕	Solve rate · Wilson 95%↓ 050100%	Solved↕	Median wall↕	Fresh tokens/solve↕	Uncached input/solve↕	Output/solve↕	Cache-read/solve↕	Cache-write/solve↕	Telemetry source / basis	Telemetry coverage
cursor	89.7% 76.4%–95.9%	35/39	25.9s	36,707	28,600	8,106	247,504	0	nativeharness_reported	proxy 0/39 · native 39/39
grokbuild	87.2% 73.3%–94.4%	34/39	42.3s	—	—	—	—	—	unavailable	proxy 0/39 · native 0/39
pi	79.5% 64.5%–89.2%	31/39	25.3s	—	—	—	—	—	unavailable	proxy 0/39 · native 38/39
opencode	76.9% 61.7%–87.4%	30/39	21.2s	55,008	45,494	9,514	250,735	0	nativevendor_split	proxy 0/39 · native 39/39

results.jsonlrelease page

OpenBench showcase: BYO aider vs pi vs opencode (deepseek-v4-flash)

kind communitydate 2026-07-20denominators matched (task, trial)matched rows 24common task/trials 8results SHA 677642cc0649task-set SHA 8b757f8abf38

deepseek-v4-flashcaveats disclosed

4 caveat(s) from the release page

Harness mode is not apples-to-apples: aider ran as a one-shot --message invocation, while pi and opencode ran full agentic loops. Token comparison reflects that difference in harness mode, not a pure like-for-like agent loop contest.
One aider cell excluded: 1 of 9 aider cells was infrastructure-classified and is excluded from solve-rate denominators (aider reported as 8/8). pi and opencode are 9/9.
Uniform total-token basis: totals include uncached input, output, and cache reads from split fields; the vendor aggregate is never used. Values are 6,192 vs 48,262 vs 111,380 tokens/solve (aider vs pi vs opencode).
Verifiable bundle: this release includes results.jsonl and provenance.json. Re-check digests with obench verify docs/releases/2026-07-20-aider-showcase.

Harness↕	Solve rate · Wilson 95%↓ 050100%	Solved↕	Median wall↕	Fresh tokens/solve↕	Uncached input/solve↕	Output/solve↕	Cache-read/solve↕	Telemetry source / basis	Telemetry coverage
aider	100.0% 67.6%–100.0%	8/8	39.0s	5,856	2,308	3,548	336	proxyproxy_measured	proxy 8/8 · native 0/8
opencode	100.0% 67.6%–100.0%	8/8	31.7s	13,669	10,328	3,342	91,600	proxyproxy_measured	proxy 8/8 · native 8/8
pi	100.0% 67.6%–100.0%	8/8	26.4s	7,742	4,928	2,814	35,456	proxyproxy_measured	proxy 8/8 · native 8/8

results.jsonlrelease page

OpenBench: GLM 5.2 quarantine-safe archive

kind releasedate 2026-07-09denominators matched (task, trial)matched rows 40common task/trials 8results SHA 5434f3f8aa66task-set SHA 46075fa27b81

glm-5.2caveats disclosed

4 caveat(s) from the release page

Quarantine-safe derived bundle: all 15 terminal-bench/cancel-async-tasks rows were excluded because its load-sensitive checker is binding-quarantined. No model was rerun.
The bundled results SHA is sealed. Current-tree task checks report expected post-run definition drift; recorded task digests are retained as run-time fingerprints.
The retained archive lacks image digests, version-source labels, and an explicit per-row timeout field; contemporaneous version-stamp drift is documented in the bundle README.
The unified board ranks only task/trial cells shared by every harness. The release page also shows the per-arm countable view, so its denominators differ.

Harness↕	Solve rate · Wilson 95%↓ 050100%	Solved↕	Median wall↕	Fresh tokens/solve↕	Uncached input/solve↕	Output/solve↕	Cache-read/solve↕	Cache-write/solve↕	Telemetry source / basis	Telemetry coverage
pi	100.0% 67.6%–100.0%	8/8	693.5s	—	—	—	—	—	unavailable	proxy 0/8 · native 6/8
opencode	87.5% 52.9%–97.8%	7/8	661.2s	94,197	54,762	39,435	973,166	0	nativevendor_split	proxy 0/8 · native 8/8
claude	62.5% 30.6%–86.3%	5/8	569.0s	—	—	—	—	—	unavailable	proxy 0/8 · native 4/8
grokbuild	62.5% 30.6%–86.3%	5/8	267.6s	—	—	—	—	—	unavailable	proxy 0/8 · native 0/8
codex	50.0% 21.5%–78.5%	4/8	801.1s	—	—	—	—	—	unavailable	proxy 0/8 · native 4/8

results.jsonlrelease page

Not ranked (2)

No result-sealed results.jsonl.

2026-07-02-m3
no results.jsonl (HTML-only release page)
2026-07-20-kimi-k3
no results.jsonl (HTML-only release page)

Managed AI gateway benchmarks

Compares request latency, throughput, and reliability across managed AI gateways under separately scheduled cold and warm conditions. Run facts below describe the latest published bundle; every board keeps separate denominators.

1 published bundle5 routesmodel match: rolling alias30/30 cold blocks30/30 warm blocks300 requestsupdated 2026-07-27

Gateway Bench measures request-level transport and serving telemetry. Cold and warm denominators are separate and are never merged.

GPT-4o mini managed gateway request benchmark

date 2026-07-27model match rolling_aliascold blocks 30/30warm blocks 30/30requests 300verified commit 2cd722da4d65experiment e3649dbc6dc4

Request-level benchmark. Cold and warm conditions retain separate denominators from Harness Bench cells.

Gateway leaderboard

OpenBench Composite: absolute TTFT latency, output throughput, and request success on a 0–100 scale. Higher is better; cost is excluded and Direct OpenAI is an unranked reference.

OpenRouter

100% request success

92.8composite

Cloudflare

100% request success

91.5composite

Concentrate

100% request success

90.4composite

Vercel

100% request success

89.4composite

Cold requests

Complete blocks: 30/30. TTFT includes DNS, TCP, and TLS setup.

Route↕	TTFT p50 / p95↕	Stream total p50 / p95↕	Response headers p50 / p95↕	First body byte p50 / p95↕	Throughput tok/s p50 / p95↕	Total / cached / cache-write tokens p50 / p95
Direct OpenAI	0.556s / 1.384s	0.735s / 1.763s	0.232s / 0.887s	0.232s / 0.888s	92.3 / 123.2	total 51.0 / 54.0 cached 0.0 / 0.0 cache write 0.0 / 0.0
Cloudflare	0.684s / 1.091s	0.812s / 1.215s	0.341s / 0.767s	0.341s / 0.767s	84.6 / 113.7	total 51.0 / 53.5 cached 0.0 / 0.0 cache write 0.0 / 0.0
Concentrate	0.794s / 0.978s	0.930s / 1.803s	0.404s / 0.538s	0.404s / 0.538s	84.9 / 123.0	total 51.0 / 54.0 cached 0.0 / 0.0 cache write 0.0 / 0.0
OpenRouter	0.501s / 0.703s	0.662s / 0.845s	0.459s / 0.668s	0.470s / 0.668s	100.6 / 156.9	total 54.0 / 56.5 cached 0.0 / 0.0 cache write — / — (0/30)
Vercel	0.842s / 2.528s	0.951s / 2.719s	0.773s / 2.485s	0.783s / 2.490s	87.9 / 345.3	total 51.0 / 54.0 cached 0.0 / 0.0 cache write 0.0 / 0.0

Warm requests

Complete blocks: 30/30. TTFT begins when the measured request is sent.

Route↕	TTFT p50 / p95↕	Stream total p50 / p95↕	Response headers p50 / p95↕	First body byte p50 / p95↕	Throughput tok/s p50 / p95↕	Total / cached / cache-write tokens p50 / p95
Direct OpenAI	0.504s / 0.977s	0.656s / 1.221s	0.184s / 0.436s	0.184s / 0.437s	95.1 / 127.2	total 52.0 / 54.5 cached 0.0 / 0.0 cache write 0.0 / 0.0
Cloudflare	0.561s / 0.739s	0.754s / 0.893s	0.260s / 0.410s	0.260s / 0.410s	96.6 / 126.4	total 52.0 / 54.5 cached 0.0 / 0.0 cache write 0.0 / 0.0
Concentrate	0.704s / 0.933s	0.896s / 1.096s	0.361s / 0.436s	0.361s / 0.436s	85.3 / 131.6	total 52.0 / 54.5 cached 0.0 / 0.0 cache write 0.0 / 0.0
OpenRouter	0.433s / 0.655s	0.587s / 0.764s	0.423s / 0.485s	0.432s / 0.632s	105.9 / 415.5	total 54.5 / 57.0 cached 0.0 / 0.0 cache write — / — (0/30)
Vercel	0.572s / 1.605s	0.726s / 2.458s	0.559s / 1.571s	0.572s / 1.601s	94.4 / 449.2	total 52.5 / 55.0 cached 0.0 / 0.0 cache write 0.0 / 0.0

Cold setup

Connection setup phases for cold requests only.

Route↕	DNS p50 / p95↕	TCP p50 / p95↕	TLS p50 / p95↕
Direct OpenAI	0.002s / 0.003s	0.011s / 0.012s	0.016s / 0.023s
Cloudflare	0.001s / 0.002s	0.011s / 0.012s	0.017s / 0.023s
Concentrate	0.001s / 0.002s	0.011s / 0.014s	0.018s / 0.023s
OpenRouter	0.001s / 0.002s	0.011s / 0.012s	0.016s / 0.020s
Vercel	0.002s / 0.148s	0.006s / 0.007s	0.024s / 0.028s

Paired request deltas

Every delta is gateway minus Direct OpenAI. These are latency metrics, so positive means slower/worse and negative means faster/better. Medians use complete paired blocks with bootstrap 95% intervals.

Gateway route↕	Condition↕	Δ response headers↕	Δ TTFT↕
Cloudflare	cold	+0.090s 95% CI +0.072s to +0.120s · paired 30/30	+0.095s 95% CI +0.011s to +0.163s · paired 30/30
Cloudflare	warm	+0.081s 95% CI +0.050s to +0.112s · paired 30/30	+0.037s 95% CI +0.003s to +0.116s · paired 30/30
Concentrate	cold	+0.169s 95% CI +0.143s to +0.190s · paired 30/30	+0.193s 95% CI +0.097s to +0.235s · paired 30/30
Concentrate	warm	+0.168s 95% CI +0.143s to +0.204s · paired 30/30	+0.202s 95% CI +0.142s to +0.230s · paired 30/30
OpenRouter	cold	+0.233s 95% CI +0.212s to +0.244s · paired 30/30	-0.088s 95% CI -0.176s to -0.009s · paired 30/30
OpenRouter	warm	+0.230s 95% CI +0.202s to +0.252s · paired 30/30	-0.054s 95% CI -0.109s to -0.018s · paired 30/30
Vercel	cold	+0.508s 95% CI +0.440s to +0.596s · paired 30/30	+0.227s 95% CI +0.171s to +0.297s · paired 30/30
Vercel	warm	+0.348s 95% CI +0.328s to +0.465s · paired 30/30	+0.058s 95% CI +0.014s to +0.214s · paired 30/30

results.jsonl

OpenBench benchmark results

Digest-verified benchmark releases, methodology, and project information.

coding-agent harnessesmanaged AI gateway routes

Releases

First-party bundles.

OpenBench M3 — Harness Comparison
2026-07-02 · gpt-5.5-medium
OpenBench: GLM 5.2 quarantine-safe archive
2026-07-09 · glm-5.2
OpenBench: grok-4.5 across 4 harnesses
2026-07-20 · grok-4.5
OpenBench: kimi-k3 across 3 harnesses
2026-07-20 · kimi-k3
OpenBench: deepseek across 5 harnesses
2026-07-20 · deepseek-v4-flash
OpenBench: gpt-5.6 across 7 harnesses
2026-07-21 · gpt-5.6-sol

Packs

Versioned task and harness packs.

openbench/core-smoke@1.0.0 tasks
Apache-2.0 · data/packs/openbench-core-smoke · 3b1e576398a8
Tiny polarity-checked smoke tasks (make-it-run, fix-failing-test)

OpenBench benchmark results

Digest-verified benchmark releases, methodology, and project information.

coding-agent harnessesmanaged AI gateway routes

What is being measured

OpenBench runs two benchmark families. They share a task contract and a checker, and no denominators.

Harness Bench

Varies the coding-agent harness — the CLI that wraps a model in a run loop, tool set, and permission policy — while holding the model and task fixed. An arm is (harness, model). A task is solved when its checker.sh exits 0; the harness's own claim of success is never trusted.

Gateway Bench

Measures one model request at a time under separately scheduled cold and warm transport conditions. It reports request success, route verification, transport and stream phase timing, throughput, usage, and per-request cost. It is not a coding-agent outcome benchmark. Gateway Bench requests and Harness Bench cells are never pooled or compared as one denominator.

The Gateway Bench leaderboard uses an absolute summary score: 30% cold TTFT median, 15% cold TTFT p95, 30% warm TTFT median, 15% warm TTFT p95, and 10% warm median output throughput. Cold TTFT includes DNS, TCP, and TLS. Latency scores linearly from 100 at zero to zero at 20 seconds; throughput scores linearly from zero at 5 tok/s to 100 at 200 tok/s. The weighted result is multiplied by request success. Cost is excluded. Direct OpenAI is an unranked reference, and the detailed measurements below the score remain the factual record.

Denominators and intervals

Denominators are countable cells. Infrastructure and rate-limit failures are excluded; other failures, including timeouts, stay in the denominator.
Harness Bench uses Wilson 95% intervals over matched (task, trial) cells whenever a bundle has two or more arms.
Gateway Bench displays complete cold and warm block counts separately. Availability uses a Wilson 95% interval over attempted requests. Phase summaries use successful, route-verified requests and retain metric-specific coverage. Paired deltas use complete gateway/direct blocks and bootstrap 95% intervals.

Efficiency and cost

Median wall time is taken among solved cells only.
Each Harness Bench arm uses one complete split-token lane across all matched result rows, preferring proxy telemetry and otherwise using native telemetry. Fresh tokens are uncached input plus output; cache reads and cache writes remain separate. Incomplete lanes produce no token metrics and report their row coverage instead.
Each per-solve token figure sums traffic from every matched attempt, including failed attempts, then divides by the number of solved cells. It measures attempted traffic required per solve, not the average size of successful attempts alone.
Harness $/solve appears only for models with a configured price.
Gateway Bench response headers are the time until HTTP response headers. First body byte and semantic TTFT are reported separately; response headers are not labeled TTFB.
Gateway Bench measured cost is the frozen-list request estimate. Charged cost is separately reported billing evidence. Each retains its own request coverage, as do total, cached-input, and cache-write token readings.
Harness defaults are not clamped.

Comparability

Cells from different bundles are never blended. Each board is one bundle; cross-bundle ranking on different task sets is not supported.
Every ranked bundle ships results.jsonl plus a provenance digest and is re-verified before it appears here. Digests show tamper-evidence, not absence of cherry-picking.
Results cover only the included tasks, trials, model deployments, harness versions, and timeout caps.

Reproducing a board

Every board links its results.jsonl. Re-check a bundle with obench verify <bundle> (harness) or obench gateway probe verify <bundle> (gateway), and rebuild this page with obench site build.

OpenBench benchmark results

Digest-verified benchmark releases, methodology, and project information.

coding-agent harnessesmanaged AI gateway routes

Contact

Want to add a gateway or harness, submit results, report a problem, or share an idea? Reach Matthew through either channel.

Open a GitHub issue Message @mattlam_ on X