OpenBench: gpt-5.6 across 7 harnesses

gpt-5.6-sol

Provenance note: image_digest differs across groups; harness_version_source mixed within group; harness_version_source mixed within group; harness_version_source mixed within group; harness_version_source mixed within group; harness_version_source mixed within group — a container-rebuild / host-vs-container stamping artifact, not a CLI version change. Pinned CLI versions are identical across arms.

Correctness

Bars show solve rate; whiskers show Wilson 95% confidence intervals.

Efficiency

Lower-right is better: more correct and leaner. Diamonds identify cursor and opencode; their token totals are self-reported.

Speed

Lower-right is better: more correct with a lower median wall time among solved runs.

Arm (harness × model)	Solved/countable	Solve rate	Wilson 95% CI	Med wall	Total tokens/solve (incl. cache reads)	Uncached in/solve	Output/solve	Cache-read/solve	Token basis	Excluded / unmatched / duplicates
cursor × gpt-5.6-sol	37/43	86.0%	72.7–93.4%	43.3s	446,257	73	5,738	440,446	self-reported	2 / 0 / 0
grokbuild × gpt-5.6-sol	37/45	82.2%	68.7–90.7%	89.4s	410,474	36,754	7,488	366,232	self-reported	0 / 0 / 0
opencode × gpt-5.6-sol	36/45	80.0%	66.2–89.1%	61.5s	362,976	36,530	6,148	320,299	self-reported	0 / 0 / 0
devin × gpt-5.6-sol	36/45	80.0%	66.2–89.1%	74.3s	—	—	—	—	self-reported	0 / 0 / 0
claude × gpt-5.6-sol	35/45	77.8%	63.7–87.5%	60.8s	194,532	33,461	5,833	155,238	self-reported	0 / 0 / 0
pi × gpt-5.6-sol	32/44	72.7%	58.2–83.7%	39.7s	149,259	33,118	4,733	111,408	self-reported	1 / 0 / 0
codex × gpt-5.6-sol	32/44	72.7%	58.2–83.7%	95.9s	—	68,168	10,877	929,384	proxy-measured, self-reported	1 / 0 / 0

Timeouts: codex: 1 timeout; finished-basis 74.4%.

Total tokens/solve (incl. cache reads) is computed uniformly from the split fields: uncached input + output + cache read. The vendor `tokens` field is never used. Split fields use each harness's own classification and are NOT cross-comparable. Badge bases: unmetered / self-reported / proxy-measured.

Harness	gpt-5.6-sol
claude	77.8%
codex	72.7%
cursor	86.0%
devin	80.0%
grokbuild	82.2%
opencode	80.0%
pi	72.7%

Methodology & limitations

Method

Denominators are countable cells. Infrastructure and rate-limit failures are excluded; other failures, including timeouts, remain in the denominator.
Correctness uses the canonical OpenBench aggregation and Wilson 95% confidence intervals. Where timeouts occur, the report shows both accuracy at the configured cap and accuracy among finished cells.
Efficiency is descriptive: median wall time among solved cells and total measured tokens divided by solves. Token basis tags identify vendor/CLI-reported versus proxy-metered values.
Harness defaults are deliberately not clamped because they are part of the product being evaluated. Sampling and provenance should be inspected in the source JSONL.

Limitations

Results cover only the included tasks, trials, model deployments, harness versions, and timeout caps; they do not establish a universal ranking.
Mixed versions or caps, incomplete cells, and differing token protocols can weaken comparability. CLI-reported and proxy-metered token values should not be treated as interchangeable.
Cost appears only for models with a configured price and includes uncached input and output according to the canonical OpenBench price calculation; subscription pricing and cache-specific pricing are not inferred.
Long-horizon work, multi-agent modes, and unsupported model/harness combinations may not be represented.