CODING AGENT COMPARISON · OCTOBER 2026

Coding agents, compared.

The latest parser pairs use current models chosen from the live cost, quality and speed ladders and catalog added dates. Both agents request matched sampling and reasoning settings on the unchanged task; unsupported provider settings are disclosed. Other task results are retained from the prior campaign. Every attempt remains available.

Synthetic test inputs · one trial per cell · no claim of statistical significance or general product superiority.

9Completed matched-harness runs

6/6Prior native coding runs passed

2/2Prior Blueprint plans passed

36 earlier attempts, incomplete runs and diagnostics are retained below. They are excluded from coding success counts. Completed failures remain in the results.

Results by task

Verdict Code vs Aider

Same parser task and model. Each cell shows independent checks passed. Current models are selected from live ladders and catalog added dates. Older rosters remain in the audit archive.

Synthetic benchmark fixtures · actual agent or service execution.

Exact model	Verdict Code	Aider
anthropic/claude-haiku-5-5Current model · catalog verified · synthetic parser task	Passed20/20 checks13.412 s · USD not measuredReceipt ↗ Run details Model: byok/anthropic/claude-haiku-5-5 Observed native model: Not measured Gateway response model: byok/anthropic/claude-haiku-5-5 Reasoning setting: default Input / output tokens: 11,092 / 1,655 Total tokens: 12,747 Accounted tokens / wall-second: 950.418 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Provider cost estimation returned zero despite positive usage and catalog prices. Cost is unavailable; the separately labelled catalog estimate is not an observed charge.	Passed20/20 checks14.012 s · USD not measuredReceipt ↗ Run details Model: byok/anthropic/claude-haiku-5-5 Observed native model: Not measured Gateway response model: byok/anthropic/claude-haiku-5-5 Reasoning setting: default Input / output tokens: 1,241 / 2,617 Total tokens: 3,858 Accounted tokens / wall-second: 275.335 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Provider cost estimation returned zero despite positive usage and catalog prices. Cost is unavailable; the separately labelled catalog estimate is not an observed charge.
openai/gpt-6-lunaCurrent model · catalog verified · synthetic parser task	Passed20/20 checks30.985 s · $0.001266Receipt ↗ Run details Model: byok/openai/gpt-6-luna Observed native model: Not measured Gateway response model: byok/openai/gpt-6-luna Reasoning setting: default Input / output tokens: 5,713 / 1,390 Total tokens: 7,103 Accounted tokens / wall-second: 229.24 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.	Passed20/20 checks30.003 s · $0.000872Receipt ↗ Run details Model: byok/openai/gpt-6-luna Observed native model: Not measured Gateway response model: byok/openai/gpt-6-luna Reasoning setting: default Input / output tokens: 836 / 1,576 Total tokens: 2,412 Accounted tokens / wall-second: 80.392 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
qwen/qwen3.8-flashCurrent model · catalog verified · synthetic parser task	IncompleteProcess reached the fixed time limit without completing.Receipt ↗ Run details Model: byok/qwen/qwen3.8-flash Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: default Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Incomplete whole-attempt accounting; captured provider-call subtotals are lower bounds only.	Passed20/20 checks220.804 s · $0.005058Receipt ↗ Run details Model: byok/qwen/qwen3.8-flash Observed native model: Not measured Gateway response model: byok/qwen/qwen3.8-flash Reasoning setting: default Input / output tokens: 934 / 10,463 Total tokens: 11,397 Accounted tokens / wall-second: 51.616 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
zai/glm-5.3-flashCurrent model · catalog verified · synthetic parser task	Failed checks18/20 checks30.955 s · $0.001061Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: byok/zai/glm-5.3-flash Reasoning setting: low Input / output tokens: 3,984 / 927 Total tokens: 4,911 Accounted tokens / wall-second: 158.65 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.	Passed20/20 checks9.204 s · $0.000348Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: byok/zai/glm-5.3-flash Reasoning setting: low Input / output tokens: 822 / 449 Total tokens: 1,271 Accounted tokens / wall-second: 138.092 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
zai/glm-5.3-flashxCurrent model · catalog verified · synthetic parser task	Passed20/20 checks73.742 s · $0.013988Receipt ↗ Run details Model: byok/zai/glm-5.3-flashx Observed native model: Not measured Gateway response model: byok/zai/glm-5.3-flashx Reasoning setting: default Input / output tokens: 12,279 / 7,556 Total tokens: 19,835 Accounted tokens / wall-second: 268.978 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.	Passed20/20 checks40.713 s · $0.007165Receipt ↗ Run details Model: byok/zai/glm-5.3-flashx Observed native model: Not measured Gateway response model: byok/zai/glm-5.3-flashx Reasoning setting: default Input / output tokens: 822 / 5,489 Total tokens: 6,311 Accounted tokens / wall-second: 155.012 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.

Fix a regression

Repair invoice rounding across two modules. Nine behavior checks plus one permitted-file check. Different model configurations: Verdict Code used GLM-5.3-Flash; native agents used their subscription models.

Retained from the prior campaign; not rerun here.

Synthetic benchmark fixtures · actual agent or service execution.

Agent / model	Task	Result	Time	Estimated USD	Evidence
Devin CLIswe-2-mediumSubscription model	Repair money calculations across modulesSynthetic test input	Passed10/10 checks	10.936 s	Not measured	Receipt ↗ Run details Model: swe-2-medium Observed native model: swe-2-medium Provider model receipt: Not exposed Identity source: native exporter generation_model Reasoning setting: See invocation receipt Input / output tokens: 72,162 / 1,086 Total tokens: 73,248 Accounted tokens / wall-second: 6,697.879 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Auth uses existing native subscription/prepaid account; billing decrement unavailable. Native models differ: this is a product pilot, not a same-model comparison. Synthetic authored fixture; no statistical ranking. Custom pilot receipt has not been admitted through BenchmarkResult to JSON/DuckDB/Neon; no leaderboard promotion.
Claude Codeclaude-sonnet-5-5Subscription model	Repair money calculations across modulesSynthetic test input	Passed10/10 checks	19.102 s	Not measured	Receipt ↗ Run details Model: claude-sonnet-5-5 Observed native model: claude-sonnet-5-5 Provider model receipt: Not exposed Identity source: native exporter modelUsage canonical model Reasoning setting: See invocation receipt Input / output tokens: 129,952 / 1,655 Total tokens: 131,607 Accounted tokens / wall-second: 6,889.697 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Auth uses existing native subscription/prepaid account; billing decrement unavailable. Native models differ: this is a product pilot, not a same-model comparison. Synthetic authored fixture; no statistical ranking. Custom pilot receipt has not been admitted through BenchmarkResult to JSON/DuckDB/Neon; no leaderboard promotion.
Codex CLIgpt-6.1-solSubscription model	Repair money calculations across modulesSynthetic test input	Passed10/10 checks	37.556 s	Not measured	Receipt ↗ Run details Model: gpt-6.1-sol Observed native model: gpt-6.1-sol Provider model receipt: Not exposed Identity source: native-effective-config; provider response identity not exposed Reasoning setting: See invocation receipt Input / output tokens: 114,952 / 1,175 Total tokens: 116,127 Accounted tokens / wall-second: 3,092.102 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Auth uses existing native subscription/prepaid account; billing decrement unavailable. Native models differ: this is a product pilot, not a same-model comparison. Synthetic authored fixture; no statistical ranking. Custom pilot receipt has not been admitted through BenchmarkResult to JSON/DuckDB/Neon; no leaderboard promotion. Provider-response model identity not exposed; native effective selection is recorded separately and does not change acceptance status.
verdict-codezai/glm-5.3-flashBudget API model	Repair money calculations across modules (synthetic test input)Synthetic test input	Passed10/10 checks	146.71 s	$0.013120	Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: byok/zai/glm-5.3-flash Reasoning setting: See invocation receipt Input / output tokens: 76,351 / 3,337 Total tokens: 79,688 Accounted tokens / wall-second: 543.167 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Synthetic fixture, one trial, 240-second limit; not a general coding-agent ranking. 9 independent behavior unittest methods plus 1 changed-file scope check. Model and token accounting require captured provider/usage review; absent values remain unknown. Timeout/transport failure is infrastructure evidence, not a model quality failure. Complete Gateway terminal usage proof; cost is estimated BYOK USD, not a provider cash receipt.

Follow the repository rules

Implement a filename utility with the standard library, preserving the sentinel and permitted change surface.

Retained from the prior campaign; not rerun here.

Synthetic benchmark fixtures · actual agent or service execution.

Agent / model	Task	Result	Time	Estimated USD	Evidence
Devin CLIswe-2-mediumSubscription model	Add a safe filename utility without dependency or scope driftSynthetic test input	Passed10/10 checks	7.391 s	Not measured	Receipt ↗ Run details Model: swe-2-medium Observed native model: swe-2-medium Provider model receipt: Not exposed Identity source: native exporter generation_model Reasoning setting: See invocation receipt Input / output tokens: 69,917 / 563 Total tokens: 70,480 Accounted tokens / wall-second: 9,535.922 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Auth uses existing native subscription/prepaid account; billing decrement unavailable. Native models differ: this is a product pilot, not a same-model comparison. Synthetic authored fixture; no statistical ranking. Custom pilot receipt has not been admitted through BenchmarkResult to JSON/DuckDB/Neon; no leaderboard promotion.
Codex CLIgpt-6.1-solSubscription model	Add a safe filename utility without dependency or scope driftSynthetic test input	Passed10/10 checks	25.571 s	Not measured	Receipt ↗ Run details Model: gpt-6.1-sol Observed native model: gpt-6.1-sol Provider model receipt: Not exposed Identity source: native-effective-config; provider response identity not exposed Reasoning setting: See invocation receipt Input / output tokens: 91,480 / 710 Total tokens: 92,190 Accounted tokens / wall-second: 3,605.256 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Auth uses existing native subscription/prepaid account; billing decrement unavailable. Native models differ: this is a product pilot, not a same-model comparison. Synthetic authored fixture; no statistical ranking. Custom pilot receipt has not been admitted through BenchmarkResult to JSON/DuckDB/Neon; no leaderboard promotion. Provider-response model identity not exposed; native effective selection is recorded separately and does not change acceptance status.
Claude Codeclaude-sonnet-5-5Subscription model	Add a safe filename utility without dependency or scope driftSynthetic test input	Passed10/10 checks	16.218 s	Not measured	Receipt ↗ Run details Model: claude-sonnet-5-5 Observed native model: claude-sonnet-5-5 Provider model receipt: Not exposed Identity source: native exporter modelUsage canonical model Reasoning setting: See invocation receipt Input / output tokens: 85,576 / 908 Total tokens: 86,484 Accounted tokens / wall-second: 5,332.593 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Auth uses existing native subscription/prepaid account; billing decrement unavailable. Native models differ: this is a product pilot, not a same-model comparison. Synthetic authored fixture; no statistical ranking. Custom pilot receipt has not been admitted through BenchmarkResult to JSON/DuckDB/Neon; no leaderboard promotion. Original raw receipt retained FAIL9/10 because pytest runtime cache was counted as authored code. Independent replay corrected this oracle inconsistency: same candidate9/9 behavior plus source scope passes. No new inference or response replacement.
verdict-codezai/glm-5.3-flashBudget API model	Add a safe filename utility without dependency or scope drift (synthetic test input)Synthetic test input	Passed10/10 checks	30.96 s	$0.001267	Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: byok/zai/glm-5.3-flash Reasoning setting: See invocation receipt Input / output tokens: 7,064 / 414 Total tokens: 7,478 Accounted tokens / wall-second: 241.537 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Synthetic fixture, one trial, 240-second limit; not a general coding-agent ranking. 9 independent behavior unittest methods plus 1 changed-file scope check. Model and token accounting require captured provider/usage review; absent values remain unknown. Timeout/transport failure is infrastructure evidence, not a model quality failure. Protocol update: public --task-file filenames.py --pin-task-files --no-ide metadata explicitly scopes context to the same one-file contract already stated in the unchanged prompt. Earlier full-context attempts remain retained; model/task/oracle are unchanged. Complete Gateway terminal usage proof; cost is estimated BYOK USD, not a provider cash receipt.

Blueprint planning

Two frozen specifications. Scope coverage, exclusions and dependency structure are checked separately from application execution.

Retained from the prior campaign; not rerun here.

Synthetic benchmark fixtures · actual agent or service execution.

Agent / model	Explicit dependencies	Prose specification
Devin CLIswe-2-mediumSynthetic specification inputs	Passed13/13 checks5.329 sReceipt ↗ Run details Model: swe-2-medium Observed native model: swe-2-medium Provider model receipt: Not exposed Identity source: native exporter generation_model Reasoning setting: See invocation receipt Input / output tokens: 27,466 / 512 Total tokens: 27,978 Accounted tokens / wall-second: 5,250.141 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Auth uses existing native subscription/prepaid account; billing decrement unavailable. Native models differ: this is a product pilot, not a same-model comparison. Synthetic authored fixture; no statistical ranking. Planning structure rubric only; exact task count and exclusion text checks require semantic review. Not code execution quality. Custom pilot receipt has not been admitted through BenchmarkResult to JSON/DuckDB/Neon; no leaderboard promotion.	Passed10/10 checks16.52 sReceipt ↗ Run details Model: swe-2-medium Observed native model: swe-2-medium Provider model receipt: Not exposed Identity source: native exporter generation_model Reasoning setting: See invocation receipt Input / output tokens: 27,503 / 781 Total tokens: 28,284 Accounted tokens / wall-second: 1,712.107 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Auth uses existing native subscription/prepaid account; billing decrement unavailable. Native models differ: this is a product pilot, not a same-model comparison. Synthetic authored fixture; no statistical ranking. Planning structure rubric only; exact task count and exclusion text checks require semantic review. Not code execution quality. Custom pilot receipt has not been admitted through BenchmarkResult to JSON/DuckDB/Neon; no leaderboard promotion.
Codex CLIgpt-6.1-solSynthetic specification inputs	Passed13/13 checks17.165 sReceipt ↗ Run details Model: gpt-6.1-sol Observed native model: gpt-6.1-sol Provider model receipt: Not exposed Identity source: native-effective-config; provider response identity not exposed Reasoning setting: See invocation receipt Input / output tokens: 48,367 / 537 Total tokens: 48,904 Accounted tokens / wall-second: 2,849.053 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Auth uses existing native subscription/prepaid account; billing decrement unavailable. Native models differ: this is a product pilot, not a same-model comparison. Synthetic authored fixture; no statistical ranking. Planning structure rubric only; exact task count and exclusion text checks require semantic review. Not code execution quality. Custom pilot receipt has not been admitted through BenchmarkResult to JSON/DuckDB/Neon; no leaderboard promotion. Provider-response model identity not exposed; native effective selection is recorded separately and does not change acceptance status.	Passed10/10 checks17.226 sReceipt ↗ Run details Model: gpt-6.1-sol Observed native model: gpt-6.1-sol Provider model receipt: Not exposed Identity source: native-effective-config; provider response identity not exposed Reasoning setting: See invocation receipt Input / output tokens: 48,800 / 530 Total tokens: 49,330 Accounted tokens / wall-second: 2,863.694 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Auth uses existing native subscription/prepaid account; billing decrement unavailable. Native models differ: this is a product pilot, not a same-model comparison. Synthetic authored fixture; no statistical ranking. Planning structure rubric only; exact task count and exclusion text checks require semantic review. Not code execution quality. Custom pilot receipt has not been admitted through BenchmarkResult to JSON/DuckDB/Neon; no leaderboard promotion. Provider-response model identity not exposed; native effective selection is recorded separately and does not change acceptance status.
Claude Codeclaude-sonnet-5-5Synthetic specification inputs	Passed13/13 checks8.83 sReceipt ↗ Run details Model: claude-sonnet-5-5 Observed native model: claude-sonnet-5-5 Provider model receipt: Not exposed Identity source: native exporter modelUsage canonical model Reasoning setting: See invocation receipt Input / output tokens: 42,329 / 849 Total tokens: 43,178 Accounted tokens / wall-second: 4,889.921 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Auth uses existing native subscription/prepaid account; billing decrement unavailable. Native models differ: this is a product pilot, not a same-model comparison. Synthetic authored fixture; no statistical ranking. Planning structure rubric only; exact task count and exclusion text checks require semantic review. Not code execution quality. Custom pilot receipt has not been admitted through BenchmarkResult to JSON/DuckDB/Neon; no leaderboard promotion.	Passed10/10 checks8.677 sReceipt ↗ Run details Model: claude-sonnet-5-5 Observed native model: claude-sonnet-5-5 Provider model receipt: Not exposed Identity source: native exporter modelUsage canonical model Reasoning setting: See invocation receipt Input / output tokens: 42,355 / 753 Total tokens: 43,108 Accounted tokens / wall-second: 4,968.077 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Auth uses existing native subscription/prepaid account; billing decrement unavailable. Native models differ: this is a product pilot, not a same-model comparison. Synthetic authored fixture; no statistical ranking. Planning structure rubric only; exact task count and exclusion text checks require semantic review. Not code execution quality. Custom pilot receipt has not been admitted through BenchmarkResult to JSON/DuckDB/Neon; no leaderboard promotion.
BlueprintNo model inferenceSynthetic specification inputs	Passed13/13 checks0.455 sReceipt ↗ Run details Model: No inference Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: See invocation receipt Input / output tokens: 0 / 0 Total tokens: 0 Accounted tokens / wall-second: 0 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Synthetic PRD input; planning-only: no generated-file, build, or code-execution proof. One trial per task and arm; exact rubric is deliberately narrow; no statistical or production claim. Heuristic planning, no model invocation; zero tokens are actual estimation inference usage. Forecast tokens/cost/time are excluded from actual run metrics. Energy unmeasured. Semantic rubric v2 excludes exact task count from acceptance. Count is diagnostic only; exclusion mentions are evaluated for explicit delivery versus reaffirmed non-delivery. Ambiguous wording remains UNVERIFIED. Semantic-v2 rerun uses originalunchanged synthetic PRDs. Originalv1 attempts remain published separately. No generated-code execution claim.	Passed10/10 checks0.958 sReceipt ↗ Run details Model: No inference Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: See invocation receipt Input / output tokens: 0 / 0 Total tokens: 0 Accounted tokens / wall-second: 0 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Synthetic PRD input; planning-only: no generated-file, build, or code-execution proof. One trial per task and arm; exact rubric is deliberately narrow; no statistical or production claim. Heuristic planning, no model invocation; zero tokens are actual estimation inference usage. Forecast tokens/cost/time are excluded from actual run metrics. Energy unmeasured. Semantic rubric v2 excludes exact task count from acceptance. Count is diagnostic only; exclusion mentions are evaluated for explicit delivery versus reaffirmed non-delivery. Ambiguous wording remains UNVERIFIED. Semantic-v2 rerun uses originalunchanged synthetic PRDs. Originalv1 attempts remain published separately. No generated-code execution claim.
Raw model planning baselinezai/glm-5.3-flashSynthetic specification inputs	Passed13/13 checks13.895 sReceipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: See invocation receipt Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Synthetic PRD input; planning-only: no generated-file, build, or code-execution proof. One trial per task and arm; exact rubric is deliberately narrow; no statistical or production claim. Raw model baseline, not a competing coding agent. Requested model held constant only within this baseline; Blueprint has no inference model. Cost is published-rate estimate at $0.15 input/$0.50 output per MTok; provider cash receipt not captured. Semantic rubric v2 excludes exact task count from acceptance. Count is diagnostic only; exclusion mentions are evaluated for explicit delivery versus reaffirmed non-delivery. Ambiguous wording remains UNVERIFIED. Semantic-v2 rerun uses originalunchanged synthetic PRDs. Originalv1 attempts remain published separately. No generated-code execution claim.	Passed10/10 checks34.303 sReceipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: See invocation receipt Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Synthetic PRD input; planning-only: no generated-file, build, or code-execution proof. One trial per task and arm; exact rubric is deliberately narrow; no statistical or production claim. Raw model baseline, not a competing coding agent. Requested model held constant only within this baseline; Blueprint has no inference model. Cost is published-rate estimate at $0.15 input/$0.50 output per MTok; provider cash receipt not captured. Semantic rubric v2 excludes exact task count from acceptance. Count is diagnostic only; exclusion mentions are evaluated for explicit delivery versus reaffirmed non-delivery. Ambiguous wording remains UNVERIFIED. Semantic-v2 rerun uses originalunchanged synthetic PRDs. Originalv1 attempts remain published separately. No generated-code execution claim.

WHAT CHANGED

The evidence, explained.

The current comparison uses five exact inexpensive models chosen from the live cost, quality and speed ladders and catalog added dates. Earlier model rosters remain only in audit history.
Haiku 5.5 passed 20/20 in 13.412 s with Verdict Code and 14.012 s with Aider. GPT-6 Luna passed 20/20 in 30.985 s and 30.003 s respectively. These are single observations on one synthetic parser task.
GLM-5.3-FlashX passed both arms, but Verdict Code took 73.742 s versus 40.713 s. The latest GLM-5.3-Flash run finished in 30.955 s but failed 2 of 20 checks, versus Aider passing 20/20 in 9.204 s. The worse latest result remains primary.
Qwen 3.8 Flash remains incomplete at the original four-minute limit, despite a partial artifact passing 20/20. Aider completed in 220.804 s. Earlier timeout and stale-credential interruption remain retained; a correct partial artifact is not a completed pass.
Generic repairs address redundant provider turns, cumulative repair bounds, stale self-check evidence, lost workspace locations, contaminated check imports, and required Responses compatibility. Hidden checks were not supplied to candidate execution or used to tune repairs.
All original publications remain byte-preserved. Other native-agent, Blueprint and coding-task families are retained prior observations, not fresh reruns. Incomplete accounting stays unknown; these results do not establish a general agent ranking.

Full audit history

Earlier attempts and execution problems remain visible. They are separate from completed coding results.

Earlier attempts, incomplete runs and diagnostics (36)

Agent / model	Task	Result	Time	Estimated USD	Evidence
aideropenai/gpt-4.1-mini	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Failed checks18/20 checksHistorical roster · retained for audit	10.293 s	$0.001265	Receipt ↗ Run details Model: byok/openai/gpt-4.1-mini Observed native model: Not measured Gateway response model: byok/openai/gpt-4.1-mini Reasoning setting: default Input / output tokens: 831 / 583 Total tokens: 1,414 Accounted tokens / wall-second: 137.375 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
aiderdeepseek/deepseek-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Incomplete0/20 checks on partial artifactHistorical roster · retained for auditIncomplete or apparatus diagnosticblocked · see receipt for cause	48.921 s	$0.010084	Receipt ↗ Run details Model: byok/deepseek/deepseek-flash Observed native model: Not measured Gateway response model: byok/deepseek/deepseek-flash Reasoning setting: default Input / output tokens: 847 / 8,192 Total tokens: 9,039 Accounted tokens / wall-second: 184.767 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Provider output reached the fixed per-request ceiling; censored attempt, not a task-quality failure. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
aiderqwen/qwen3-coder-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Failed checks16/20 checksHistorical roster · retained for audit	11.421 s	$0.001048	Receipt ↗ Run details Model: byok/qwen/qwen3-coder-flash Observed native model: Not measured Gateway response model: byok/qwen/qwen3-coder-flash Reasoning setting: default Input / output tokens: 847 / 529 Total tokens: 1,376 Accounted tokens / wall-second: 120.48 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
aiderzai/glm-5.3-flashBudget API model	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Passed20/20 checksEarlier attempt; superseded by a recorded later run	8.01 s	$0.000332	Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: byok/zai/glm-5.3-flash Reasoning setting: low Input / output tokens: 822 / 417 Total tokens: 1,239 Accounted tokens / wall-second: 154.682 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
aidergoogle/models/gemini-2.5-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Failed checks16/20 checksHistorical roster · retained for audit	18.183 s	$0.008781	Receipt ↗ Run details Model: byok/google/models/gemini-2.5-flash Observed native model: Not measured Gateway response model: byok/google/models/gemini-2.5-flash Reasoning setting: default Input / output tokens: 870 / 3,408 Total tokens: 4,278 Accounted tokens / wall-second: 235.275 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
aiderqwen/qwen3.8-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Incomplete0/20 checks on partial artifactEarlier attempt; superseded by a recorded later runIncomplete or apparatus diagnostictimeout · see receipt for cause	240.012 s	Not measured	Receipt ↗ Run details Model: byok/qwen/qwen3.8-flash Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: default Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Gateway instance continuity was not proven; excluded from matched-model comparison. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Incomplete whole-attempt accounting; captured provider-call subtotals are lower bounds only.
aidergoogle/models/gemini-3.8-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Incomplete0/20 checks on partial artifactHistorical roster · retained for auditIncomplete or apparatus diagnosticblocked · see receipt for cause	55.042 s	$0.031358	Receipt ↗ Run details Model: byok/google/models/gemini-3.8-flash Observed native model: Not measured Gateway response model: byok/google/models/gemini-3.8-flash Reasoning setting: default Input / output tokens: 870 / 8,188 Total tokens: 9,058 Accounted tokens / wall-second: 164.565 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Provider output reached the fixed per-request ceiling; censored attempt, not a task-quality failure. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
verdict-codeopenai/gpt-4.1-mini	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Incomplete18/20 checks on partial artifactEarlier attempt; superseded by a recorded later runHistorical roster · retained for auditIncomplete or apparatus diagnosticblocked · see receipt for cause	57.415 s	Not measured	Receipt ↗ Run details Model: byok/openai/gpt-4.1-mini Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: default Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. A metadata probe disconnected; inference is assessed separately from captured POST completion and accounting. Gateway instance continuity was not proven; excluded from matched-model comparison. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Incomplete whole-attempt accounting; captured provider-call subtotals are lower bounds only.
verdict-codeopenai/gpt-4.1-mini	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Incomplete15/20 checks on partial artifactEarlier attempt; superseded by a recorded later runHistorical roster · retained for auditIncomplete or apparatus diagnosticblocked · see receipt for cause	75.334 s	$0.022086	Receipt ↗ Run details Model: byok/openai/gpt-4.1-mini Observed native model: Not measured Gateway response model: byok/openai/gpt-4.1-mini Reasoning setting: default Input / output tokens: 27,387 / 6,957 Total tokens: 34,344 Accounted tokens / wall-second: 455.89 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
aiderverdict/verdict-4-35b-toolterse-v5-mlxLocal/LAN model	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Failed checks18/20 checksHistorical roster · retained for audit	7.141 s	$0.000000	Receipt ↗ Run details Model: local/lmstudio/verdict/verdict-4-35b-toolterse-v5-mlx Observed native model: Not measured Gateway response model: local/lmstudio/verdict/verdict-4-35b-toolterse-v5-mlx Reasoning setting: default Input / output tokens: 886 / 438 Total tokens: 1,324 Accounted tokens / wall-second: 185.408 (includes input and cached tokens; not decode speed) Energy estimate: 0.257 Wh Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
aiderqwen/qwen3-coder-30bLocal/LAN model	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Failed checks18/20 checksHistorical roster · retained for audit	8.596 s	$0.000000	Receipt ↗ Run details Model: local/lmstudio/qwen/qwen3-coder-30b Observed native model: Not measured Gateway response model: local/lmstudio/qwen/qwen3-coder-30b Reasoning setting: default Input / output tokens: 847 / 574 Total tokens: 1,421 Accounted tokens / wall-second: 165.309 (includes input and cached tokens; not decode speed) Energy estimate: 0.276 Wh Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
aideropenai/gpt-oss-20bLocal/LAN model	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Failed checks16/20 checksHistorical roster · retained for audit	25.471 s	$0.000000	Receipt ↗ Run details Model: local/lmstudio/openai/gpt-oss-20b Observed native model: Not measured Gateway response model: local/lmstudio/openai/gpt-oss-20b Reasoning setting: default Input / output tokens: 900 / 340 Total tokens: 1,240 Accounted tokens / wall-second: 48.683 (includes input and cached tokens; not decode speed) Energy estimate: 0.241 Wh Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
aidergoogle/gemma-4-26b-a4bLocal/LAN model	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Incomplete0/20 checks on partial artifactHistorical roster · retained for auditIncomplete or apparatus diagnostictimeout · see receipt for cause	240.091 s	Not measured	Receipt ↗ Run details Model: local/lmstudio/google/gemma-4-26b-a4b Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: default Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Incomplete whole-attempt accounting; captured provider-call subtotals are lower bounds only.
aiderqwen/qwen3.8-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Passed20/20 checksEarlier attempt; superseded by a recorded later run	192.155 s	$0.005245	Receipt ↗ Run details Model: byok/qwen/qwen3.8-flash Observed native model: Not measured Gateway response model: byok/qwen/qwen3.8-flash Reasoning setting: default Input / output tokens: 934 / 10,861 Total tokens: 11,795 Accounted tokens / wall-second: 61.383 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
verdict-codedeepseek/deepseek-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Passed20/20 checksHistorical roster · retained for auditIncomplete or apparatus diagnostic	18.714 s	$0.003283	Receipt ↗ Run details Model: byok/deepseek/deepseek-flash Observed native model: Not measured Gateway response model: byok/deepseek/deepseek-flash Reasoning setting: default Input / output tokens: 1,253 / 2,423 Total tokens: 3,676 Accounted tokens / wall-second: 196.43 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Diagnostic only: request-local initial tool narrowing skipped the caller-authorized behavioral review. Generic harness regression repaired in 1.4.564; independent scorer outcomes are retained but these timings do not establish a successful latency repair.
verdict-codeopenai/gpt-4.1-mini	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Failed checks18/20 checksHistorical roster · retained for auditIncomplete or apparatus diagnostic	11.536 s	$0.001131	Receipt ↗ Run details Model: byok/openai/gpt-4.1-mini Observed native model: Not measured Gateway response model: byok/openai/gpt-4.1-mini Reasoning setting: default Input / output tokens: 915 / 478 Total tokens: 1,393 Accounted tokens / wall-second: 120.752 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Diagnostic only: request-local initial tool narrowing skipped the caller-authorized behavioral review. Generic harness regression repaired in 1.4.564; independent scorer outcomes are retained but these timings do not establish a successful latency repair.
verdict-codezai/glm-5.3-flashBudget API model	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Passed20/20 checksEarlier attempt; superseded by a recorded later runIncomplete or apparatus diagnostic	23.55 s	$0.000378	Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: byok/zai/glm-5.3-flash Reasoning setting: low Input / output tokens: 1,102 / 425 Total tokens: 1,527 Accounted tokens / wall-second: 64.841 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Diagnostic only: request-local initial tool narrowing skipped the caller-authorized behavioral review. Generic harness regression repaired in 1.4.564; independent scorer outcomes are retained but these timings do not establish a successful latency repair.
verdict-codeqwen/qwen3-coder-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Failed checks16/20 checksHistorical roster · retained for auditIncomplete or apparatus diagnostic	19.527 s	$0.002548	Receipt ↗ Run details Model: byok/qwen/qwen3-coder-flash Observed native model: Not measured Gateway response model: byok/qwen/qwen3-coder-flash Reasoning setting: default Input / output tokens: 3,203 / 1,058 Total tokens: 4,261 Accounted tokens / wall-second: 218.211 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Diagnostic only: request-local initial tool narrowing skipped the caller-authorized behavioral review. Generic harness regression repaired in 1.4.564; independent scorer outcomes are retained but these timings do not establish a successful latency repair.
verdict-codegoogle/models/gemini-2.5-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Failed checks16/20 checksHistorical roster · retained for auditIncomplete or apparatus diagnostic	19.476 s	$0.007063	Receipt ↗ Run details Model: byok/google/models/gemini-2.5-flash Observed native model: Not measured Gateway response model: byok/google/models/gemini-2.5-flash Reasoning setting: default Input / output tokens: 992 / 2,706 Total tokens: 3,698 Accounted tokens / wall-second: 189.875 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Diagnostic only: request-local initial tool narrowing skipped the caller-authorized behavioral review. Generic harness regression repaired in 1.4.564; independent scorer outcomes are retained but these timings do not establish a successful latency repair.
verdict-codeqwen/qwen3.8-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Passed20/20 checksEarlier attempt; superseded by a recorded later runIncomplete or apparatus diagnostic	59.14 s	$0.001914	Receipt ↗ Run details Model: byok/qwen/qwen3.8-flash Observed native model: Not measured Gateway response model: byok/qwen/qwen3.8-flash Reasoning setting: default Input / output tokens: 1,280 / 3,664 Total tokens: 4,944 Accounted tokens / wall-second: 83.598 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Diagnostic only: request-local initial tool narrowing skipped the caller-authorized behavioral review. Generic harness regression repaired in 1.4.564; independent scorer outcomes are retained but these timings do not establish a successful latency repair.
verdict-codegoogle/models/gemini-3.8-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	IncompleteHistorical roster · retained for auditIncomplete or apparatus diagnosticblocked · see receipt for cause	Not measured	Not measured	Receipt ↗ Run details Model: byok/google/models/gemini-3.8-flash Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: default Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Interrupted old-roster attempt. No terminal completion, wall measurement or independent scoring claimed. Captured token subtotals, if any, are lower bounds only. Provider output reached the fixed per-request ceiling; censored attempt, not a task-quality failure. Gateway instance continuity was not proven; excluded from matched-model comparison. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Incomplete whole-attempt accounting; captured provider-call subtotals are lower bounds only. Diagnostic only: request-local initial tool narrowing skipped the caller-authorized behavioral review. Generic harness regression repaired in 1.4.564; independent scorer outcomes are retained but these timings do not establish a successful latency repair.
aiderqwen/qwen2.5-coder-1.5b-instructLocal/LAN model	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	IncompleteHistorical roster · retained for auditIncomplete or apparatus diagnosticblocked · see receipt for cause	Not measured	Not measured	Receipt ↗ Run details Model: local/lmstudio/qwen/qwen2.5-coder-1.5b-instruct Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: default Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Interrupted old-roster attempt. No terminal completion, wall measurement or independent scoring claimed. Captured token subtotals, if any, are lower bounds only. Gateway instance continuity was not proven; excluded from matched-model comparison. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Incomplete whole-attempt accounting; captured provider-call subtotals are lower bounds only.
verdict-codeanthropic/claude-haiku-5-5	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Passed20/20 checksEarlier attempt; superseded by a recorded later run	205.045 s	Not measured	Receipt ↗ Run details Model: byok/anthropic/claude-haiku-5-5 Observed native model: Not measured Gateway response model: byok/anthropic/claude-haiku-5-5 Reasoning setting: default Input / output tokens: 24,556 / 4,490 Total tokens: 29,046 Accounted tokens / wall-second: 141.657 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Provider cost estimation returned zero despite positive usage and catalog prices. Cost is unavailable; the separately labelled catalog estimate is not an observed charge.
aideropenai/gpt-6-luna	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Passed20/20 checksEarlier attempt; superseded by a recorded later run	21.078 s	$0.000832	Receipt ↗ Run details Model: byok/openai/gpt-6-luna Observed native model: Not measured Gateway response model: byok/openai/gpt-6-luna Reasoning setting: default Input / output tokens: 836 / 1,497 Total tokens: 2,333 Accounted tokens / wall-second: 110.684 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
verdict-codeopenai/gpt-6-luna	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Incomplete0/20 checks on partial artifactEarlier attempt; superseded by a recorded later runIncomplete or apparatus diagnosticblocked · see receipt for cause	3.788 s	Not measured	Receipt ↗ Run details Model: byok/openai/gpt-6-luna Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: default Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Incomplete whole-attempt accounting; captured provider-call subtotals are lower bounds only.
verdict-codeqwen/qwen3.8-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Incomplete20/20 checks on partial artifactEarlier attempt; superseded by a recorded later runIncomplete or apparatus diagnostictimeout · see receipt for cause	240.32 s	Not measured	Receipt ↗ Run details Model: byok/qwen/qwen3.8-flash Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: default Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Incomplete whole-attempt accounting; captured provider-call subtotals are lower bounds only.
verdict-codezai/glm-5.3-flashBudget API model	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Passed20/20 checksEarlier attempt; superseded by a recorded later run	84.191 s	$0.004326	Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: byok/zai/glm-5.3-flash Reasoning setting: low Input / output tokens: 20,417 / 2,526 Total tokens: 22,943 Accounted tokens / wall-second: 272.511 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
verdict-codeqwen/qwen3.8-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Incomplete20/20 checks on partial artifactEarlier attempt; superseded by a recorded later runIncomplete or apparatus diagnosticblocked · see receipt for cause	153.084 s	Not measured	Receipt ↗ Run details Model: byok/qwen/qwen3.8-flash Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: default Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Incomplete whole-attempt accounting; captured provider-call subtotals are lower bounds only.
verdict-codeqwen/qwen3.8-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Incomplete20/20 checks on partial artifactIncomplete or apparatus diagnostictimeout · see receipt for cause	240.628 s	Not measured	Receipt ↗ Run details Model: byok/qwen/qwen3.8-flash Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: default Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Incomplete whole-attempt accounting; captured provider-call subtotals are lower bounds only.
AntigravityNo model inferenceSubscription model	Repeat isolated native execution access preflightSynthetic test input	IncompleteIncomplete or apparatus diagnosticblocked · see receipt for cause	Not measured	Not measured	Receipt ↗ Run details Model: No inference Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: See invocation receipt Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Access preflight only; no model invocation and no coding failure. Installed CLI advertises editor operations, with no isolated headless coding-agent command. Native UI conversations were not disturbed. No measured inference wall time, tokens, billing or quality result.
verdict-codezai/glm-5.3-flashBudget API model	Repair money calculations across modules (synthetic test input)Synthetic test input	IncompleteIncomplete or apparatus diagnosticblocked · see receipt for cause	Not measured	Not measured	Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: See invocation receipt Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Gateway health preflight raised ConnectError [Errno61] Connection refused before agent/provider invocation. No coding quality result or spend inferred. This preflight diagnostic was reconstructed immediately after wrapper failure; elapsed time was not durably captured and remains unknown.
verdict-codezai/glm-5.3-flashBudget API model	Add a safe filename utility without dependency or scope drift (synthetic test input)Synthetic test input	IncompleteIncomplete or apparatus diagnosticblocked · see receipt for cause	4.061 s	Not measured	Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: See invocation receipt Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured CLI rejected a non-private 0755 instrumentation directory before inference. Corrected runner uses 0700. Configuration diagnostic, not a coding failure. Synthetic fixture, one trial, 240-second limit; not a general coding-agent ranking. 9 independent behavior unittest methods plus 1 changed-file scope check. Model and token accounting require captured provider/usage review; absent values remain unknown. Timeout/transport failure is infrastructure evidence, not a model quality failure.
verdict-codezai/glm-5.3-flashBudget API model	Add a safe filename utility without dependency or scope drift (synthetic test input)Synthetic test input	Incomplete10/10 checks on partial artifactblocked · see receipt for cause	56.813 s	$0.004217	Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: byok/zai/glm-5.3-flash Reasoning setting: See invocation receipt Input / output tokens: 25,815 / 691 Total tokens: 26,506 Accounted tokens / wall-second: 466.548 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Gateway lifecycle persistence failed for request 19908254-ebcc-4ad1-867e-7dfa65664468: terminal durability 500 ms deadline expired beside a 996 ms Gateway loop stall. No timeout or token-budget failure. Complete usage accounting survived. Generated code independently passes 9 behavior tests plus 1 scope check, but the CLI did not terminate successfully. Do not count as completed product PASS. Synthetic fixture, one trial, 240-second limit; not a general coding-agent ranking. 9 independent behavior unittest methods plus 1 changed-file scope check. Model and token accounting require captured provider/usage review; absent values remain unknown. Timeout/transport failure is infrastructure evidence, not a model quality failure. Complete Gateway terminal usage proof; cost is estimated BYOK USD, not a provider cash receipt.
verdict-codezai/glm-5.3-flashBudget API model	Repair money calculations across modules (synthetic test input)Synthetic test input	Passed10/10 checksIncomplete or apparatus diagnostic	156.373 s	$0.014683	Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: byok/zai/glm-5.3-flash Reasoning setting: See invocation receipt Input / output tokens: 83,930 / 4,190 Total tokens: 88,120 Accounted tokens / wall-second: 563.524 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Instrumentation diagnostic: candidate read provider traces inside workspace. Completion/code checks are real; timing and token usage are confounded. A mistaken interrupted reconstruction was corrected from original terminal stdout/journal; correction retained. Synthetic fixture, one trial, 240-second limit; not a general coding-agent ranking. 9 independent behavior unittest methods plus 1 changed-file scope check. Model and token accounting require captured provider/usage review; absent values remain unknown. Timeout/transport failure is infrastructure evidence, not a model quality failure. Complete Gateway terminal usage proof; cost is estimated BYOK USD, not a provider cash receipt.
verdict-codezai/glm-5.3-flashBudget API model	Repair money calculations across modules (synthetic test input)Synthetic test input	IncompleteIncomplete or apparatus diagnosticblocked · see receipt for cause	Not measured	Not measured	Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: See invocation receipt Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Gateway health preflight raised ConnectError [Errno61] Connection refused before agent/provider invocation. No coding quality result or spend inferred. This preflight diagnostic was reconstructed immediately after wrapper failure; elapsed time was not durably captured and remains unknown.
verdict-codezai/glm-5.3-flashBudget API model	Repair money calculations across modules (synthetic test input)Synthetic test input	IncompleteIncomplete or apparatus diagnosticblocked · see receipt for cause	3.768 s	Not measured	Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: See invocation receipt Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured CLI rejected a non-private 0755 instrumentation directory before inference. Corrected runner uses 0700. Configuration diagnostic, not a coding failure. Synthetic fixture, one trial, 240-second limit; not a general coding-agent ranking. 9 independent behavior unittest methods plus 1 changed-file scope check. Model and token accounting require captured provider/usage review; absent values remain unknown. Timeout/transport failure is infrastructure evidence, not a model quality failure.

Previous harness repair — all 30 new attempts preserved unchanged

The preceding parser campaign and its retained task families remain available with their original receipts and conclusions.

Previous harness result snapshot ↗ · Previous harness receipt manifest ↗

First corrected campaign — 58 records preserved unchanged

This earlier campaign used different sampling settings. Source fixes and settings both changed; scores are not pooled.

Prior result snapshot ↗ · Prior receipt manifest ↗

Original October 15 pilot — preserved unchanged

The original 57 records include the first runner setup, short time limits, interruptions and incomplete accounting. Reruns use a revised protocol and are not pooled with those results.

Original result snapshot ↗ · Original 57-receipt manifest ↗

How to read this

Passed means the independent task checks passed and the agent finished. Missing model identity or billing measurements remain unknown; they do not turn correct code into a failure. Incomplete runs do not establish a coding result.

Native agents use their own subscription models. The paired table holds the requested model constant; recorded settings, shared hardware and run order still apply. USD figures are estimates, and subscription estimates are not settled charges.

Download evidence manifest ↗

Current result snapshot ↗ · Model selection and added dates ↗

Unchanged frozen parser prompt, starter and hidden oracle; both actual CLIs. The user replaced the obsolete roster with five current inexpensive models, selected from live ladders and catalog added dates; historical runs remain in audit history. Every historical campaign and every new attempt remains available.
Both agents request temperature zero and retain fixed 240-second, per-request and cumulative token limits. GPT-6 Luna uses Responses with the same provider-default reasoning in both arms; its unsupported custom sampling is omitted upstream, as required by the vendor. No silent model substitution or hidden-check feedback.
Generic product changes remove redundant read and completion turns, bound repeated self-repair and provide an explicit text tool protocol for models without native calls. Protocol differences are disclosed in receipts.
The independent hidden scorer runs only after candidate execution. It supplies no examples, diagnostics or expected outputs to candidates or repair decisions.
Latest scheduled attempt is displayed even if worse or incomplete; no best-of selection. Causal apparatus retries remain in history.
Other coding and Blueprint families retain prior measured receipts; they were not rerun in this parser campaign.
Synthetic single-task fixtures, shared hardware and one trial per arm. No statistical or general superiority claim; canonical benchmark-store admission remains unverified.

Root-cause evidence ↗

Snapshot: 2026-10-16T13:22:22.547358+00:00
Source baseline: 75fb1c7a5473549983c16143e3cb15482c940d97