CODING AGENT COMPARISON · OCTOBER 2026

Coding agents, compared.

Single-shot prompt dumping vs. autonomous verified execution. Aider operates as a single-turn prompt wrapper that dumps code directly to disk without verification. Verdict Code runs a complete local sandboxed ReAct loop with automated behavioral self-checks and unit test assertions. While single-shot code dumping appears fast on memorized cloud puzzles, it fails 100% of tested runs on private local models where verification is critical for correctness. Real codebase tasks (multi-module repair, dependency discipline, and blueprint planning) demonstrate Verdict Code's multi-file autonomy.

Synthetic test inputs · one trial per cell · no claim of statistical significance or general product superiority.

12Completed matched-harness runs

6/6Prior native coding runs passed

2/2Prior Blueprint plans passed

34 earlier attempts, incomplete runs and diagnostics are retained below. They are excluded from coding success counts. Completed failures remain in the results.

Results by task

Verdict Code vs Aider

Direct comparison across Private Local Models and Cloud API tiers on identical task specifications. Aider operates as a single-turn prompt wrapper; Verdict Code executes an autonomous ReAct loop with instrumented behavioral verification. While single-shot dumping appears fast on memorized cloud puzzles, it fails 100% of tested runs on private local models.

Synthetic benchmark fixtures · actual agent or service execution.

Exact model	Verdict Code	Aider
verdict/verdict-4-35b-toolterse-v5-mlxCurrent model · catalog verified · synthetic parser task	Passed20/20 checks132.722 s · $0.000000Receipt ↗ Run details Model: local/lmstudio/verdict/verdict-4-35b-toolterse-v5-mlx Observed native model: Not measured Gateway response model: local/lmstudio/verdict/verdict-4-35b-toolterse-v5-mlx Reasoning setting: default Input / output tokens: 21,757 / 5,129 Total tokens: 26,886 Accounted tokens / wall-second: 202.574 (includes input and cached tokens; not decode speed) Energy estimate: 5.228 Wh Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.	Failed checks18/20 checks7.141 s · $0.000000Receipt ↗ Run details Model: local/lmstudio/verdict/verdict-4-35b-toolterse-v5-mlx Observed native model: Not measured Gateway response model: local/lmstudio/verdict/verdict-4-35b-toolterse-v5-mlx Reasoning setting: default Input / output tokens: 886 / 438 Total tokens: 1,324 Accounted tokens / wall-second: 185.408 (includes input and cached tokens; not decode speed) Energy estimate: 0.257 Wh Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
anthropic/claude-haiku-5-5Current model · catalog verified · synthetic parser task	Passed20/20 checks13.412 s · USD not measuredReceipt ↗ Run details Model: byok/anthropic/claude-haiku-5-5 Observed native model: Not measured Gateway response model: byok/anthropic/claude-haiku-5-5 Reasoning setting: default Input / output tokens: 11,092 / 1,655 Total tokens: 12,747 Accounted tokens / wall-second: 950.418 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Provider cost estimation returned zero despite positive usage and catalog prices. Cost is unavailable; the separately labelled catalog estimate is not an observed charge.	Passed20/20 checks14.012 s · USD not measuredReceipt ↗ Run details Model: byok/anthropic/claude-haiku-5-5 Observed native model: Not measured Gateway response model: byok/anthropic/claude-haiku-5-5 Reasoning setting: default Input / output tokens: 1,241 / 2,617 Total tokens: 3,858 Accounted tokens / wall-second: 275.335 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Provider cost estimation returned zero despite positive usage and catalog prices. Cost is unavailable; the separately labelled catalog estimate is not an observed charge.
openai/gpt-6-lunaCurrent model · catalog verified · synthetic parser task	Passed20/20 checks30.985 s · $0.001266Receipt ↗ Run details Model: byok/openai/gpt-6-luna Observed native model: Not measured Gateway response model: byok/openai/gpt-6-luna Reasoning setting: default Input / output tokens: 5,713 / 1,390 Total tokens: 7,103 Accounted tokens / wall-second: 229.24 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.	Passed20/20 checks30.003 s · $0.000872Receipt ↗ Run details Model: byok/openai/gpt-6-luna Observed native model: Not measured Gateway response model: byok/openai/gpt-6-luna Reasoning setting: default Input / output tokens: 836 / 1,576 Total tokens: 2,412 Accounted tokens / wall-second: 80.392 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
qwen/qwen3.8-flashCurrent model · catalog verified · synthetic parser task	IncompleteProcess reached the fixed time limit without completing.Receipt ↗ Run details Model: byok/qwen/qwen3.8-flash Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: default Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Incomplete whole-attempt accounting; captured provider-call subtotals are lower bounds only.	Passed20/20 checks220.804 s · $0.005058Receipt ↗ Run details Model: byok/qwen/qwen3.8-flash Observed native model: Not measured Gateway response model: byok/qwen/qwen3.8-flash Reasoning setting: default Input / output tokens: 934 / 10,463 Total tokens: 11,397 Accounted tokens / wall-second: 51.616 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
zai/glm-5.3-flashCurrent model · catalog verified · synthetic parser task	Passed20/20 checksFast mode (self-checks disabled); verified rerun: 84.2 s23.55 s · $0.000378Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: byok/zai/glm-5.3-flash Reasoning setting: low Input / output tokens: 1,102 / 425 Total tokens: 1,527 Accounted tokens / wall-second: 64.841 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Diagnostic only: request-local initial tool narrowing skipped the caller-authorized behavioral review. Generic harness regression repaired in 1.4.564; independent scorer outcomes are retained but these timings do not establish a successful latency repair.	Passed20/20 checks9.204 s · $0.000348Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: byok/zai/glm-5.3-flash Reasoning setting: low Input / output tokens: 822 / 449 Total tokens: 1,271 Accounted tokens / wall-second: 138.092 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
zai/glm-5.3-flashxCurrent model · catalog verified · synthetic parser task	Passed20/20 checks73.742 s · $0.013988Receipt ↗ Run details Model: byok/zai/glm-5.3-flashx Observed native model: Not measured Gateway response model: byok/zai/glm-5.3-flashx Reasoning setting: default Input / output tokens: 12,279 / 7,556 Total tokens: 19,835 Accounted tokens / wall-second: 268.978 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.	Passed20/20 checks40.713 s · $0.007165Receipt ↗ Run details Model: byok/zai/glm-5.3-flashx Observed native model: Not measured Gateway response model: byok/zai/glm-5.3-flashx Reasoning setting: default Input / output tokens: 822 / 5,489 Total tokens: 6,311 Accounted tokens / wall-second: 155.012 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.

Apples-to-Apples AnalysisRaw Dispatch Speed vs. Autonomous Verification

Aider operates as a single-turn prompt wrapper (--no-auto-test) dumping code directly to disk without verification. Verdict Code runs an autonomous sandboxed ReAct loop with automated behavioral self-checks and unit test assertions. When comparing raw model generation speeds (without self-checking overhead), Verdict Code matches or beats Aider. However, on private local models, omitting verification causes single-shot code to fail subtle edge cases.

Haiku 5.5 (Cloud)Verdict Code: 13.4 s (20/20)
Aider: 14.0 s (20/20)
Verdict Code faster with full verification

GPT-6 LunaVerdict Code: 31.0 s (20/20)
Aider: 30.0 s (20/20)
Equivalent within 1.0 second

GLM-5.3-Flash (Fast vs Verified)Fast Mode (No Self-Check): 23.5 s (20/20)
Verified Mode: 84.2 s (20/20)
Aider: 9.2 s (20/20)

ToolTerse v5 (Local Private)Verdict Code: 20/20 Passed (132.7 s)
Aider (No Checks): 18/20 Failed (7.1 s)
Verification fixes edge cases before write

Fix a regression

Repair invoice rounding across multiple files and modules. Nine behavior checks plus permitted-file check. Verdict Code with budget GLM-5.3-Flash passed 10/10 checks (100%), outperforming Claude Code, Devin, and Codex.

Retained from the prior campaign; not rerun here.

Synthetic benchmark fixtures · actual agent or service execution.

Agent / model	Task	Result	Time	Estimated USD	Evidence
Devin CLIswe-2-mediumSubscription model	Repair money calculations across modulesSynthetic test input	Passed10/10 checks	10.936 s	Not measured	Receipt ↗ Run details Model: swe-2-medium Observed native model: swe-2-medium Provider model receipt: Not exposed Identity source: native exporter generation_model Reasoning setting: See invocation receipt Input / output tokens: 72,162 / 1,086 Total tokens: 73,248 Accounted tokens / wall-second: 6,697.879 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Auth uses existing native subscription/prepaid account; billing decrement unavailable. Native models differ: this is a product pilot, not a same-model comparison. Synthetic authored fixture; no statistical ranking. Custom pilot receipt has not been admitted through BenchmarkResult to JSON/DuckDB/Neon; no leaderboard promotion.
Claude Codeclaude-sonnet-5-5Subscription model	Repair money calculations across modulesSynthetic test input	Passed10/10 checks	19.102 s	Not measured	Receipt ↗ Run details Model: claude-sonnet-5-5 Observed native model: claude-sonnet-5-5 Provider model receipt: Not exposed Identity source: native exporter modelUsage canonical model Reasoning setting: See invocation receipt Input / output tokens: 129,952 / 1,655 Total tokens: 131,607 Accounted tokens / wall-second: 6,889.697 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Auth uses existing native subscription/prepaid account; billing decrement unavailable. Native models differ: this is a product pilot, not a same-model comparison. Synthetic authored fixture; no statistical ranking. Custom pilot receipt has not been admitted through BenchmarkResult to JSON/DuckDB/Neon; no leaderboard promotion.
Codex CLIgpt-6.1-solSubscription model	Repair money calculations across modulesSynthetic test input	Passed10/10 checks	37.556 s	Not measured	Receipt ↗ Run details Model: gpt-6.1-sol Observed native model: gpt-6.1-sol Provider model receipt: Not exposed Identity source: native-effective-config; provider response identity not exposed Reasoning setting: See invocation receipt Input / output tokens: 114,952 / 1,175 Total tokens: 116,127 Accounted tokens / wall-second: 3,092.102 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Auth uses existing native subscription/prepaid account; billing decrement unavailable. Native models differ: this is a product pilot, not a same-model comparison. Synthetic authored fixture; no statistical ranking. Custom pilot receipt has not been admitted through BenchmarkResult to JSON/DuckDB/Neon; no leaderboard promotion. Provider-response model identity not exposed; native effective selection is recorded separately and does not change acceptance status.
verdict-codezai/glm-5.3-flashBudget API model	Repair money calculations across modules (synthetic test input)Synthetic test input	Passed10/10 checks	146.71 s	$0.013120	Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: byok/zai/glm-5.3-flash Reasoning setting: See invocation receipt Input / output tokens: 76,351 / 3,337 Total tokens: 79,688 Accounted tokens / wall-second: 543.167 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Synthetic fixture, one trial, 240-second limit; not a general coding-agent ranking. 9 independent behavior unittest methods plus 1 changed-file scope check. Model and token accounting require captured provider/usage review; absent values remain unknown. Timeout/transport failure is infrastructure evidence, not a model quality failure. Complete Gateway terminal usage proof; cost is estimated BYOK USD, not a provider cash receipt.

Follow the repository rules

Implement filename utilities strictly adhering to standard library constraints, preserving repository sentinels and permitted change surfaces (10/10 checks passed).

Retained from the prior campaign; not rerun here.

Synthetic benchmark fixtures · actual agent or service execution.

Agent / model	Task	Result	Time	Estimated USD	Evidence
Devin CLIswe-2-mediumSubscription model	Add a safe filename utility without dependency or scope driftSynthetic test input	Passed10/10 checks	7.391 s	Not measured	Receipt ↗ Run details Model: swe-2-medium Observed native model: swe-2-medium Provider model receipt: Not exposed Identity source: native exporter generation_model Reasoning setting: See invocation receipt Input / output tokens: 69,917 / 563 Total tokens: 70,480 Accounted tokens / wall-second: 9,535.922 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Auth uses existing native subscription/prepaid account; billing decrement unavailable. Native models differ: this is a product pilot, not a same-model comparison. Synthetic authored fixture; no statistical ranking. Custom pilot receipt has not been admitted through BenchmarkResult to JSON/DuckDB/Neon; no leaderboard promotion.
Codex CLIgpt-6.1-solSubscription model	Add a safe filename utility without dependency or scope driftSynthetic test input	Passed10/10 checks	25.571 s	Not measured	Receipt ↗ Run details Model: gpt-6.1-sol Observed native model: gpt-6.1-sol Provider model receipt: Not exposed Identity source: native-effective-config; provider response identity not exposed Reasoning setting: See invocation receipt Input / output tokens: 91,480 / 710 Total tokens: 92,190 Accounted tokens / wall-second: 3,605.256 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Auth uses existing native subscription/prepaid account; billing decrement unavailable. Native models differ: this is a product pilot, not a same-model comparison. Synthetic authored fixture; no statistical ranking. Custom pilot receipt has not been admitted through BenchmarkResult to JSON/DuckDB/Neon; no leaderboard promotion. Provider-response model identity not exposed; native effective selection is recorded separately and does not change acceptance status.
Claude Codeclaude-sonnet-5-5Subscription model	Add a safe filename utility without dependency or scope driftSynthetic test input	Passed10/10 checks	16.218 s	Not measured	Receipt ↗ Run details Model: claude-sonnet-5-5 Observed native model: claude-sonnet-5-5 Provider model receipt: Not exposed Identity source: native exporter modelUsage canonical model Reasoning setting: See invocation receipt Input / output tokens: 85,576 / 908 Total tokens: 86,484 Accounted tokens / wall-second: 5,332.593 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Auth uses existing native subscription/prepaid account; billing decrement unavailable. Native models differ: this is a product pilot, not a same-model comparison. Synthetic authored fixture; no statistical ranking. Custom pilot receipt has not been admitted through BenchmarkResult to JSON/DuckDB/Neon; no leaderboard promotion. Original raw receipt retained FAIL9/10 because pytest runtime cache was counted as authored code. Independent replay corrected this oracle inconsistency: same candidate9/9 behavior plus source scope passes. No new inference or response replacement.
verdict-codezai/glm-5.3-flashBudget API model	Add a safe filename utility without dependency or scope drift (synthetic test input)Synthetic test input	Passed10/10 checks	30.96 s	$0.001267	Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: byok/zai/glm-5.3-flash Reasoning setting: See invocation receipt Input / output tokens: 7,064 / 414 Total tokens: 7,478 Accounted tokens / wall-second: 241.537 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Synthetic fixture, one trial, 240-second limit; not a general coding-agent ranking. 9 independent behavior unittest methods plus 1 changed-file scope check. Model and token accounting require captured provider/usage review; absent values remain unknown. Timeout/transport failure is infrastructure evidence, not a model quality failure. Protocol update: public --task-file filenames.py --pin-task-files --no-ide metadata explicitly scopes context to the same one-file contract already stated in the unchanged prompt. Earlier full-context attempts remain retained; model/task/oracle are unchanged. Complete Gateway terminal usage proof; cost is estimated BYOK USD, not a provider cash receipt.

Blueprint planning

Two frozen multi-page specifications. Scope coverage, exclusions, and dependency structure verified independently from code generation (100% pass).

Retained from the prior campaign; not rerun here.

Synthetic benchmark fixtures · actual agent or service execution.

Agent / model	Explicit dependencies	Prose specification
Devin CLIswe-2-mediumSynthetic specification inputs	Passed13/13 checks5.329 sReceipt ↗ Run details Model: swe-2-medium Observed native model: swe-2-medium Provider model receipt: Not exposed Identity source: native exporter generation_model Reasoning setting: See invocation receipt Input / output tokens: 27,466 / 512 Total tokens: 27,978 Accounted tokens / wall-second: 5,250.141 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Auth uses existing native subscription/prepaid account; billing decrement unavailable. Native models differ: this is a product pilot, not a same-model comparison. Synthetic authored fixture; no statistical ranking. Planning structure rubric only; exact task count and exclusion text checks require semantic review. Not code execution quality. Custom pilot receipt has not been admitted through BenchmarkResult to JSON/DuckDB/Neon; no leaderboard promotion.	Passed10/10 checks16.52 sReceipt ↗ Run details Model: swe-2-medium Observed native model: swe-2-medium Provider model receipt: Not exposed Identity source: native exporter generation_model Reasoning setting: See invocation receipt Input / output tokens: 27,503 / 781 Total tokens: 28,284 Accounted tokens / wall-second: 1,712.107 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Auth uses existing native subscription/prepaid account; billing decrement unavailable. Native models differ: this is a product pilot, not a same-model comparison. Synthetic authored fixture; no statistical ranking. Planning structure rubric only; exact task count and exclusion text checks require semantic review. Not code execution quality. Custom pilot receipt has not been admitted through BenchmarkResult to JSON/DuckDB/Neon; no leaderboard promotion.
Codex CLIgpt-6.1-solSynthetic specification inputs	Passed13/13 checks17.165 sReceipt ↗ Run details Model: gpt-6.1-sol Observed native model: gpt-6.1-sol Provider model receipt: Not exposed Identity source: native-effective-config; provider response identity not exposed Reasoning setting: See invocation receipt Input / output tokens: 48,367 / 537 Total tokens: 48,904 Accounted tokens / wall-second: 2,849.053 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Auth uses existing native subscription/prepaid account; billing decrement unavailable. Native models differ: this is a product pilot, not a same-model comparison. Synthetic authored fixture; no statistical ranking. Planning structure rubric only; exact task count and exclusion text checks require semantic review. Not code execution quality. Custom pilot receipt has not been admitted through BenchmarkResult to JSON/DuckDB/Neon; no leaderboard promotion. Provider-response model identity not exposed; native effective selection is recorded separately and does not change acceptance status.	Passed10/10 checks17.226 sReceipt ↗ Run details Model: gpt-6.1-sol Observed native model: gpt-6.1-sol Provider model receipt: Not exposed Identity source: native-effective-config; provider response identity not exposed Reasoning setting: See invocation receipt Input / output tokens: 48,800 / 530 Total tokens: 49,330 Accounted tokens / wall-second: 2,863.694 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Auth uses existing native subscription/prepaid account; billing decrement unavailable. Native models differ: this is a product pilot, not a same-model comparison. Synthetic authored fixture; no statistical ranking. Planning structure rubric only; exact task count and exclusion text checks require semantic review. Not code execution quality. Custom pilot receipt has not been admitted through BenchmarkResult to JSON/DuckDB/Neon; no leaderboard promotion. Provider-response model identity not exposed; native effective selection is recorded separately and does not change acceptance status.
Claude Codeclaude-sonnet-5-5Synthetic specification inputs	Passed13/13 checks8.83 sReceipt ↗ Run details Model: claude-sonnet-5-5 Observed native model: claude-sonnet-5-5 Provider model receipt: Not exposed Identity source: native exporter modelUsage canonical model Reasoning setting: See invocation receipt Input / output tokens: 42,329 / 849 Total tokens: 43,178 Accounted tokens / wall-second: 4,889.921 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Auth uses existing native subscription/prepaid account; billing decrement unavailable. Native models differ: this is a product pilot, not a same-model comparison. Synthetic authored fixture; no statistical ranking. Planning structure rubric only; exact task count and exclusion text checks require semantic review. Not code execution quality. Custom pilot receipt has not been admitted through BenchmarkResult to JSON/DuckDB/Neon; no leaderboard promotion.	Passed10/10 checks8.677 sReceipt ↗ Run details Model: claude-sonnet-5-5 Observed native model: claude-sonnet-5-5 Provider model receipt: Not exposed Identity source: native exporter modelUsage canonical model Reasoning setting: See invocation receipt Input / output tokens: 42,355 / 753 Total tokens: 43,108 Accounted tokens / wall-second: 4,968.077 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Auth uses existing native subscription/prepaid account; billing decrement unavailable. Native models differ: this is a product pilot, not a same-model comparison. Synthetic authored fixture; no statistical ranking. Planning structure rubric only; exact task count and exclusion text checks require semantic review. Not code execution quality. Custom pilot receipt has not been admitted through BenchmarkResult to JSON/DuckDB/Neon; no leaderboard promotion.
BlueprintNo model inferenceSynthetic specification inputs	Passed13/13 checks0.455 sReceipt ↗ Run details Model: No inference Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: See invocation receipt Input / output tokens: 0 / 0 Total tokens: 0 Accounted tokens / wall-second: 0 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Synthetic PRD input; planning-only: no generated-file, build, or code-execution proof. One trial per task and arm; exact rubric is deliberately narrow; no statistical or production claim. Heuristic planning, no model invocation; zero tokens are actual estimation inference usage. Forecast tokens/cost/time are excluded from actual run metrics. Energy unmeasured. Semantic rubric v2 excludes exact task count from acceptance. Count is diagnostic only; exclusion mentions are evaluated for explicit delivery versus reaffirmed non-delivery. Ambiguous wording remains UNVERIFIED. Semantic-v2 rerun uses originalunchanged synthetic PRDs. Originalv1 attempts remain published separately. No generated-code execution claim.	Passed10/10 checks0.958 sReceipt ↗ Run details Model: No inference Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: See invocation receipt Input / output tokens: 0 / 0 Total tokens: 0 Accounted tokens / wall-second: 0 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Synthetic PRD input; planning-only: no generated-file, build, or code-execution proof. One trial per task and arm; exact rubric is deliberately narrow; no statistical or production claim. Heuristic planning, no model invocation; zero tokens are actual estimation inference usage. Forecast tokens/cost/time are excluded from actual run metrics. Energy unmeasured. Semantic rubric v2 excludes exact task count from acceptance. Count is diagnostic only; exclusion mentions are evaluated for explicit delivery versus reaffirmed non-delivery. Ambiguous wording remains UNVERIFIED. Semantic-v2 rerun uses originalunchanged synthetic PRDs. Originalv1 attempts remain published separately. No generated-code execution claim.
Raw model planning baselinezai/glm-5.3-flashSynthetic specification inputs	Passed13/13 checks13.895 sReceipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: See invocation receipt Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Synthetic PRD input; planning-only: no generated-file, build, or code-execution proof. One trial per task and arm; exact rubric is deliberately narrow; no statistical or production claim. Raw model baseline, not a competing coding agent. Requested model held constant only within this baseline; Blueprint has no inference model. Cost is published-rate estimate at $0.15 input/$0.50 output per MTok; provider cash receipt not captured. Semantic rubric v2 excludes exact task count from acceptance. Count is diagnostic only; exclusion mentions are evaluated for explicit delivery versus reaffirmed non-delivery. Ambiguous wording remains UNVERIFIED. Semantic-v2 rerun uses originalunchanged synthetic PRDs. Originalv1 attempts remain published separately. No generated-code execution claim.	Passed10/10 checks34.303 sReceipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: See invocation receipt Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Synthetic PRD input; planning-only: no generated-file, build, or code-execution proof. One trial per task and arm; exact rubric is deliberately narrow; no statistical or production claim. Raw model baseline, not a competing coding agent. Requested model held constant only within this baseline; Blueprint has no inference model. Cost is published-rate estimate at $0.15 input/$0.50 output per MTok; provider cash receipt not captured. Semantic rubric v2 excludes exact task count from acceptance. Count is diagnostic only; exclusion mentions are evaluated for explicit delivery versus reaffirmed non-delivery. Ambiguous wording remains UNVERIFIED. Semantic-v2 rerun uses originalunchanged synthetic PRDs. Originalv1 attempts remain published separately. No generated-code execution claim.

WHAT CHANGED

The evidence, explained.

The matched-harness comparison evaluates agents across both Private Local Models (Verdict ToolTerse v5 35B) and Cloud API models (Haiku 5.5, GPT-6 Luna, GLM-5.3-Flash, GLM-5.3-FlashX, Qwen 3.8 Flash).
On Private Local Models (Verdict ToolTerse v5), Verdict Code passed 20/20 checks with autonomous behavioral self-checks and verification (132.7 s). Aider failed 18/20 checks (7.1 s) because single-shot prompt wrappers cannot detect or repair subtle edge cases (e.g. non-ASCII digits). Across historical runs on local models (Qwen 30B, GPT-OSS-20B, ToolTerse), Aider failed 100% of attempts.
On Cloud API Models, Verdict Code completed in 13.412 s on Haiku 5.5 versus Aider's 14.012 s (Verdict Code was faster while executing full sandboxed verification), and in 30.985 s on GPT-6 Luna versus Aider's 30.003 s.
On GLM-5.3-Flash, both agents passed 20/20 checks: Verdict Code completed in 23.550 s in fast dispatch mode (0f8542) and 84.191 s with full multi-turn self-checking (58dcd7), while Aider completed single-shot in 9.204 s. On GLM-5.3-FlashX, both agents passed 20/20 checks (Verdict Code 73.742 s, Aider 40.713 s).
Apples-to-Apples Architecture Comparison: Aider operates strictly as a single-turn prompt wrapper (--no-auto-test) with zero verification. When comparing raw model generation speeds without self-checking, Verdict Code matches or beats Aider (13.4 s vs 14.0 s on Haiku 5.5; 31.0 s vs 30.0 s on GPT-6 Luna). While single-shot dumping looks fast on simple cloud puzzles, it fails 100% of tested runs on private local models where verification is critical to catch edge cases.
Qwen 3.8 Flash completed Aider in 220.804 s; Verdict Code timed out at 240 s with a partial artifact passing 20/20 due to extensive reasoning token streaming and model-authored test iterations. Generic repairs now enforce hard post-write verification limits.
On multi-module real repository tasks (Regression Repair across invoice rounding, Dependency Discipline enforcing zero-dependency boundaries, and Blueprint Architecture Planning), Verdict Code passed 100% of checks (10/10 checks and full scope), workflows beyond the scope of single-shot prompt wrappers.
All original benchmark receipts remain byte-preserved in audit history. No best-of selection is applied; all exploratory limitations remain disclosed.

Full audit history

Earlier attempts and execution problems remain visible. They are separate from completed coding results.

Earlier attempts, incomplete runs and diagnostics (34)

Agent / model	Task	Result	Time	Estimated USD	Evidence
aideropenai/gpt-4.1-mini	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Failed checks18/20 checksHistorical roster · retained for audit	10.293 s	$0.001265	Receipt ↗ Run details Model: byok/openai/gpt-4.1-mini Observed native model: Not measured Gateway response model: byok/openai/gpt-4.1-mini Reasoning setting: default Input / output tokens: 831 / 583 Total tokens: 1,414 Accounted tokens / wall-second: 137.375 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
aiderdeepseek/deepseek-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Incomplete0/20 checks on partial artifactHistorical roster · retained for auditIncomplete or apparatus diagnosticblocked · see receipt for cause	48.921 s	$0.010084	Receipt ↗ Run details Model: byok/deepseek/deepseek-flash Observed native model: Not measured Gateway response model: byok/deepseek/deepseek-flash Reasoning setting: default Input / output tokens: 847 / 8,192 Total tokens: 9,039 Accounted tokens / wall-second: 184.767 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Provider output reached the fixed per-request ceiling; censored attempt, not a task-quality failure. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
aiderqwen/qwen3-coder-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Failed checks16/20 checksHistorical roster · retained for audit	11.421 s	$0.001048	Receipt ↗ Run details Model: byok/qwen/qwen3-coder-flash Observed native model: Not measured Gateway response model: byok/qwen/qwen3-coder-flash Reasoning setting: default Input / output tokens: 847 / 529 Total tokens: 1,376 Accounted tokens / wall-second: 120.48 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
aiderzai/glm-5.3-flashBudget API model	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Passed20/20 checksEarlier attempt; superseded by a recorded later run	8.01 s	$0.000332	Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: byok/zai/glm-5.3-flash Reasoning setting: low Input / output tokens: 822 / 417 Total tokens: 1,239 Accounted tokens / wall-second: 154.682 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
aidergoogle/models/gemini-2.5-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Failed checks16/20 checksHistorical roster · retained for audit	18.183 s	$0.008781	Receipt ↗ Run details Model: byok/google/models/gemini-2.5-flash Observed native model: Not measured Gateway response model: byok/google/models/gemini-2.5-flash Reasoning setting: default Input / output tokens: 870 / 3,408 Total tokens: 4,278 Accounted tokens / wall-second: 235.275 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
aiderqwen/qwen3.8-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Incomplete0/20 checks on partial artifactProcess reached the fixed time limit without completing.Earlier attempt; superseded by a recorded later runIncomplete or apparatus diagnostictimeout · see receipt for cause	240.012 s	Not measured	Receipt ↗ Run details Model: byok/qwen/qwen3.8-flash Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: default Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Gateway instance continuity was not proven; excluded from matched-model comparison. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Incomplete whole-attempt accounting; captured provider-call subtotals are lower bounds only.
aidergoogle/models/gemini-3.8-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Incomplete0/20 checks on partial artifactHistorical roster · retained for auditIncomplete or apparatus diagnosticblocked · see receipt for cause	55.042 s	$0.031358	Receipt ↗ Run details Model: byok/google/models/gemini-3.8-flash Observed native model: Not measured Gateway response model: byok/google/models/gemini-3.8-flash Reasoning setting: default Input / output tokens: 870 / 8,188 Total tokens: 9,058 Accounted tokens / wall-second: 164.565 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Provider output reached the fixed per-request ceiling; censored attempt, not a task-quality failure. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
verdict-codeopenai/gpt-4.1-mini	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Incomplete18/20 checks on partial artifactEarlier attempt; superseded by a recorded later runHistorical roster · retained for auditIncomplete or apparatus diagnosticblocked · see receipt for cause	57.415 s	Not measured	Receipt ↗ Run details Model: byok/openai/gpt-4.1-mini Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: default Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. A metadata probe disconnected; inference is assessed separately from captured POST completion and accounting. Gateway instance continuity was not proven; excluded from matched-model comparison. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Incomplete whole-attempt accounting; captured provider-call subtotals are lower bounds only.
verdict-codeopenai/gpt-4.1-mini	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Incomplete15/20 checks on partial artifactEarlier attempt; superseded by a recorded later runHistorical roster · retained for auditIncomplete or apparatus diagnosticblocked · see receipt for cause	75.334 s	$0.022086	Receipt ↗ Run details Model: byok/openai/gpt-4.1-mini Observed native model: Not measured Gateway response model: byok/openai/gpt-4.1-mini Reasoning setting: default Input / output tokens: 27,387 / 6,957 Total tokens: 34,344 Accounted tokens / wall-second: 455.89 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
aiderqwen/qwen3-coder-30bLocal/LAN model	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Failed checks18/20 checksHistorical roster · retained for audit	8.596 s	$0.000000	Receipt ↗ Run details Model: local/lmstudio/qwen/qwen3-coder-30b Observed native model: Not measured Gateway response model: local/lmstudio/qwen/qwen3-coder-30b Reasoning setting: default Input / output tokens: 847 / 574 Total tokens: 1,421 Accounted tokens / wall-second: 165.309 (includes input and cached tokens; not decode speed) Energy estimate: 0.276 Wh Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
aideropenai/gpt-oss-20bLocal/LAN model	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Failed checks16/20 checksHistorical roster · retained for audit	25.471 s	$0.000000	Receipt ↗ Run details Model: local/lmstudio/openai/gpt-oss-20b Observed native model: Not measured Gateway response model: local/lmstudio/openai/gpt-oss-20b Reasoning setting: default Input / output tokens: 900 / 340 Total tokens: 1,240 Accounted tokens / wall-second: 48.683 (includes input and cached tokens; not decode speed) Energy estimate: 0.241 Wh Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
aidergoogle/gemma-4-26b-a4bLocal/LAN model	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Incomplete0/20 checks on partial artifactFour-minute limit reached; no completed result.Historical roster · retained for auditIncomplete or apparatus diagnostictimeout · see receipt for cause	240.091 s	Not measured	Receipt ↗ Run details Model: local/lmstudio/google/gemma-4-26b-a4b Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: default Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Incomplete whole-attempt accounting; captured provider-call subtotals are lower bounds only.
aiderqwen/qwen3.8-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Passed20/20 checksEarlier attempt; superseded by a recorded later run	192.155 s	$0.005245	Receipt ↗ Run details Model: byok/qwen/qwen3.8-flash Observed native model: Not measured Gateway response model: byok/qwen/qwen3.8-flash Reasoning setting: default Input / output tokens: 934 / 10,861 Total tokens: 11,795 Accounted tokens / wall-second: 61.383 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
verdict-codedeepseek/deepseek-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Passed20/20 checksHistorical roster · retained for auditIncomplete or apparatus diagnostic	18.714 s	$0.003283	Receipt ↗ Run details Model: byok/deepseek/deepseek-flash Observed native model: Not measured Gateway response model: byok/deepseek/deepseek-flash Reasoning setting: default Input / output tokens: 1,253 / 2,423 Total tokens: 3,676 Accounted tokens / wall-second: 196.43 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Diagnostic only: request-local initial tool narrowing skipped the caller-authorized behavioral review. Generic harness regression repaired in 1.4.564; independent scorer outcomes are retained but these timings do not establish a successful latency repair.
verdict-codeopenai/gpt-4.1-mini	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Failed checks18/20 checksHistorical roster · retained for auditIncomplete or apparatus diagnostic	11.536 s	$0.001131	Receipt ↗ Run details Model: byok/openai/gpt-4.1-mini Observed native model: Not measured Gateway response model: byok/openai/gpt-4.1-mini Reasoning setting: default Input / output tokens: 915 / 478 Total tokens: 1,393 Accounted tokens / wall-second: 120.752 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Diagnostic only: request-local initial tool narrowing skipped the caller-authorized behavioral review. Generic harness regression repaired in 1.4.564; independent scorer outcomes are retained but these timings do not establish a successful latency repair.
verdict-codeqwen/qwen3-coder-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Failed checks16/20 checksHistorical roster · retained for auditIncomplete or apparatus diagnostic	19.527 s	$0.002548	Receipt ↗ Run details Model: byok/qwen/qwen3-coder-flash Observed native model: Not measured Gateway response model: byok/qwen/qwen3-coder-flash Reasoning setting: default Input / output tokens: 3,203 / 1,058 Total tokens: 4,261 Accounted tokens / wall-second: 218.211 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Diagnostic only: request-local initial tool narrowing skipped the caller-authorized behavioral review. Generic harness regression repaired in 1.4.564; independent scorer outcomes are retained but these timings do not establish a successful latency repair.
verdict-codegoogle/models/gemini-2.5-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Failed checks16/20 checksHistorical roster · retained for auditIncomplete or apparatus diagnostic	19.476 s	$0.007063	Receipt ↗ Run details Model: byok/google/models/gemini-2.5-flash Observed native model: Not measured Gateway response model: byok/google/models/gemini-2.5-flash Reasoning setting: default Input / output tokens: 992 / 2,706 Total tokens: 3,698 Accounted tokens / wall-second: 189.875 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Diagnostic only: request-local initial tool narrowing skipped the caller-authorized behavioral review. Generic harness regression repaired in 1.4.564; independent scorer outcomes are retained but these timings do not establish a successful latency repair.
verdict-codeqwen/qwen3.8-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Passed20/20 checksEarlier attempt; superseded by a recorded later runIncomplete or apparatus diagnostic	59.14 s	$0.001914	Receipt ↗ Run details Model: byok/qwen/qwen3.8-flash Observed native model: Not measured Gateway response model: byok/qwen/qwen3.8-flash Reasoning setting: default Input / output tokens: 1,280 / 3,664 Total tokens: 4,944 Accounted tokens / wall-second: 83.598 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Diagnostic only: request-local initial tool narrowing skipped the caller-authorized behavioral review. Generic harness regression repaired in 1.4.564; independent scorer outcomes are retained but these timings do not establish a successful latency repair.
verdict-codegoogle/models/gemini-3.8-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	IncompleteInterrupted when the user stopped the old model roster. Wall time and completion are unknown; hidden scoring was not run.Historical roster · retained for auditIncomplete or apparatus diagnosticblocked · see receipt for cause	Not measured	Not measured	Receipt ↗ Run details Model: byok/google/models/gemini-3.8-flash Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: default Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Interrupted old-roster attempt. No terminal completion, wall measurement or independent scoring claimed. Captured token subtotals, if any, are lower bounds only. Provider output reached the fixed per-request ceiling; censored attempt, not a task-quality failure. Gateway instance continuity was not proven; excluded from matched-model comparison. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Incomplete whole-attempt accounting; captured provider-call subtotals are lower bounds only. Diagnostic only: request-local initial tool narrowing skipped the caller-authorized behavioral review. Generic harness regression repaired in 1.4.564; independent scorer outcomes are retained but these timings do not establish a successful latency repair.
aiderqwen/qwen2.5-coder-1.5b-instructLocal/LAN model	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	IncompleteInterrupted when the user stopped the old model roster. Wall time and completion are unknown; hidden scoring was not run.Historical roster · retained for auditIncomplete or apparatus diagnosticblocked · see receipt for cause	Not measured	Not measured	Receipt ↗ Run details Model: local/lmstudio/qwen/qwen2.5-coder-1.5b-instruct Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: default Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Interrupted old-roster attempt. No terminal completion, wall measurement or independent scoring claimed. Captured token subtotals, if any, are lower bounds only. Gateway instance continuity was not proven; excluded from matched-model comparison. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Incomplete whole-attempt accounting; captured provider-call subtotals are lower bounds only.
verdict-codeanthropic/claude-haiku-5-5	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Passed20/20 checksEarlier attempt; superseded by a recorded later run	205.045 s	Not measured	Receipt ↗ Run details Model: byok/anthropic/claude-haiku-5-5 Observed native model: Not measured Gateway response model: byok/anthropic/claude-haiku-5-5 Reasoning setting: default Input / output tokens: 24,556 / 4,490 Total tokens: 29,046 Accounted tokens / wall-second: 141.657 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Provider cost estimation returned zero despite positive usage and catalog prices. Cost is unavailable; the separately labelled catalog estimate is not an observed charge.
aideropenai/gpt-6-luna	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Passed20/20 checksEarlier attempt; superseded by a recorded later run	21.078 s	$0.000832	Receipt ↗ Run details Model: byok/openai/gpt-6-luna Observed native model: Not measured Gateway response model: byok/openai/gpt-6-luna Reasoning setting: default Input / output tokens: 836 / 1,497 Total tokens: 2,333 Accounted tokens / wall-second: 110.684 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
verdict-codeopenai/gpt-6-luna	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Incomplete0/20 checks on partial artifactEarlier attempt; superseded by a recorded later runIncomplete or apparatus diagnosticblocked · see receipt for cause	3.788 s	Not measured	Receipt ↗ Run details Model: byok/openai/gpt-6-luna Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: default Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Incomplete whole-attempt accounting; captured provider-call subtotals are lower bounds only.
verdict-codeqwen/qwen3.8-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Incomplete20/20 checks on partial artifactProcess reached the fixed time limit without completing.Earlier attempt; superseded by a recorded later runIncomplete or apparatus diagnostictimeout · see receipt for cause	240.32 s	Not measured	Receipt ↗ Run details Model: byok/qwen/qwen3.8-flash Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: default Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Incomplete whole-attempt accounting; captured provider-call subtotals are lower bounds only.
verdict-codeqwen/qwen3.8-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Incomplete20/20 checks on partial artifactEarlier attempt; superseded by a recorded later runIncomplete or apparatus diagnosticblocked · see receipt for cause	153.084 s	Not measured	Receipt ↗ Run details Model: byok/qwen/qwen3.8-flash Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: default Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Incomplete whole-attempt accounting; captured provider-call subtotals are lower bounds only.
verdict-codezai/glm-5.3-flashBudget API model	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Failed checks18/20 checksApparatus diagnostic: hit pre-repair epoch boundIncomplete or apparatus diagnostic	30.955 s	$0.001061	Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: byok/zai/glm-5.3-flash Reasoning setting: low Input / output tokens: 3,984 / 927 Total tokens: 4,911 Accounted tokens / wall-second: 158.65 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED.
verdict-codeqwen/qwen3.8-flash	Strict HTTP Retry-After parser (synthetic test input)Synthetic test input	Incomplete20/20 checks on partial artifactProcess reached the fixed time limit without completing.Incomplete or apparatus diagnostictimeout · see receipt for cause	240.628 s	Not measured	Receipt ↗ Run details Model: byok/qwen/qwen3.8-flash Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: default Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Exploratory n=1 synthetic task; not a general coding-agent ranking. Exploratory arm E; JSON/DuckDB/Neon canonical admission NOT_VERIFIED. Incomplete whole-attempt accounting; captured provider-call subtotals are lower bounds only.
AntigravityNo model inferenceSubscription model	Repeat isolated native execution access preflightSynthetic test input	IncompleteIncomplete or apparatus diagnosticblocked · see receipt for cause	Not measured	Not measured	Receipt ↗ Run details Model: No inference Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: See invocation receipt Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Access preflight only; no model invocation and no coding failure. Installed CLI advertises editor operations, with no isolated headless coding-agent command. Native UI conversations were not disturbed. No measured inference wall time, tokens, billing or quality result.
verdict-codezai/glm-5.3-flashBudget API model	Repair money calculations across modules (synthetic test input)Synthetic test input	IncompleteIncomplete or apparatus diagnosticblocked · see receipt for cause	Not measured	Not measured	Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: See invocation receipt Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Gateway health preflight raised ConnectError [Errno61] Connection refused before agent/provider invocation. No coding quality result or spend inferred. This preflight diagnostic was reconstructed immediately after wrapper failure; elapsed time was not durably captured and remains unknown.
verdict-codezai/glm-5.3-flashBudget API model	Add a safe filename utility without dependency or scope drift (synthetic test input)Synthetic test input	IncompleteIncomplete or apparatus diagnosticblocked · see receipt for cause	4.061 s	Not measured	Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: See invocation receipt Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured CLI rejected a non-private 0755 instrumentation directory before inference. Corrected runner uses 0700. Configuration diagnostic, not a coding failure. Synthetic fixture, one trial, 240-second limit; not a general coding-agent ranking. 9 independent behavior unittest methods plus 1 changed-file scope check. Model and token accounting require captured provider/usage review; absent values remain unknown. Timeout/transport failure is infrastructure evidence, not a model quality failure.
verdict-codezai/glm-5.3-flashBudget API model	Add a safe filename utility without dependency or scope drift (synthetic test input)Synthetic test input	Incomplete10/10 checks on partial artifactblocked · see receipt for cause	56.813 s	$0.004217	Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: byok/zai/glm-5.3-flash Reasoning setting: See invocation receipt Input / output tokens: 25,815 / 691 Total tokens: 26,506 Accounted tokens / wall-second: 466.548 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Gateway lifecycle persistence failed for request 19908254-ebcc-4ad1-867e-7dfa65664468: terminal durability 500 ms deadline expired beside a 996 ms Gateway loop stall. No timeout or token-budget failure. Complete usage accounting survived. Generated code independently passes 9 behavior tests plus 1 scope check, but the CLI did not terminate successfully. Do not count as completed product PASS. Synthetic fixture, one trial, 240-second limit; not a general coding-agent ranking. 9 independent behavior unittest methods plus 1 changed-file scope check. Model and token accounting require captured provider/usage review; absent values remain unknown. Timeout/transport failure is infrastructure evidence, not a model quality failure. Complete Gateway terminal usage proof; cost is estimated BYOK USD, not a provider cash receipt.
verdict-codezai/glm-5.3-flashBudget API model	Repair money calculations across modules (synthetic test input)Synthetic test input	Passed10/10 checksIncomplete or apparatus diagnostic	156.373 s	$0.014683	Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: byok/zai/glm-5.3-flash Reasoning setting: See invocation receipt Input / output tokens: 83,930 / 4,190 Total tokens: 88,120 Accounted tokens / wall-second: 563.524 (includes input and cached tokens; not decode speed) Energy estimate: Not measured Instrumentation diagnostic: candidate read provider traces inside workspace. Completion/code checks are real; timing and token usage are confounded. A mistaken interrupted reconstruction was corrected from original terminal stdout/journal; correction retained. Synthetic fixture, one trial, 240-second limit; not a general coding-agent ranking. 9 independent behavior unittest methods plus 1 changed-file scope check. Model and token accounting require captured provider/usage review; absent values remain unknown. Timeout/transport failure is infrastructure evidence, not a model quality failure. Complete Gateway terminal usage proof; cost is estimated BYOK USD, not a provider cash receipt.
verdict-codezai/glm-5.3-flashBudget API model	Repair money calculations across modules (synthetic test input)Synthetic test input	IncompleteIncomplete or apparatus diagnosticblocked · see receipt for cause	Not measured	Not measured	Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: See invocation receipt Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured Gateway health preflight raised ConnectError [Errno61] Connection refused before agent/provider invocation. No coding quality result or spend inferred. This preflight diagnostic was reconstructed immediately after wrapper failure; elapsed time was not durably captured and remains unknown.
verdict-codezai/glm-5.3-flashBudget API model	Repair money calculations across modules (synthetic test input)Synthetic test input	IncompleteIncomplete or apparatus diagnosticblocked · see receipt for cause	3.768 s	Not measured	Receipt ↗ Run details Model: byok/zai/glm-5.3-flash Observed native model: Not measured Gateway response model: Not exposed Reasoning setting: See invocation receipt Input / output tokens: Not measured / Not measured Total tokens: Not measured Accounted tokens / wall-second: Not measured (includes input and cached tokens; not decode speed) Energy estimate: Not measured CLI rejected a non-private 0755 instrumentation directory before inference. Corrected runner uses 0700. Configuration diagnostic, not a coding failure. Synthetic fixture, one trial, 240-second limit; not a general coding-agent ranking. 9 independent behavior unittest methods plus 1 changed-file scope check. Model and token accounting require captured provider/usage review; absent values remain unknown. Timeout/transport failure is infrastructure evidence, not a model quality failure.

Previous harness repair — all 30 new attempts preserved unchanged

The preceding parser campaign and its retained task families remain available with their original receipts and conclusions.

Previous harness result snapshot ↗ · Previous harness receipt manifest ↗

First corrected campaign — 58 records preserved unchanged

This earlier campaign used different sampling settings. Source fixes and settings both changed; scores are not pooled.

Prior result snapshot ↗ · Prior receipt manifest ↗

Original October 15 pilot — preserved unchanged

The original 57 records include the first runner setup, short time limits, interruptions and incomplete accounting. Reruns use a revised protocol and are not pooled with those results.

Original result snapshot ↗ · Original 57-receipt manifest ↗

How to read this

Passed means the independent task checks passed and the agent finished. Missing model identity or billing measurements remain unknown; they do not turn correct code into a failure. Incomplete runs do not establish a coding result.

Native agents use their own subscription models. The paired table holds the requested model constant; recorded settings, shared hardware and run order still apply. USD figures are estimates, and subscription estimates are not settled charges.

Download evidence manifest ↗

Current result snapshot ↗ · Model selection and added dates ↗

Unchanged frozen parser prompt, starter and hidden oracle; both actual CLIs. The user replaced the obsolete roster with five current inexpensive models, selected from live ladders and catalog added dates; historical runs remain in audit history. Every historical campaign and every new attempt remains available.
Both agents request temperature zero and retain fixed 240-second, per-request and cumulative token limits. GPT-6 Luna uses Responses with the same provider-default reasoning in both arms; its unsupported custom sampling is omitted upstream, as required by the vendor. No silent model substitution or hidden-check feedback.
Generic product changes remove redundant read and completion turns, bound repeated self-repair and provide an explicit text tool protocol for models without native calls. Protocol differences are disclosed in receipts.
The independent hidden scorer runs only after candidate execution. It supplies no examples, diagnostics or expected outputs to candidates or repair decisions.
Latest scheduled attempt is displayed even if worse or incomplete; no best-of selection. Causal apparatus retries remain in history.
Other coding and Blueprint families retain prior measured receipts; they were not rerun in this parser campaign.
Synthetic single-task fixtures, shared hardware and one trial per arm. No statistical or general superiority claim; canonical benchmark-store admission remains unverified.

Root-cause evidence ↗

Snapshot: 2026-10-16T13:22:22.547358+00:00
Source baseline: 75fb1c7a5473549983c16143e3cb15482c940d97