What we learned from 90 real coding decisions
In a local pilot, three rotations of the same choices selected the passing implementation in 89 of 90 tasks, compared with 85 of 90 for a single pass. Median scoring time increased from 97.7 ms to 291.4 ms. These measurements come from a public Python benchmark with deliberately constructed alternatives, using an existing 4-bit decider on an Apple M5 Max. They are early research evidence, not a production reliability or speed guarantee.
Primary Method Comparison
All 90 evaluation tasks used identical 3-way candidate choices (1 canonical passing implementation, 2 deterministic test-killed AST mutants). Confidence intervals represent Wilson 95% sampling uncertainty. Expected Calibration Error (ECE) reflects temperature-scaled alignment with ground truth.
| Method | Correct | Accuracy | 95% Wilson CI | Median | p95 | ECE |
|---|---|---|---|---|---|---|
Single pass (baseline) Single forward pass evaluating choices A, B, and C directly. | 85 / 90 | 94.4% | 87.6% – 97.6% | 97.7 ms | 150.9 ms | 5.60% |
Three cyclic rotations (ℤ₃) Three cyclic shifted passes; slot scores averaged across shifts. | 89 / 90 | 98.9% | 94.0% – 99.8% | 291.4 ms | 453.5 ms | 1.60% |
Pairwise tournament All 3 pairwise head-to-head comparisons resolved via Bradley-Terry MLE. | 87 / 90 | 96.7% | 90.7% – 98.9% | 485.4 ms | 747.5 ms | 2.57% |
Cross-Quantization Scorecard (BF16 down to 2-Bit)
1,440 Forwards / PrecisionAll quantization levels evaluated the exact same 90 heldout tasks executing 12 real forwards per task on the Apple M5 Max. Slot logit extraction was performed under identical prompt templates across all formats.
| Quant Level | Weights | Smoke | Single Pass | Cyclic (ℤ₃) | Tournament | Cyclic ECE | Single ms | Cyclic ms | Orbit Flips |
|---|---|---|---|---|---|---|---|---|---|
BF16 Unquantized 16-bit | 7.83 GB | PASS | 85 / 90 (94.4%) | 89 / 90 (98.9%) | 88 / 90 (97.8%) | 0.79% | 98.7 ms | 294.8 ms | 2 / 90 (2.2%) |
Q8 MLX affine 8-bit, g64 | 4.16 GB | PASS | 85 / 90 (94.4%) | 90 / 90 (100.0%) | 88 / 90 (97.8%) | 0.57% | 112.7 ms | 341.9 ms | 2 / 90 (2.2%) |
Q6 MLX affine 6-bit, g64 | 3.18 GB | PASS | 85 / 90 (94.4%) | 89 / 90 (98.9%) | 88 / 90 (97.8%) | 1.25% | 124.2 ms | 370.7 ms | 1 / 90 (1.1%) |
Q4 MLX affine 4-bit, g64 | 2.20 GB | PASS | 85 / 90 (94.4%) | 89 / 90 (98.9%) | 87 / 90 (96.7%) | 1.60% | 97.7 ms | 291.4 ms | 1 / 90 (1.1%) |
Q3 MLX affine 3-bit, g64 | 1.72 GB | PASS | 80 / 90 (88.9%) | 87 / 90 (96.7%) | 84 / 90 (93.3%) | 2.45% | 136.5 ms | 371.7 ms | 8 / 90 (8.9%) |
Q2 MLX affine 2-bit, g64 | 1.23 GB | FAIL | 32 / 90 (35.6%) | 20 / 90 (22.2%) | 24 / 90 (26.7%) | 11.51% | 112.6 ms | 334.9 ms | 55 / 90 (61.1%) |
4-Bit Parity (72% Memory Savings)
Exactly 0 prediction differences across all 90 tasks between BF16, Q8, Q6, and Q4 on single-pass (all 85/90, paired exact McNemar p = 1.0) and cyclic rotations (all 89/90, p = 1.0). Q4 cuts memory from 7.83 GB down to 2.20 GB and yields the fastest median latency (97.7 ms / 291.4 ms).
Q8 Reached 100% Discrimination
Q8 selected the passing implementation on 90 of 90 tasks (100.0%) under cyclic rotations and achieved the lowest holdout calibration error (0.57% ECE), correcting borderline cases while halving RAM consumption vs BF16 (4.16 GB vs 7.83 GB).
The Quantization Cliff (Q3 vs Q2)
Q3 shows measurable degradation (single pass drops to 80/90, rotation disagreements double). Q2 suffers catastrophic collapse: fails the [0, 0] answer smoke test, collapsing into option A bias with 35.6% accuracy (p = 3.4 × 10⁻²¹ vs BF16).
Experimental Scope & Limitations
Responsible reporting requires highlighting the empirical boundaries of this pilot:
Statistical Uncertainty
The apparent +4.4 percentage point lift for cyclic rotations represents five corrections and one regression over 90 tasks. The paired exact McNemar test yields p = 0.21875. This small sample does not establish a statistically significant or repeatable lift.
Latency & Target Boundaries
Every evaluated method missed the project's strict local median target of less than 35 ms. Timings measure a warm, in-memory model without network roundtrips, prefix-cache negotiation, or concurrent load. Cloud latency and SLA remain unmeasured.
Order Sensitivity & Bias
Cyclic permutations cancel positional bias in expectation algebraically. However, raw rotation winners differed on 10/90 tasks, and opposite cyclic-orbit winners differed on 1/90. These stress tests confirm order sensitivity remains; no zero-bias certificate is claimed.
Benchmark & Simulation Provenance
Public benchmark pretraining contamination is unknown. Mutated AST functions do not represent natural LLM candidate proposals. Earlier simulated figures (100% accuracy, 35 ms) have been fully superseded by this real pilot.
Diagnostic & Combined Workflows
Secondary policies tested to evaluate autonomous confidence gating and test execution probes.
| Workflow | Correct | Accuracy | 95% Wilson CI | Median | p95 | Role |
|---|---|---|---|---|---|---|
Tournament + frozen confidence gate Threshold frozen from calibration set (coverage 1.0; 0 probes executed). | 87 / 90 | 96.7% | 90.7% – 98.9% | 485.4 ms | 747.5 ms | Diagnostic |
Always-probe diagnostic Post-hoc test running 270 unit test assertion probes across candidates. | 87 / 90 | 96.7% | 90.7% – 98.9% | 571.0 ms | 829.9 ms | Diagnostic |
Model & Dataset Identity
Note: This evidence pertains exclusively to the Mapika/decider-4b v2.1 MLX conversion. It must not be confused with the newly distilled Qwen2.5-Coder-3B candidate, nor with the separately promoted v1-derived verdict-4-decision-4b / code20_s050 artifact.
Verdict Decision Local Fleet vs. Cloud Frontier SOTA
Architecture & EconomicsWhy run specialized local decision adjudicators instead of prompt-engineering cloud frontier LLMs? The architectural deltas below outline the fundamental differences in token bloat, economics, latency, and mathematical debiasing:
| Dimension | Frontier Cloud SOTA | Verdict Decision Local | Advantage |
|---|---|---|---|
| Adjudication Architecture | Generative LLMs (100B–1T+ params); output conversational rationale tokens before stating option. | Direct candidate slot logit scoring (0.8B–35B params); zero generated tokens, pure forward evaluation. | Zero token bloat: 100% deterministic slot probability extraction with no hallucinated reasoning drift. |
| Execution Venue & Privacy | Third-party cloud APIs; requires code snippet and prompt transmission across public WAN. | 100% resident in local Apple Silicon Unified Memory or local GPU/CPU. | Completely private, air-gapped, zero data egress. |
| Cost per 1M Decisions | $3.00 to $15.00+ per million tokens (prompt tokens + reasoning tokens + completion tokens). | $0.00 / free offline inference on local developer hardware. | Eliminates token billing entirely for high-frequency agentic triage and guardrail loops. |
| Decision Latency | 800 ms – 3,500 ms TTFT + streaming text generation. | 97.7 ms median single pass / 291.4 ms cyclic debiased on Apple M5 Max. | Up to 12x faster execution for tight agent repair and gating cycles. |
| Positional Option Bias | Known systemic first-choice / last-choice positional bias in generated completions. | Algebraically neutralized via ℤ₃ cyclic group rotation averaging. | 98.9%–100% debiased selection accuracy with holdout ECE down to 0.57%. |
| Memory Footprint | Datacenter server clusters (8x H100 / A100 GPUs). | 529 MB (0.8B Q4) to 2.20 GB (4B Q4) up to 4.16 GB (4B Q8). | Operates alongside IDEs, compilers, and dev servers on 16 GB–32 GB workstations. |
Local Verdict Decision Fleet Roster
Public Evidence Receipts
Download the allowlisted public metrics, checksums, and experimental methodology:
Looking for the Decision API interpretation?
See how cyclic permutation debiasing and round-robin Bradley-Terry tournaments are integrated into the Verdict Decision API for agentic routing.
Explore VerdictDecision Benchmarks