# Real Coding Decisions Local Pilot Methodology

**Date:** October 7, 2026  
**Venue:** Local Apple M5 Max, 128 GiB unified memory  
**Release Evidence:** `llm-tuning-lab` commit `9fb9a99` (runner checkpoint `64bda7d`)  
**Study ID:** `coding-pilot-2026-10-07`  
**Model:** `Mapika/decider-4b v2.1` MLX 4-bit conversion (Qwen3.5-4B-Base derivative, group size 64)  
**Weight SHA-256:** `d0ba3ca5953871030be574146a6eef2d8073d556041756647753921112afaa7f`  
**Dataset SHA-256:** `2f7994d4be5d3b80a040dde3422d1dd1dda2422536d3dd79409598769dbe87f9`  

---

## Abstract

This study evaluates the performance, latency, and positional bias effects of decision orchestration methods on functional code selection. On a 90-task evaluation split constructed from HumanEval specifications and test-killed AST mutants, three primary decision methods and two secondary diagnostic workflows were evaluated:

1. **Single pass baseline:** A single forward pass reading letter logits (A/B/C).
2. **Three cyclic rotations ($\mathbb{Z}_3$):** Three sequential shifted prompt evaluations whose slot scores are averaged across shifts.
3. **Pairwise tournament:** All 3 pairwise comparisons evaluated with Bradley-Terry Maximum Likelihood Estimation.
4. **Tournament with frozen confidence gate:** Autonomous acceptance threshold frozen from calibration split; abstains or accepts based on confidence threshold.
5. **Always-probe diagnostic:** Secondary diagnostic running assertion probes against candidate implementations.

---

## Findings Summary

All rows use the same 90 evaluation tasks; 30 separate tasks supplied calibration. Confidence intervals are Wilson 95% intervals.

| Method | Correct | Accuracy | 95% Wilson CI | Median Latency | p95 Latency | Expected Calibration Error | Status |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :--- |
| Single pass | 85/90 | 94.4% | 87.6–97.6% | 97.7 ms | 150.9 ms | 5.60% | Primary baseline |
| Three cyclic rotations | 89/90 | 98.9% | 94.0–99.8% | 291.4 ms | 453.5 ms | 1.60% | Primary method |
| Pairwise tournament | 87/90 | 96.7% | 90.7–98.9% | 485.4 ms | 747.5 ms | 2.57% | Primary method |
| Tournament + frozen gate | 87/90 | 96.7% | 90.7–98.9% | 485.4 ms | 747.5 ms | Not measured* | Diagnostic |
| Always-probe diagnostic | 87/90 | 96.7% | 90.7–98.9% | 571.0 ms | 829.9 ms | Not measured | Diagnostic |

*\*Bayesian assumed posteriors not reported as calibrated confidence; combined ECE not measured.*

---

## Disclosures & Limitations

1. **Statistical Significance:** The cyclic rotation improvement represents 5 corrections and 1 regression relative to baseline. The paired exact McNemar p-value is 0.21875. In a 90-task sample, this does not establish a statistically significant or repeatable lift.
2. **Confidence Gate Bounds:** The frozen calibration gate accepted all 90 evaluation tasks (coverage 1.0) and ran zero probes. Its 96.7% observed accuracy has a 90.7% lower Wilson bound, meaning the pilot does not prove at least 92% autonomous reliability.
3. **Post-Hoc Diagnostic:** The always-probe diagnostic was added after observing the primary result. Its 270 probes did not improve accuracy over pairwise scoring alone.
4. **Timing Boundaries:** Measurements reflect warm, loaded model inference in a local environment. They do not include model loading, network hops, prefix-cache negotiation, or concurrent user traffic.
5. **Speed Target:** All evaluated methods exceeded the strict local median target of less than 35 ms. Cloud latency and production SLA remain unmeasured.
6. **Positional Bias & Order Sensitivity:** Raw rotation winners differed on 10 of 90 tasks, and opposite cyclic-orbit winners differed on 1 of 90 tasks. While cyclic permutation eliminates slot bias in expectation, these stress tests demonstrate residual order sensitivity. No zero-bias certification is claimed.
7. **Benchmark Scope:** Evaluated candidates consist of canonical HumanEval implementations and synthetic AST mutants. Performance on natural LLM candidate proposals or diverse production tasks may differ.

---

## Model & Dataset Provenance

- **Model Identity:** `Mapika/decider-4b v2.1`, derived from Qwen3.5-4B-Base, quantized to MLX 4-bit affine format (group size 64).
- **Execution Hardware:** Apple M5 Max with 128 GiB unified memory running macOS 15 Sequoia.
- **Dataset Construction:** 120 HumanEval tasks admitted where the canonical implementation passed and both mutants failed original tests.
- **Split Isolation:** 30 tasks reserved for temperature calibration; 90 disjoint tasks used exclusively for evaluation. Position balance was strictly 30 A / 30 B / 30 C across the evaluation set. Gold labels were hidden from all scoring routines.
- **Audit Verification:** Complete forward-pass prompt hash tracking confirmed 1,440 genuine forward evaluations across calibration, evaluation, and sensitivity passes.