v0.4 reference validation — one Windows PC#
These are independently reviewed LocalAgentBridge results, not universal model rankings. The five v0.3 rows are preserved same-PC measurements from the same frozen suite, not fresh v0.4 reruns. All raw historical evidence remains private; the public aggregate JSON contains no request/session IDs, personal paths, prompts, usage database or source excerpts.
Machine and protocol#
Reference resource class: 16 GiB RAM and 6 GiB VRAM, Windows, Ollama 0.32.5. Exact CPU/GPU identity, driver and machine inventory are withheld for privacy. These coarse resource classes help interpret fit; timing is not portable.
Eleven frozen v0.2 task/message bodies, verified hashes, temperature 0, 8,192-token context, 600 generated-token limit, think=false, keep_alive=0. One run per model; no automatic retries or best-of selection. The timeout is 600 seconds per request. Returns remain bounded JSON, normally 3,000 bytes and 1,350 for the consensus participant exercise. A/C/R grades come from Codex's independent comparison with the supplied source/contract. Model confidence never establishes correctness.
Wall time includes model loading and prompt evaluation. Other desktop activity and the later model downloads overlapped some earlier benchmarks, so small timing differences are not controlled speed rankings. No model ran concurrently with another inference benchmark. Cold-load tests differ from a warm production model.
New exact model tags#
| Exact installed Ollama tag | Architecture; total / active parameters | Quantization | Artifact bytes / GiB | Digest prefix |
|---|---|---|---|---|
| qwen3.5:9b-q4_K_M | Dense hybrid; 9,653,104,368 / same | Q4_K_M | 6,594,474,711 / 6.14 | 6488c96fa5fa |
| qwen2.5-coder:14b-instruct-q4_K_M | Dense; 14,770,033,664 / same | Q4_K_M | 8,988,124,298 / 8.37 | 9ec8897f747e |
| gpt-oss:20b | MoE; 20,914,757,184 / about 3.6B | MXFP4 | 13,793,441,244 / 12.85 | 17052f91a42e |
Tags and local /api/show parameter counts/quantization were verified. Model-class metadata was checked before downloading; installed artifacts were then checked again. GPT-OSS local metadata confirms 32 experts with 4 used per token; its rounded active parameter count is from the upstream model card. Total memory must not be estimated from active parameters alone. Qwen3.5's vision capability was identified but only text workloads were tested. No 120B model was installed. Exact tag references: Qwen3.5, Coder 14B, GPT-OSS. Models retain their own licenses.
Quality#
Accepted = no substantive correction; corrected = useful but requiring correction; rejected = materially unreliable or no usable answer. Error flags count cases, not individual assertions, and can overlap. API/schema failures are not evidence of a model's intrinsic reasoning ability. This suite measures its usefulness through the bridge's actual bounded-output protocol.
| Model | Accepted | Corrected | Rejected | API/schema failures | Factual / missed / unsupported / wrong target |
|---|---|---|---|---|---|
| Qwen3 4B | 2 | 5 | 4 | 0 | 6 / 4 / 3 / 0 |
| Coder 7B | 3 | 2 | 6 | 0 | 6 / 4 / 1 / 1 |
| Phi4 Mini | 3 | 4 | 4 | 0 | 6 / 6 / 3 / 3 |
| Qwen3 14B | 6 | 5 | 0 | 0 | 4 / 2 / 0 / 0 |
| DeepSeek Lite | 2 | 4 | 5 | 0 | 4 / 6 / 3 / 2 |
| Qwen3.5 9B | 2 | 7 | 2 | 0 | 3 / 7 / 4 / 2 |
| Coder 14B | 4 | 3 | 4 | 0 | 5 / 7 / 0 / 3 |
| GPT-OSS 20B | 0 | 0 | 11 | 11 | 0 / 11 / 0 / 0 |
| GPT-OSS 20B (low) | 7 | 4 | 0 | 0 | 1 / 3 / 1 / 1 |
A/C/R by workload (one case each; analyze_code has three related cases):
| Workload | Qwen3 4B | Coder 7B | Phi4 Mini | Qwen3 14B | DeepSeek Lite | Qwen3.5 9B | Coder 14B | GPT-OSS 20B | GPT-OSS 20B (low) |
|---|---|---|---|---|---|---|---|---|---|
| repository_discovery | A | C | C | A | C | C | C | R | C |
| code_search | A | A | A | A | A | A | A | R | A |
| code_summarization | C | R | C | C | R | C | C | R | A |
| simple_analysis | C | A | A | A | C | C | A | R | A |
| implementation_suggestion | R | R | R | C | R | R | R | R | A |
| coding_task | C | R | R | C | R | C | R | R | C |
| test_failure | R | R | R | A | C | A | R | R | A |
| diff_review | C | A | A | A | C | C | A | R | C |
| general_reasoning | R | R | C | C | R | R | R | R | A |
| task_triage | C | C | C | C | R | C | C | R | C |
| consensus_review | R | R | R | A | A | C | A | R | A |
Qwen3.5 reduced outright rejections versus Qwen3 4B (2 versus 4), but accepted only the same number of cases (two), with different strengths. It falsely allowed remote role endpoints, double-counted cached input tokens, missed the retry-cap fix and omitted the registry-port test. Its roughly 3x mean wall latency does not justify replacing the cheap default. Useful correctable summaries/discovery are a possible future targeted experiment.
Coder 14B improved the whole-suite count modestly over Coder 7B (4/3/4 versus 3/2/6), including the None-return opinion. However, both scored 1 accepted and 2 rejected across the three analyze_code cases. The 14B still missed max→min, gave the wrong hostname:port expectation, blamed correct addition code for a bad test, and supported a 175-token total despite reasons implying 115. This is not a dramatic specialization improvement and does not displace Qwen3 14B escalation.
Qwen3 14B remains the strongest historical whole-suite result (6/5/0), but needs verification: its corrected cases include endpoint policy, retry boundary values, missing registry-port coverage, token-accounting uncertainty and triage. High-risk or architectural decisions remain with Codex even if a model was previously accepted.
GPT-OSS produced no usable structured JSON in all 11 original think=false cases. Those requests returned provider counters and consumed time; they are recorded as 11 rejected integration results, not silently discarded. A separate compatibility pass changed only think to low, as supported by Ollama's thinking documentation. Its same 11 tasks returned valid JSON: 7 accepted / 4 corrected / 0 rejected. This is a protocol-adjusted comparison, not identical-settings pairing.
The corrected cases omit config.py from discovery, place the correct hostname:port pair outside the explicitly requested findings field, overclaim that all addition tests fail after subtraction (b=0 still agrees), and omit explicit Codex patch review from triage. The retry expression and boundary descriptions are correct; that task does not require literal numeric expectations. Grades follow the original rubric's core requirements, rather than model confidence or a blanket JSON-success score.
Performance#
| Model | Calls | Mean / median wall s | Total wall / generation s | Input / output tokens | Generation tok/s | Load s |
|---|---|---|---|---|---|---|
| Qwen3 4B | 11 | 15.6 / 14.7 | 171.1 / 64.1 | 8,049 / 1,822 | 28.43 | 64.6 |
| Coder 7B | 11 | 22.4 / 21.0 | 246.9 / 87.8 | 8,027 / 1,035 | 11.79 | 105.1 |
| Phi4 Mini | 11 | 14.2 / 12.9 | 156.5 / 48.2 | 7,764 / 1,311 | 27.20 | 68.8 |
| Qwen3 14B | 11 | 77.1 / 73.6 | 848.1 / 547.4 | 8,115 / 1,619 | 2.96 | 212.2 |
| DeepSeek Lite | 11 | 34.8 / 32.9 | 382.4 / 112.9 | 8,852 / 1,278 | 11.32 | 212.9 |
| Qwen3.5 9B | 11 | 47.7 / 45.4 | 524.6 / 198.3 | 8,070 / 1,154 | 5.82 | 260.1 |
| Coder 14B | 11 | 61.4 / 56.7 | 675.5 / 362.0 | 8,027 / 1,039 | 2.87 | 220.0 |
| GPT-OSS 20B | 11 | 54.5 / 50.7 | 599.6 / 78.9 | 8,490 / 780 | 9.88 | 419.2 |
| GPT-OSS 20B (low) | 11 | 54.2 / 53.8 | 595.7 / 78.5 | 9,111 / 853 | 10.86 | 378.1 |
Generation tok/s is total output tokens divided by final generation duration, not wall duration. Local output counters can include thinking when the provider reports it. Cached/reasoning breakdowns for Ollama are unavailable unless separately reported. Failed requests still consume time and any reported tokens; unavailable counters must not be interpreted as zero usage. Full totals are in the JSON.
Memory and offload#
| Model | Ollama / GPU / host allocation GiB | Ollama RSS GiB | System used GiB | Whole GPU peak MiB |
|---|---|---|---|---|
| Qwen3 4B | 3.61 / 3.61 / 0.00 | unavailable | 10.88 | 4,142 |
| Coder 7B | 5.01 / 3.96 / 1.05 | unavailable | 11.57 | 4,491 |
| Phi4 Mini | 3.38 / 3.38 / 0.00 | unavailable | 10.25 | 3,887 |
| Qwen3 14B | 10.08 / 3.89 / 6.19 | 6.81 | 14.83 | 4,373 |
| DeepSeek Lite | 10.76 / 4.04 / 6.71 | 7.16 | 14.00 | 4,520 |
| Qwen3.5 9B | 5.99 / 3.33 / 2.65 | 3.26 | 10.80 | 4,831 |
| Coder 14B | 10.04 / 3.90 / 6.13 | 6.58 | 13.90 | 4,139 |
| GPT-OSS 20B | 13.07 / 3.81 / 9.26 | 9.78 | 15.65 | 4,400 |
| GPT-OSS 20B (low) | 13.07 / 3.81 / 9.26 | 9.74 | 15.25 | 4,415 |
One-second sampling gives approximate observed peaks, not continuous maxima. Ollama RSS sums its processes and descendants and can include shared pages. System-used and whole-GPU figures include other applications. API allocation is neither artifact size nor process RSS; host allocation is API size minus GPU size, an offload indicator rather than exact resident CPU weight memory.
The original three small-model RSS samples used an incomplete process filter, so their table entries stay unavailable. Separate v0.3 summary-only rechecks measured about 0.80 GiB Qwen4, 1.51 GiB Coder7 and 0.99 GiB Phi RSS; these are not whole-suite peaks. Larger-model and v0.4 samples include llama-server descendants. Qwen3.5's text-only API allocation can be smaller than its artifact because the artifact also contains vision components. Do not label this a full-weight VRAM fit.
Local compression and cloud accounting#
| Model | Valid input bytes | Returned bytes | Local compression | Potential context avoided, bytes |
|---|---|---|---|---|
| Qwen3 4B | 16,511 | 10,385 | 37.1% | 6,126 |
| Coder 7B | 16,511 | 7,306 | 55.8% | 9,205 |
| Phi4 Mini | 16,511 | 9,027 | 45.3% | 7,484 |
| Qwen3 14B | 16,511 | 10,210 | 38.2% | 6,301 |
| DeepSeek Lite | 16,511 | 10,068 | 39.0% | 6,443 |
| Qwen3.5 9B | 16,511 | 8,064 | 51.2% | 8,447 |
| Coder 14B | 16,511 | 7,809 | 52.7% | 8,702 |
| GPT-OSS 20B | 0 | 0 | unavailable | unavailable |
| GPT-OSS 20B (low) | 16,511 | 7,301 | 55.8% | 9,210 |
Compression = 1 - returned bytes / supplied-context bytes. The suite supplies 16,511 context bytes per model; the table includes only valid returned payloads. It excludes objective/instruction/schema overhead. Error-only runs have no valid compression measurement, not 100% savings. Short synthetic exercises can expand; the discovery/search/summary subset is reported separately in the JSON.
Potential context avoided — not measured Codex token savings. Codex tool definitions, orchestration, verification and repeated cloud context are outside this byte comparison. Measured cloud calls avoided and Codex token reduction are unavailable: there is no controlled with/without-delegation run. Counting local calls as avoided cloud calls would overstate the result. Correctable/rejected answers also incur verification and repair work that these byte figures cannot value.
Routing decision and limits#
| Cap | Enabled candidate models / roles | Production recommendation |
|---|---|---|
| Light | qwen3:4b general; phi4-mini reasoning | Cheap evidence/summary/opinion work, with Codex checking results |
| Medium | Light plus qwen2.5-coder:7b coding | Retained default; per-tool quality exclusions still apply |
| Large | Medium plus qwen3:14b-q4_K_M general/coding/reasoning | Reviewed stronger escalation, especially code analysis |
| Maximum | Large plus gpt-oss:20b general/coding/reasoning, thinking=low | Enabled experimental heavy escalation; never the default first model |
Production remains Medium. Its combined memory budget was raised from 12 to 16 GiB to permit the explicitly Maximum-only GPT profile; the Medium tier's own 12 GiB ceiling still applies. GPT's registry weight plus reserve is about 14.85 GiB, and measured Ollama allocation is about 13.07 GiB, with only 3.81 GiB on GPU. The benchmark-only budget was 18 GiB. The portable public example retains 12 GiB and the three small role examples; it does not import this machine's preferences. Memory is tight on this PC and these are configured budgets, not OS reservations.
Qwen3.5 9B and Coder14 remain installed but disabled experiments. DeepSeek remains disabled with its v0.3 history. No existing enabled model was removed or deleted. Coder7 is retained for inexpensive first-pass review; its high rejection rate means it is not a dependable general coding worker. The coarse analyze_code gate excludes it, so that tool hands back to Codex at Medium and begins with Qwen3 14B at Large. At Maximum its next stronger eligible candidate is GPT-OSS. Other simple calls start with Qwen4, Coder7 or Phi; moderate/stronger hints may skip weaker candidates. Maximum does not override the two-attempt budget, so a third model is not automatically run after two attempts. GPT is useful when it is within the remaining attempt budget.
A separate real stdio MCP follow-up used an independently corrected Qwen3 14B request and selected GPT-OSS with reason previous_attempt_failed at Maximum. It completed in 70.76 seconds: 1,156 input and 229 output tokens, 2,765 supplied bytes and a 1,270-byte final routing envelope (54.1% local compression). The test cap was isolated; production stayed Medium. The result included correct bare/tagged/port pairs but an incomplete optional pytest snippet, so Codex reported corrected. The provider filled a 240-character schema field; the bridge did not clip it. Structurally valid JSON and truncated=false cannot certify complete executable code. No code snippet was executed or applied automatically.
The new benchmark has 44 calls: 11 each for Qwen3.5 and Coder14, plus two 11-case GPT protocols. Add one routed GPT follow-up. Thus new-model calls are 11 / 11 / 23, including 11 strict-protocol failures. Independent clean-install verification calls are recorded separately. Each paired suite has one second-opinion/consensus-participant exercise; real multi-participant consensus was also exercised by the setup smoke. Completed metadata and explicit feedback were copied into the local dashboard store without changing its cap or duplicating existing request IDs. Historical strict protocol failures remain visible; the public comparison separates them from low mode.
The router selects the smallest eligible configured capability strength for the requested role. Explicit tier assignments use observed usefulness, specialization, memory and latency; they do not bypass memory/latency limits. Generic profiles can provide optional per-tool reviewed reliability without changing code. Local reviewed data takes precedence. Unreviewed profiles are clearly marked, not represented as validated. Three reviewed cases trigger the current 25% acceptance / 50% maximum rejection gate; the tiny sample size is a remaining weakness.
Input/attempt limits remain 10,000 characters and two local attempts. The delegation policy recommends local analysis of at most eight files, medium minimum confidence, and 25% minimum compression for sufficiently large contexts. Selected-file reading has its own 20-file hard bound plus the input budget. Low confidence, invalid/truncated output, contradictory evidence, unsupported requirements and failed attempts can escalate within the selected cap. Ambiguous, complex multi-file, architectural and high-risk work belongs with Codex under the project policy; complex routing hints force a handoff. Caller-supplied scope/risk hints and Codex review remain necessary: the bridge cannot prove semantic complexity or correctness from schema/size alone. Agreement is not proof and no autonomous agent loop was added. The coarse per-tool gate currently blocks Coder7 analyze_code after its three reviewed cases; Medium can therefore hand this back immediately. Better task-level evidence is needed before expanding that model's routing.
Dashboard and release validation#
See dashboard screenshots and metric meanings and the public-release audit. All screenshots use synthetic data. Personal model configuration, telemetry, raw benchmark output, old machine-specific reports and session IDs remain excluded from public files.
All source, CLI, telemetry and native UI regression checks passed: 126 passed, one Windows Developer-Mode symlink skip. Lint passed. The isolated initial clean installation made three exact-model verification calls plus nine smoke calls covering all seven handlers and consensus. Generated registration was accepted by an isolated Codex CLI config. Final clean-package/source audit evidence and publication checks are recorded in the release checklist.