I wanted a local coding assistant that could read enough of a repository to make a change, use its tools, and get the code right. Speed came after that. The results gave me plenty of reasons to keep that order.
In my earlier LM Studio experiment, I got a local model to create a file, read it back, and verify it. That told me the connection worked. This time I wanted harder tasks I could repeat, so I could see how much useful coding work the agent would actually finish.
For this round I used a Windows desktop with an RTX 4060 Ti 16 GB, a Threadripper 1950X, and 128 GiB of RAM. I set a seven-day execution ceiling and allowed 300 GB of additional model weights. I reused installed files where I could and ran one model workload at a time.
The recorded work took about 56 hours of elapsed time, including downloads and pauses. By the end there were 63 registered configurations and 1,093 attempt records. That count includes coding tasks, capacity checks, timing, retries, and interrupted attempts. It would be a much larger claim to call them 1,093 different programming problems.
Nothing passed the complete qualification contract. I came away with two configurations I’d keep for supervised coding, a compression result worth investigating, and several failures where “the model is bad” would have been a poor explanation.
What counted as a pass
I started with a 65,536-token context window: 61,440 tokens of fully templated input, with 4,096 reserved for output. I wanted to fill most of that window with a fresh prompt and measure the wait. Setting a large context limit and sending a tiny question wouldn’t tell me what I needed to know.
To qualify, a configuration had to complete all of this:
- At least 27 passes across 36 regular Python and TypeScript coding runs, with complete coverage.
- At least nine passes across 12 long-context coding runs.
- All nine capacity markers retrieved across three seeded requests.
- Twenty valid fresh measurements and twenty separately measured cached follow-ups.
- A working isolated agent demonstration.
The harness included public tests, hidden checks, and known-good implementations. Codex helped me build and operate it. The local models handled the fixture tasks, and their code ran in isolated CPU Docker graders.
Each coding run allowed up to 16 tool turns, a generated-token budget, and a 900-second task deadline. The agent could inspect the source, patch it, run the public tests, and revise. A failing score came after that opportunity to work on the problem.
These were demanding gates. I chose them before I knew which model would look good, and kept them in place when the results arrived.
Three ways to measure the wait
The first token didn’t tell me when the agent could get to work.
That first streamed event could be reasoning. An agent could start talking almost immediately while I was still waiting for a usable tool call. I measured first output and first useful action separately. To count as useful, the action had to be completely parsed and valid for the advertised tool schema.
I also measured decode speed: verified output tokens per decode second. That described how quickly the model produced output once it was decoding. The time it spent ingesting a fresh repository was a separate part of the wait.
For three of the original 64K configurations, the screening measurements were:
| Configuration | Fresh first output | Fresh first useful action | Sustained decode |
|---|---|---|---|
| Qwen3.6 27B IQ2_M, upstream llama.cpp | 119.54-119.68 s | 123.33-124.55 s | 17.30 tok/s |
| Qwen3.6 27B IQ2_M, bundled llama.cpp | 107.05-107.19 s | 110.75-112.83 s | 17.13 tok/s |
| OxCoder 9B Q4_K_M, upstream llama.cpp | 35.23-35.40 s | 37.62-37.81 s | 31.79 tok/s |
There are three fresh samples per configuration here, plus one sustained full-context throughput sample. These are screening results. They don’t establish the required final twenty-sample p95, and the request timings exclude model launch startup.
I kept cached follow-ups separate too. No candidate reached and completed the final fresh/cached qualification sequence, so I don’t have a qualified cached-latency winner.
A high output rate can be real while telling me very little about how long I’ll wait for the agent to act on a fresh repository.
Qwen3.6 came closest
Qwen3.6 27B in the i1-IQ2_M representation had the strongest regular coding result. Both measured llama.cpp builds passed 28 of 36 regular coding runs.
In those 36 runs, the upstream build had one malformed or unpermitted tool failure. The bundled build had two. I’d need more than those small counts to claim that one engine was generally more reliable.
Both configurations retrieved all nine full-context capacity markers as well.
The long-context runs were where they stopped qualifying. Both passed four of eight completed runs. Four failures made the nine-of-twelve target unreachable, so the final four runs were skipped.
The failure records matter here. All four failed upstream long runs violated the requirement for exactly one final action. The bundled build had three failures of that kind and one timeout. Those records establish a failure of the required agent behavior. They don’t establish that the model forgot the repository.
Qwen3.6 IQ2_M would still be my starting point when coding quality matters most and I’m supervising the work. It passed the regular gate with relatively few regular tool failures. The full 64K qualification remained out of reach.
For a faster option, I’d keep OxCoder 9B Q4_K_M on upstream llama.cpp. It passed 11 of 21 completed coding runs with zero tool-error runs. All ten failures were deterministic coding-test failures.
That leaves a clearer problem to work with. The tool loop functioned, but the code often needed better decisions. These runs already allowed revisions, so I can’t assume another prompt would fix the result. I’d use this faster configuration for small edits with tests and review close at hand.
What changed at 16K
I later authorized a separate 16K pass. It used a 16,384-token window, 12,288 tokens of input, and the same 4,096-token output reserve. The requirements also included full GPU layer offload, f16 KV cache, and at least 512 MiB of measured memory headroom.
For the installed family models, I added an early tool-use check. After four coding runs, the screen would stop a model if two or more runs had ended in tool-call errors.
This is where that queue ended:
| Installed model | Coding passes / completed runs | Tool-error runs |
|---|---|---|
| Devstral Small 2 24B Q3_K_L | 2/12 | 3 |
| Ministral 3 14B Reasoning Q4_K_M | 4/14 | 3 |
| GPT-OSS 20B Q4_K_S | 0/4 | 4 |
| Gemma 4 12B QAT Q4_0 | 2/12 | 6 |
| Gemma 4 E4B Q6_K | 5/15 | 5 |
| Nemotron 3 Nano 4B Q6_K | 2/12 | 3 |
These denominators are early-stopped subsets of the suite. Tool failures are included among the coding failures, so the columns shouldn’t be added together. Only GPT-OSS triggered the first-four cutoff. The other models later reached ten regular task failures.
Gemma E4B and Nemotron responded quickly in the diagnostic workload. Their median fresh useful actions were about 5.11 and 3.91 seconds respectively, with native 256-output-token screens around 50.4 and 53.5 tok/s. Those requests used the smaller 12,288-token input. Comparing those times directly with the original 61,440-token results would hide how much less input they had to process.
The speed was useful to see. The coding and tool failures still kept these configurations from qualifying.
The Nemotron result covers the installed Nano 4B file. I can’t extend it to the rest of the Nemotron family.
Where GPT-OSS tool use broke
Broken tool use is a more serious obstacle for me than incorrect code. A working tool loop can run a test and expose a mistake. If the next action is rejected, the agent may never get that far.
GPT-OSS needed a closer look at exactly where that happened.
All four failed coding runs successfully used the native file-listing and reading tools. The next patch call supplied JSON-valid arguments, but the patch itself used the Codex-style *** Begin Patch and *** Update File format.
The frozen patch tool required standard numbered unified diffs. It rejected the different patch dialect.
A separate native Harmony parser error occurred in the third capacity request. The first two seeded requests had succeeded, retrieving six of the nine required markers.
The tested integration failed, and the records show an interface mismatch worth retesting. They don’t support a blanket claim that GPT-OSS can’t call tools. I’d run a bounded retest with a clearly followed patch contract and compatible parsing. Accepting the wrong format silently while keeping the same grade would change what the benchmark measured.
What fit in GPU memory
I initially thought shorter context would mainly give me room for higher quantization. It does provide that room. Sending less actual input also reduces the fresh prefill work, though, and both effects matter when choosing a configuration.
The file sizes didn’t make the memory boundary obvious:
| 16K trial | Weight file size, decimal GB | Measured free headroom |
|---|---|---|
| Qwen3.8 27B UD-Q4_K_M | 16.46 | 290 MiB |
| Qwen3.8 27B UD-Q4_K_S | 15.36 | 364 MiB |
| Qwen3.8 27B UD-IQ4_XS | 14.25 | 1,354 MiB |
| Qwen3.6 27B IQ4_XS | 15.44 | 312 MiB |
| Qwen3.6 27B UD-Q3_K_XL | 14.47 | 1,164 MiB |
The two Qwen3.8 Q4 files and Qwen3.6 IQ4_XS missed the headroom rule, and their full-prompt probes timed out. Qwen3.8 IQ4_XS and Qwen3.6 Q3_K_XL fit and passed all nine capacity markers.
A GGUF file tells me what’s on disk. The GPU also has to hold the weight buffers, KV cache, compute buffers, and runtime allocations while leaving room for Windows activity. I corrected a headroom calculation after finding about 270 MiB of reserved memory that “total minus used” had counted as available.
The fitting Qwen3.6 Q3_K_XL stack ended with eight passes in eighteen coding runs, including nine tool failures. Qwen3.8 IQ4_XS had six passes in eleven completed runs, including four tool failures, before I stopped it.
Higher precision didn’t automatically fix these configurations. I changed context and weight representation together, so these results can’t isolate what a particular quantization did to reasoning.
What Bonsai saved in memory
Ternary Bonsai 2 27B was the compression experiment I most wanted to check. The PTQ1_0 weight file was about 5.95 GB decimal, and running it required the Prism ternary fork.
It fully offloaded to the GPU and retrieved all nine 64K capacity markers. Peak GPU use in those full-context capacity requests was roughly 10,426-10,554 MiB.
Then there was the wait. With 61,440 fresh input tokens, the three clean capacity samples took 167.5-169.2 seconds to first output and 187.5-192.1 seconds to the useful action.
A smaller probe took 40.7 seconds to first output and 54.1 seconds to useful action. That request contained 16,384 input tokens, with the engine still configured for 65,536. It wasn’t the separate 16K-window benchmark.
I stopped Bonsai after five completed coding smokes: three passed and two failed on tool streams. A sixth run was interrupted. That belongs among the interrupted attempts, not the completed failures.
I’d keep Bonsai around for memory-efficiency research. These results wouldn’t make it my everyday coding choice.
Drafting tokens with MTP and DFlash
I tested speculative decoding with Qwen3.6 IQ2_M, using q8 and q4 target-cache profiles. The native counters confirmed that the methods drafted tokens and that draft tokens were accepted.
MTP acceptance was approximately 90-92%. DFlash acceptance was approximately 58-59%.
Some auxiliary decode samples were promising: roughly 24.2-25.5 tok/s for MTP, and 36.7 tok/s for one clean DFlash q4-cache capacity sample. There were very few clean samples, and several other intervals overlapped with foreign GPU activity.
Every speculative arm failed screening. None produced a qualifying set of fresh useful-action measurements.
I’d keep investigating these methods, with the extra draft weights and cache pressure accounted for. The long fresh prompt still has to be processed. Draft acceptance and decode speed leave the coding agent’s tool behavior to prove.
The engines I couldn’t compare
The search covered upstream and bundled llama.cpp paths, an eligible CUDA TurboQuant fork, and attempts to compare ik_llama.cpp, ExLlamaV3/TabbyAPI, vLLM, SGLang, LM Studio, and Ollama.
Some comparisons stopped at setup or integration. The ik container was missing a CUDA library. Tabby was missing snapshot metadata. Docker storage headroom blocked vLLM and SGLang, while managed loading or request controls remained unresolved for LM Studio and Ollama. Bounded setup budgets also expired.
Authentication wasn’t what ultimately stopped LM Studio.
Those are comparisons I couldn’t complete. I don’t have measurements showing that the engines are slower or worse. A fair ranking would require resolving those barriers and running the same tasks successfully.
What I’d use and test next
When supervised coding quality is the priority, I’d start with Qwen3.6 27B IQ2_M. For a faster assistant with consistent tool behavior across its completed regular tasks, I’d keep OxCoder 9B Q4_K_M on upstream llama.cpp.
GPT-OSS gives me an interface to repair and retest. Bonsai gives me a memory-efficiency result to investigate. The faster MoE and small-model output rates were interesting, but the coding or tool failures kept those configurations off my practical shortlist.
My next experiment would be a small, clearly versioned GPT-OSS interface retest. Then I’d run a matched 16K comparison of Qwen3.6 IQ2_M and OxCoder to make the latency tradeoff easier to judge. I’d return to speculative decoding once the target agent had earned confidence in its tools.
I came away with measured limits I can use when choosing a configuration. I still don’t have the fully qualified local coding agent I went looking for, so tests and review remain part of the job.
Benchmark Source Code
I’m going to clean up the source code to run the benchmark and release it. Just got to get it into a shape that isn’t a pile of hot garbage.