By Sridhar Vanka · All writing
I started this project to learn how to run models locally and build an evaluation I could understand. With Claude helping me write the harness, I ran four local models on my Mac through Ollama, five times each.
Each agent received a specification for a Python expense-splitting library, a shell, and three example tests. A separate grader checked the code it left behind against 88 hidden tests.
None of the twenty submissions passed all the hidden tests. Agents declared completion in ten runs, and nine of those submissions still failed at least one example test.
The grader exposed the failures after the agent stopped. My harness accepted its completion signal without checking the examples first. I began looking more closely at what the agents had seen and how my own setup contributed.
Four of the ten claims came from agents that had never requested a test run. Another three had requested tests in a second code block, but my harness executed only the first block. Their test requests never ran.
The remaining three claims followed failures the agents had seen. Each attempted a change afterwards. In one case, the harness rejected the attempted fix because of its code-block format. The other two agents changed the code and declared completion without retesting it.
I was concerned by how easily unfinished work could be accepted. I still could not tell whether a model knew its code was wrong when it stopped. In some runs, I had not even shown it the test result it had asked for.
The harness had other problems too. Its response limit included the model's thinking, cutting off some replies before executable commands appeared. I needed to fix that before comparing prompts or completion rules in another batch.
The final grader was already separate from the coding agent. I added an optional check at the point where the agent tried to finish.
On a completion claim, the harness ran the original three examples. If one failed, it returned the result to the agent, which could keep working within its remaining budget. The final submission still went through the hidden tests.
Passing the examples could not establish that the whole task was solved. In the original study, one submission passed all three examples but achieved a weighted hidden-test score of 83%. The completion check would have accepted it with requirements still unmet.
I wanted to measure whether requiring the check improved the final code, as well as how much more time and how many tokens it consumed.
While checking the evaluator, I found that specially written candidate code could make the old design accept a fabricated passing report. I constructed that submission to test the harness; I did not observe a model doing this during the experiment.
The repaired evaluator runs test assertions and records their results outside the environment executing the candidate code. Candidate code runs in restricted Docker workers and cannot supply the trusted test report. The same constructed attempt fails against the repaired evaluator.
Before a result can be published, the evaluator grades the saved submission again and checks that the outcomes match. The records identify the exact file that was graded. I still need to check whether the tests cover the requirements adequately.
The 18 October update recorded one trial per condition using qwen3:14b: the original instruction or an explicit instruction to test before finishing, each with and without the completion check.
| Condition | Examples passed | Hidden tests passed | Agent time |
|---|---|---|---|
| Original instruction | 2/3 | 49/88 | 3.8 minutes |
| Original instruction + completion check | 3/3 | 52/88 | 52.3 minutes |
| Explicit testing instruction | 1/3 | 31/88 | 13.5 minutes |
| Explicit instruction + completion check | 2/3 | 35/88 | 240 minutes; limit reached |
I ran these trials to check the revised setup: one task and one seed per condition. The counts include all 88 hidden tests, including the stress tier. Agent time excludes preflight and final grading.
With the original instruction, adding the check produced a submission that passed the examples and three more hidden tests. It also took much longer. With both the explicit instruction and the check, the run exhausted four hours and still failed an example.
I need repeated runs to find out whether either change improves the code and how much extra time and how many tokens it uses. The original twenty-run study mixed time budgets, and its submissions have not been rechecked under the repaired evaluator. I have kept those results separate.
I also began connecting the existing sandbox and separate grader to Inspect. Four saved submissions produced identical results across both grading paths: all 364 individual test outcomes matched. That checked the grading of those submissions. Comparing agent behavior under the two runners remained further work at the time of the update.
Download the redacted update data →I expect to face this on a future team using coding agents. When an agent says it has finished, I want to know what we have actually checked before we accept the work.
I want to keep testing how much further separation of responsibilities helps, and where the returns diminish. I also want to see what happens when the agent cannot resolve a failure. Can the workflow recognize that it is stuck? What does it leave for a person to investigate?
Eventually, I want to understand how far we can take writing and deploying code without human touchpoints, and what we need when we reach that boundary. I have not tested deployment here. I am still working on the checks that let a coding agent finish, what they miss, and how much work they add.
The revised account draws on the 11 and 18 October 2026 records. The questions about future teams and autonomy reflect my perspective in November 2026.
These sections preserve the original exploratory study and its methods at the time. The grading-report weakness and subsequent changes are described above. The original results mixed time budgets and have not been rechecked under the repaired evaluator.
The experiment runs on a 24 GB Mac. A Python agent sends prompts to each model through Ollama, executes its shell commands, and returns the output. The model can read files, write code, and run tests until it signals that it has finished or uses up its steps.
Each model gets the same specification for an expense-splitting library. It has to support four ways of dividing an expense, track payments, and work out who owes whom. Amounts use Python’s Decimal type. The spec also says who gets a leftover cent and how to choose between people with equal balances when settling debts. Three example tests are available to the agent; a separate grader checks its saved code against 88 hidden tests.
Commands run in a Docker container with no network access and limits on CPU, memory, and processes. For grading, the solution is copied into a fresh directory alongside the hidden tests and the original examples. Changes the agent makes to its own tests do not affect that score.
| Model | Mean ± SD | Range | Core | Splits | Settle-up | Validation | Stress |
|---|---|---|---|---|---|---|---|
| qwen3:14b | 70% ± 12% | 49%–83% | 91% | 44% | 57% | 90% | 3% |
| qwen2.5-coder:14b | 41% ± 16% | 21%–71% | 62% | 12% | 17% | 75% | 1% |
| qwen2.5-coder:7b | 31% ± 4% | 26%–37% | 34% | 1% | 0% | 88% | 0% |
| llama3.1:8b | 13% ± 11% | 0%–26% | 14% | 1% | 0% | 35% | 0% |
The score is the average of four test-group pass rates: Core (13 tests), Splits (15), Settle-up (6), and Validation (24). Each group has equal weight. Stress is the pass rate on another 30 tests that compare random ledgers with the reference implementation. No run passed all 88 hidden tests.
Download the 20 run summaries →The agents signalled completion in ten runs. Nine of those still failed at least one original example test. The highest-scoring run passed all three examples and received a weighted score of 83%. It passed 50 of the 88 hidden tests.
In qwen2.5-coder:7b run 01, the agent wrote a test file at step 5, then repeated the same command through step 40. That was 35 repeats with no change to the command. You can follow this run in the viewer below.
qwen3:14b had the highest average score, at 70%. Its transcripts contain longer reasoning before code writes and fewer test runs than the coder models. I have not tested whether either difference explains the score.
Some solutions gave leftover cents to the wrong person or expected weights in a format different from the spec. Others returned what a person paid when asked what they spent. There were Python errors too: assigning a dictionary to self.balances hid the balances() method, so calling it failed.
Choose a run to see each recorded shell command and the output returned to the agent. The agent received shortened output when a command produced more than 4,000 characters. The viewer leaves out model reasoning and the later hidden-test grading output.
Scoring. With 24 validation tests and six settle-up tests, counting every pass equally gave validation four times as much weight. I changed the main score to give each of the four groups equal weight. Stress is reported separately.
Tests that hang or crash. The grader now runs each test in a separate child process with time and memory limits. A test that hangs fails on its own. The score uses the expected number of tests in each group, including when a solution fails to import and pytest cannot collect its tests.
Time limits. Ten runs stopped at the old 30-minute limit. Those were rerun with a 40-step budget and a four-hour cap to catch hangs. Earlier runs that finished were kept, including one initial trial with a ten-minute limit. All runs had a 40-step ceiling, but the table still combines different time limits.
qwen3:14b run 04 scored 55% before that change and 83% when rerun from the start with the same seed. The extra time may have helped, though one rerun cannot establish how much of the difference it explains. Earlier generation speeds also varied enough to suggest the Mac had been sleeping or throttling. I used the logged tokens per second to identify those slow runs.
Models I could not evaluate. gpt-oss:20b returned tool calls that the agent and Ollama could not handle in this setup. Mistral Small 24B ran too slowly at the shared 32k context on this Mac. I kept their records but left them out of the score table because I could not give them a usable coding run.
These results cover one task on one Mac, with five runs per model. A different task or agent could produce different results. I saved model tags, but did not record the exact model-file identifiers or all hardware and software versions. I also cannot establish whether similar expense-splitting tasks appeared in the models’ training data.
The next comparisons need fresh runs with consistent settings. After checking the prompt and completion policy, I would like to try frontier models on the same task, including the existing Stress tests.
I have not published the repository. Before sharing it, I need to leave out the hidden tests and reference code so later evaluations can keep using them. The downloads on this page contain run summaries and the shell commands and output shown in the viewer.
Based on the saved results, transcripts, code, and analysis in the project as of 11 October 2026.