raykin
Every trial against a local model (`ollama/…` or `gpustack/…`) ends in `harness_crash` before it can be scored, even when the agent solved the task and the verifier would have passed. ### What happens ``` invalid hello-world harness_crash 23.2s error ArgumentError: missing keyword: :registry task agent model reward outcome cost_usd steps tokens duration hello-world miniswen <local model> - harness_crash - - - 23.2 ``` ### Repro ```sh # bench.yml # agent: # name: miniswen # model: ollama/<any-model-served-by-an-openai-compatible-endpoint> OLLAMA_API_BASE=http://localhost:8080/v1 lemans run --backend docker --task hello-world ``` Any OpenAI-compatible local server works — this was llama.cpp's `llama-server`, reached through ruby_llm's `ollama` provider. The crash is in result assembly, so it reproduces regardless of what the model does inside the sandbox. ### Where it comes from The two gems disagree about whether `registry` is optional: - `miniswen/agent.rb:205` — `CostSource#to_h` ends in `.compact`. - `miniswen/agent.rb` `cost_source` — for a local provider it builds `CostSource.new(name: :local_provider, model:, priced_as:, registry: nil)`. `.compact` then drops the nil `registry:` key. - `lemans/result.rb:54` — `CostSource = Data.define(:name, :model, :priced_as, :registry)` requires all four. - `lemans/agents/miniswen.rb:95` — `Result::CostSource.new(**result.cost_source.to_h)` splats the compacted hash straight in, so the missing key raises. Cloud models never hit this: their `cost_source` carries a non-nil `registry` (the pricing registry revision), so `.compact` keeps the key. Only the `LOCAL_PROVIDERS` path (`agent.rb:29`) sets it to nil. `lemans/result.rb:71` already handles the same shape defensively when loading a stored result, which looks like the intended contract — `agents/miniswen.rb:95` just doesn't do it. I have this working locally with a small change at the call site, but I have not sent a PR since the fix could equally belong on the miniswen side (not compacting, or not treating a local run as having no registry) and you may prefer one over the other. ### Versions - lemans 1.0.0 - miniswen 0.0.1 - ruby_llm 1.16.0 - Ruby 4.0.1, Ubuntu, `--backend docker`