// blog

We Tested AI Agent Harnesses on Our Own GPUs. One Claimed Success Without Doing Anything.

Every “best AI agent harness” listicle you can find right now benchmarks the same way: vendor APIs, datacenter models, pass rates quoted from someone else’s run. The best published study in the genre — springbrand’s 8-harness, 240-run DeepSeek V4 comparison — openly admits its winner only passed 66.7% of tasks, and that dsh launched two days after their study closed, so it wasn’t tested at all.

So we ran the harnesses ourselves, on the hardware homelab owners actually have: a dual-RTX-3090 box serving a local 27B model, with a cheap cloud endpoint as the second column. Identical 5-task suite for every config. Every check verified on the filesystem — not by asking the agent whether it succeeded.

One result matters more than the rest: out of the box, one of these harnesses reported success on tasks it never performed.

The configs

Tested on: Ubuntu, dual RTX 3090 (48 GB, PCIe — no NVLink), llama.cpp serving qwen3.8-27b Q8_0 locally; Nous Portal DeepSeek V4 Flash as the cloud column; Node 24 LTS; dsh 0.1.1-rc.2; TrueForge 0.1.4. The 5-task suite: create a file, edit a file, compute-and-write results, count directory entries, multi-step rename. Full raw checkpoints: raw checkpoints on GitHub (JSON per run, env-stamped).

Finding 1: the default-config trap

TrueForge’s standalone install ships no execution tools. Its architecture docs are upfront that a code sandbox is “a tool, provisioned only when needed” — and against a local model, it’s never provisioned. The agent describes creating files in confident prose, reports success, and nothing on disk has changed. 0/5, every task, with success claims on every task.

None of the existing coverage mentions this, because nobody tests the default config — especially not against a local model. And it’s not just a correctness problem. A harness that reports success with zero tool-call events in its audit trail is an observability failure too: if you’re running agents on your own box, the tool-call log is the only ground truth you have. This config gives you none.

To be fair to TrueForge: with its intended setup — MCP filesystem server plus a cloud model — it passes 5/5. We tested both its out-of-box state and its configured path and report both, because “it works if you configure it right” and “it claims success when it does nothing” are both true.

Finding 2: the indirection tax

On the identical 5 tasks where dsh’s direct function-calling loop needed 5 model calls, TrueForge’s call_tool wrapper needed 22 — roughly 4× the model calls for the same work. That overhead is measured on TrueForge’s intended configuration, so it stands regardless of how you feel about the default-config finding. On a local model it’s pure latency; on a paid endpoint it’s pure cost.

Finding 3: the small-model hazard

The same qwen3.8-27b checkpoint that went 5/5 through dsh’s direct loop failed every task through TrueForge’s indirection — it copied the JSON schema into the arguments instead of filling them in. This is exactly the class of result API-only reviews structurally can’t produce: how a harness’s plumbing treats a 27B model you already own. If you’re running local models, harness choice isn’t cosmetic — it’s the difference between working and silently not.

Finding 4: cost per task, on hardware you own

The TrueForge number is small in absolute terms — the point isn’t the pennies, it’s that the figure is measured on our meter, not quoted from a pricing page. Incumbent coverage quotes stale list prices (several still predate DeepSeek’s August 16 price change). Cost-per-task on your own hardware can’t be copied without the rig.

Setup friction, for the record

Things the install guides don’t tell you: dsh requires Node ^22.19 || >=24 (Node 23 fails silently-ish); its sharp dependency’s Linux binary can be skipped by npm and only surfaces as a plugin-load failure; OpenAI-compatible endpoints need the /v1 suffix and a plugin patch to set the default model; the web UI refuses --host 0.0.0.0, so it’s localhost-only unless you proxy it. Full provenance (versions, GPU, driver, per-run JSON) ships with the checkpoints.

What we didn’t test

MSFT’s harness and Temporal’s durability-layer approach (wrapping Gemini/OpenAI Agents SDK/PydanticAI) are on the list for round two — this piece is about the two harnesses homelab users are actually picking between right now, tested the way nobody else tests them: on your hardware, with your model, with the filesystem as the referee.

The takeaway: the harness layer is not commodity. Out of the box, one popular harness will tell you it succeeded when it didn’t; the other does the work directly and costs almost nothing against a local model. If you self-host, run your own 5-task suite before you commit — it takes fifteen minutes and the default configs will surprise you.