Field Notes · On Harnesses
The hardest problem in AI-assisted coding isn’t picking a model. It’s engineering the loop around it.
The hardest problem in AI-assisted coding isn’t picking a model. It’s engineering the loop around it. It will soon be quaint to ask which model is best? As reasoning and tool-use models converge in capabilities, the model matters less than the harness you build for it. This month alone: Anthropic released Fable, yanked it because guardrails ‘could not be trusted,’ then re-released it. OpenAI dropped three frontier models just this week. And Meta shipped its first agentic model. Yet agentic coding issues continue.
A strong model in a weak system is unruly and expensive. A weaker model in a disciplined one ships useful work, at lower cost and lower risk.
Yes, the model matters. But the system around it increasingly matters more. A strong model inside a weak system can be unruly and expensive, or even downright destructive. A weaker model in a disciplined system ships more useful work, at lower cost and lower risk.
That’s why two IDEs running the same model feel nothing alike. One reads the right files, makes a focused change, runs the relevant tests, and reports uncertainty. The other burns context, edits unrelated code, retries failed approaches, and declares victory.
The difference isn’t the model. It’s the harness.
Benchmarks measure what a model can do under controlled conditions. The same model posts different scores on the same benchmark depending on the platform. But developers don’t care about marketing numbers. They care what a system does inside a messy repository.
So the competitive edge isn’t access to the smartest model. It’s the best execution environment around whatever model you have.
Prompting is a single request. Loop engineering is the operating system around the request.
Developers need to stop treating agents like chatbots that need elaborate instructions. A giant spec is not a workflow. It’s an undocumented program with weak control flow and no failure handling.
Agents have their own ADHD. They keep working after the useful work is done: refactoring unrelated code, rewriting tests, “cleaning up.” You know the moment. You set the task up perfectly, stepped away from the terminal, and came back to a WTF?! diff.
More prompting doesn’t fix this. Boundaries do.
Stop prompting a model to behave. An agent without a harness is a
driver on a road with no lanes, no signs, no speed limit —
just a post-it called AGENTS.md. A harness paints road
lanes, puts up stoplights, adds a highway patrol. Imagine driving in
a foreign country where roads didn’t have signs or markers. Do
you drive on the left or the right, or is it all one road?
A harness loop defines the task, gathers context, plans, makes a bounded change, runs checks, inspects failures, retries selectively, records decisions, and STOPS at a success condition or budget limit. Instructions should not just tell agents “Do not do.” A harness should detect and stop changes. It should restrict paths, inspect diffs, enforce scope, and reject work that violates a contract.
No harness → a driver on a road with no lanes, no signs, no
speed limit. Just a post-it called AGENTS.md.
A harness paints the lanes, puts up the stoplights, and adds the highway patrol.
My Python harness and the TypeScript port are wrappers around a bash loop with a hard timeout. None of it is clever. It enforces boring coding standards humans try to follow — and agents do, when guardrails are set. Like most software, it’s a work in progress.
Agents generate code faster than humans can verify it. The bottleneck has moved from generation to diff review, and we’re accruing verification debt — like technical debt, except it piles up before you’ve proven anything works. A 1,000-line agent patch compiles. It passes the tests the agent wrote. Neither proves intended behavior. Maybe it misunderstood the requirement. Mocked away the hard part. Quietly changed a public interface.
The cost is measurable. In METR’s trial, experienced open-source developers took 19% longer with AI tools while thinking they’d been 20% faster. Review sucks time. But going fast on a merry-go-round isn’t speed.
And the backlash against “AI slop” is earned. Five agents don’t produce five times the output. They produce conflicting abstractions and five times the review burden. Engineers were never graded on lines of code. There is elegance in less. Yet “tasks completed” is now a boast, and a dangerous metric.
Verification belongs inside the loop: executable acceptance criteria, independent tests, static analysis, type checks, security scans, and human review proportional to risk.
Experienced open-source developers, using AI tools, took 19% longer — while believing they’d been 20% faster. The review burden negates the speed gain.
The failure that should make you nervous isn’t one bad patch. It’s cumulative degradation. An agent solves today’s narrow task while making the codebase harder to maintain. Every change is individually defensible. The sum? Entropy.
You see it when agents operate without context. They recreate functions that already exist because they never looked. They route around architectural constraints they don’t understand. They add dependencies for problems an installed library already solves.
Fixing this takes more than updating AGENTS.md. The
workflow is controlled delegation — a harness with local
quality gates:
Humans build the harness. Agents work inside it. The harness manages context, tools, permissions, budgets, checkpoints, and verification. That makes engineering the loop central work.
The best developers won’t write the most code. They’ll build the loops that write the best code. The model is rented. Your harness is an asset.
No harness? Big problem. Run the audit:
Originally published on
dev.to. Reference harness implementations, in Python and TypeScript, are open
source:
rxdt/loopgate_harness.
More writing:
A Frontend Loop Needs a Real App
·
The First (and Last) Intent-Inference Conference.
Back to Rox dT