Harness engineering: everything around the model

How it works: harness engineering

Harness engineering is designing every part of an agent harness deliberately. The usual parts:

1. Instructions: the system prompt and project instruction files (next section).
2. Tools and skills: the actions available (tool calling, MCP) and the procedures it can load (skills).
3. Context management: what the model sees each step, compaction, sub-agents (context engineering).
4. Permissions and sandbox: what the agent may read, write and run, where (least privilege).
5. Hooks and mechanical enforcement: code that runs at fixed points to block or fix things (section below).
6. Verification: tests, linters, type checks, evals that decide whether "done" is really done.
7. State and memory: plans, notes and checkpoints that survive long tasks and restarts (memory, checkpoints).
8. Observability: traces of every step, cost and outcomes, so you can see why it failed (tracing).
9. Feedback: turning observed failures into harness changes (a new rule, a new hook, a better tool), then measuring.

From prompt engineering to loop engineering

The field's vocabulary grew in layers, each wrapping the previous one:

None replaces the others: a good loop runs a well-harnessed agent, whose harness manages good context, which contains good prompts.

How it works: instruction files

1. An instruction file lives in the repository, commonly AGENTS.md (a cross-tool convention) or tool-specific names such as CLAUDE.md. Coding agents read it at the start of each session.
2. Put in it what a new teammate would need: how to install, build, test and lint; project structure; conventions; things never to touch; how to verify work.
3. Keep it short and specific: commands and rules, not essays. Every line costs context on every session.
4. Nest them where tools support it: a root file for the repo, smaller files in sub-folders for local rules.
5. Version and review it like code; update it when the agent repeatedly gets something wrong.

How it works: hooks and mechanical enforcement

1. A hook is code the harness runs automatically at a fixed point: before a tool call, after a file edit, when the agent says it's done, when a session starts or ends.
2. Block: a pre-tool hook refuses dangerous commands (rm -rf, force-push, editing protected paths), whatever the model intended.
3. Fix: a post-edit hook runs the formatter, so style never depends on the model remembering.
4. Verify: a "before done" hook runs tests and rejects completion if they fail.
5. Record: hooks log actions for audit.
6. Back it up with CI: linters, structural tests and eval gates enforce the same rules on every change, agent-made or human-made.
7. The principle: instructions guide; hooks, tests and CI enforce. Anything that must always hold should be enforced mechanically.

import re
 
# Part of a harness: rules the MODEL reads (instructions) and rules the HARNESS enforces (hooks).
AGENTS_MD = """# Project rules for coding agents
- Run `pytest -q` before saying a task is done.
- Never edit files in migrations/.
- Use the repo's formatter: `ruff format`."""
 
BLOCKED = [r"\brm\s+-rf\b", r"git\s+push\s+--force", r"\bDROP\s+TABLE\b"]
 
def pre_tool_hook(tool: str, args: dict) -> tuple[bool, str]:
    """Runs before EVERY tool call. Code, not a prompt, so the model can't talk its way past it."""
    if tool == "bash" and any(re.search(p, args["command"], re.I) for p in BLOCKED):
        return False, "blocked by policy: destructive command"
    if tool == "edit_file" and args["path"].startswith("migrations/"):
        return False, "blocked by policy: migrations are read-only"
    return True, "allowed"
 
def verify_done(test_exit_code: int) -> str:
    """A verification step the harness runs before accepting 'done'."""
    return "accept" if test_exit_code == 0 else "reject: tests fail, keep working"
 
for tool, args in [("bash", {"command": "pytest -q"}),
                   ("bash", {"command": "rm -rf /"}),
                   ("edit_file", {"path": "migrations/0007.sql"}),
                   ("edit_file", {"path": "app/routes.py"})]:
    print(f"{tool:9} {list(args.values())[0]:22} -> {pre_tool_hook(tool, args)[1]}")
print("model says done, tests exit 1 ->", verify_done(1))

bash      pytest -q              -> allowed
bash      rm -rf /               -> blocked by policy: destructive command
edit_file migrations/0007.sql    -> blocked by policy: migrations are read-only
edit_file app/routes.py          -> allowed
model says done, tests exit 1 -> reject: tests fail, keep working