Loop engineering: agents that run on repeat

How it works: loop engineering

Loop engineering designs a few building blocks:

1. Trigger: what starts a run. A schedule (every night), an event (CI failed, new issue, new PR comment), or a queue of tasks.
2. Task source: where work comes from (failing tests, a backlog label, alerts) and how each task is described to the agent.
3. The agent and its harness: tools, instructions, permissions and sandbox (harness engineering).
4. Verifier: a machine-checkable definition of "done" (tests pass, lint clean, eval score above a threshold, a reviewer agent approves).
5. Stop conditions and budgets: attempt caps, token and cost caps, time limits (stop conditions).
6. Human gates: where people review (usually the final output, like a pull request), never every step.
7. State and memory: what the loop records between runs (what was tried, what failed) so it doesn't repeat mistakes.
8. Observability: runs, pass rates, costs and failure reasons on a dashboard.

How it works: maker-checker loops

1. The maker (an agent) produces something: a code change, a document, a data fix.
2. The checker verifies it independently: run the tests, the linter, the type checker, an eval, or a separate reviewer agent with a strict rubric. Independence matters: a maker grading itself is too lenient (reflection).
3. On failure, the checker's specific feedback goes back to the maker for another attempt.
4. On success, the result moves to the human gate (e.g. a pull request).
5. The loop is bounded: a maximum number of attempts and a budget, then it reports failure with what it learned.

How it works: scheduled and event-triggered runs

1. Schedules: cron-style triggers for routine work (nightly dependency updates, weekly doc sync). CI systems (GitHub Actions schedule) and hosted agent platforms offer scheduling.
2. Events: webhooks and CI events (a failed build on main, a new issue with a label, a security alert) start a run with that event as the task.
3. Queues: tasks wait in a queue; workers pick them up with concurrency limits (message queues).
4. Idempotency and dedupe: the same event shouldn't start two runs on the same task.
5. Output where people already work: pull requests, ticket comments, chat messages.

How it works: long-running autonomous agents

1. Long tasks outgrow one context: keep a plan and a progress file (or checkpointed state) that records what's done, what's next and decisions made.
2. Work in small verified increments: change, test, commit; never hours of unverified work.
3. Resume cleanly: a new session (or a restarted process) reads the progress file and continues (context compaction, checkpoints).
4. Budget the whole run (tokens, time, money) and report progress regularly.
5. Stop at the gate: autonomous runs end in something a person reviews.

# A maker-checker loop: one agent proposes, an independent checker verifies, repeat until it passes.
def maker(task: str, feedback: str | None, attempt: int) -> str:
    """Stand-in for a coding agent. Each attempt uses the checker's feedback."""
    drafts = ["def add(a, b): return a - b",
              "def add(a, b): return a + b"]
    return drafts[min(attempt, 1)]
 
def checker(code: str) -> str | None:
    """Machine-checkable: run the tests. Returns None if they pass, else the failure."""
    scope = {}
    exec(code, scope)
    cases = [((2, 3), 5), ((-1, 1), 0)]
    for args, want in cases:
        got = scope["add"](*args)
        if got != want:
            return f"add{args} returned {got}, expected {want}"
    return None
 
def run_loop(task: str, max_attempts: int = 4) -> str:
    feedback = None
    for attempt in range(max_attempts):              # bounded: the loop always ends
        code = maker(task, feedback, attempt)
        feedback = checker(code)
        print(f"attempt {attempt + 1}: {'PASS' if feedback is None else 'FAIL - ' + feedback}")
        if feedback is None:
            return "open a pull request for human review"
    return "give up, report what failed"
 
print("->", run_loop("Implement add(a, b)"))

attempt 1: FAIL - add(2, 3) returned -1, expected 5
attempt 2: PASS
-> open a pull request for human review

name: nightly-fix-flaky-tests
on:
  schedule:
    - cron: "0 2 * * *"          # every day at 02:00 UTC
  workflow_dispatch: {}          # also allow manual runs
jobs:
  loop:
    runs-on: ubuntu-latest
    timeout-minutes: 60          # hard time budget for the whole loop
    steps:
      - uses: actions/checkout@v4
      - run: python scripts/agent_loop.py --task flaky-tests --max-attempts 4 --max-cost-usd 5
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}