Stop conditions, errors and cost control

How it works: stop conditions

1. Success: the model gives a final answer or calls a finish tool. The cleanest stop condition.
2. Step limit: a hard maximum number of loop turns (e.g. 10–50 depending on the task). Always set one.
3. Budgets: maximum tokens or estimated cost per task (a token budget), and a wall-clock time limit.
4. Loop detection: the same tool with the same arguments several times, or no new information over several steps, means stuck.
5. Unrecoverable errors: authentication failures, permission denied, invalid configuration.
6. Human stop: a cancel button the user can press at any time.
7. Stop gracefully: return what was done, the best partial answer, and what's missing or failed.

How it works: error handling and retries

1. Tool errors become tool results marked as errors with a useful message, so the model can fix arguments or change approach (tool calling).
2. Transient failures (timeouts, 429 rate limits, 5xx errors) are retried by the harness with exponential backoff and jitter (wait 1 s, 2 s, 4 s…), a few times at most (retries with backoff).
3. Model API errors are retried similarly; persistent ones can fall back to another model (fallbacks).
4. Actions must be safe to retry: use idempotency keys so a retried "send payment" doesn't pay twice.
5. Check the model's stop reason: a reply cut off by max_tokens or a refusal isn't a normal answer; handle it explicitly.
6. Count errors: too many in a row is itself a stop condition.

How it works: cost and loop control

1. Know where the money goes: every turn resends the whole context, so long histories and big tool results dominate cost (agent loop).
2. Set per-task budgets and stop (or ask the user) when exceeded.
3. Keep context lean: concise tool results, clear old results, compaction (context engineering).
4. Use prompt caching for the stable system prompt and tool definitions.
5. Use cheaper models for simple sub-steps, summaries and checks; reserve strong models for planning and hard reasoning.
6. Lower effort where deep reasoning isn't needed (reasoning models).
7. Monitor and alert on cost per task and runaway runs (cost per request).

import time
 
class LoopGuard:
    """Stop conditions for an agent loop: steps, tokens, time, and repeated actions."""
    def __init__(self, max_steps=8, max_tokens=20_000, max_seconds=60):
        self.max_steps, self.max_tokens, self.max_seconds = max_steps, max_tokens, max_seconds
        self.steps = self.tokens = 0
        self.start = time.monotonic()
        self.seen = []
    def check(self, action: str, tokens_used: int) -> str | None:
        self.steps += 1; self.tokens += tokens_used
        if self.steps > self.max_steps: return "max steps"
        if self.tokens > self.max_tokens: return "token budget"
        if time.monotonic() - self.start > self.max_seconds: return "time budget"
        if self.seen[-2:] == [action, action]: return "same action 3 times in a row"
        self.seen.append(action)
        return None
 
def call_tool_with_retry(fn, attempts=3, base_delay=0.01):
    for i in range(attempts):
        try:
            return fn(i)
        except TimeoutError:
            time.sleep(base_delay * 2 ** i)              # exponential backoff between tries
    return "error: tool unavailable, tell the user"
 
guard = LoopGuard()
actions = ["search('refund')", "search('refund')", "search('refund')"]
for a in actions:
    reason = guard.check(a, tokens_used=1_500)
    print(f"{a:18} -> {'STOP: ' + reason if reason else 'continue'}")
 
flaky = lambda i: (_ for _ in ()).throw(TimeoutError()) if i < 2 else "ok on try 3"
print("flaky tool:", call_tool_with_retry(flaky))

search('refund')   -> continue
search('refund')   -> continue
search('refund')   -> STOP: same action 3 times in a row
flaky tool: ok on try 3