Most multi-agent systems work in one of two ways. A lead agent calls helpers one at a time and waits on every report, or an engineer hard-codes a pipeline that runs the same steps for every job. A newer approach, dynamic workflow introduced by Anthropic, lets the lead write the whole plan once as a short program, and plain code runs it with many workers in parallel.

Here is the architecture we are going to build, one component at a time:

• A benchmark: 72 planted bugs and a scorer that checks findings against a hidden key.
• An answer ledger: caching, replay and crash recovery.
• A runtime: workers that run in parallel phases under one budget.
• An audit kit: scan workers, and skeptics that vote on what they find.
• A plan language: restricted Python, checked before it can spend.
• The lead: it reads a 500-token fact sheet instead of the code.
• An evaluation: five designs and an ablation.

By the end, a lead that reads only that sheet finds 0.45 of the planted bugs on average, against 0.30 for a hand-written plan and 0.02 for one agent alone. The full code is in my GitHub repository, dynamic-workflow-agents.

These are the files we are going to write:

dynamic-workflows/
├── build/
│   ├── config.py     # model roles, prices, paths
│   ├── bench.py      # the hidden key and the scorer
│   ├── shards.py     # code packed into shards
│   ├── llm.py        # replies, ledger, front door, budget
│   ├── runtime.py    # threads, parallel, background runs
│   ├── kit.py        # scan, check, vote and report
│   ├── plans.py      # whitelist, dry run, approval gate
│   ├── lead.py       # fact sheet in, program out
│   └── prompts.json  # the instructions each role reads
└── data/algorithms/  # the library, the bugs and the key

dynamic-workflows/
├── build/
│   ├── config.py     # model roles, prices, paths
│   ├── bench.py      # the hidden key and the scorer
│   ├── shards.py     # code packed into shards
│   ├── llm.py        # replies, ledger, front door, budget
│   ├── runtime.py    # threads, parallel, background runs
│   ├── kit.py        # scan, check, vote and report
│   ├── plans.py      # whitelist, dry run, approval gate
│   ├── lead.py       # fact sheet in, program out
│   └── prompts.json  # the instructions each role reads
└── data/algorithms/  # the library, the bugs and the key

Why Multi-Agent Systems Often Fail

Most failures come from two limits. One agent can hold only so much text, and on a large job it forgets what it read or stops early. A lead that hands out tasks one by one becomes the bottleneck, and its context grows with every report.

Our audit shows both: 849 Python files, about 1.35 million tokens, with 72 bugs I planted myself. One agent alone found 0, 1 and 4 of them in three runs and stopped on its own each time. A lead with helpers covered all 14 shards but found only 12 bugs.

Designing Dynamic Workflows

Every job goes through six steps:

• Scout: plain code reads the job and writes a one-page fact sheet.
• Plan: the lead writes a short Python program from that sheet.
• Check: a whitelist refuses whatever a plan may not do.
• Rehearse: a free stand-in model runs the plan once to price it.
• Approve: a gate lets the plan go or holds it for a person.
• Run: the runtime executes the plan in the background under one budget.

Before the comparison runs I wrote down eleven predictions. The evaluation checks five of them:

• H1: the workflow beats one agent alone.
• H1b: it also beats one agent given the same money.
• H5: a second round of scanning adds bugs.
• H4: skeptics raise precision by at least 0.15 and cost at most 0.05 recall.
• H6: running a finding’s failing call makes the skeptic more accurate.

Environment Setup

We start from an empty project with one package, jsonschema, and any OpenAI-compatible endpoint.

uv init dynamic-workflows --python 3.12 && cd dynamic-workflows
uv add jsonschema
export LLM_BASE_URL="https://your-endpoint/v1"   # any OpenAI-compatible server
export LLM_API_KEY="your key"                    # read once, never printed

uv init dynamic-workflows --python 3.12 && cd dynamic-workflows
uv add jsonschema
export LLM_BASE_URL="https://your-endpoint/v1"   # any OpenAI-compatible server
export LLM_API_KEY="your key"                    # read once, never printed

MODELS gives each role its model, and PRICE` is the cost of one call in credits.

"""config.py: the settings every other file reads."""
import json, os
from pathlib import Path

CAP_CREDITS = 1500.0                                     # the cap of a front door given no budget of its own
BASE_URL = os.environ.get("LLM_BASE_URL", "http://127.0.0.1:8000/v1")      # any OpenAI-compatible endpoint
API_KEY = os.environ.get("LLM_API_KEY", "")              # read once from the environment, never printed
MODELS = {"lead": "Qwen/Qwen3.8-Max", "worker": "Qwen/Qwen3.8-27B",        # the code asks for a role, not a model
          "judge": "Qwen/Qwen3.7-Plus", "fast": "Qwen/Qwen3.8-Flash-Next"}
PRICE = {"Qwen/Qwen3.8-Max": 0.5, "Qwen/Qwen3.8-27B": 0.2, "Qwen/Qwen3.7-Plus": 0.2, "Qwen/Qwen3.8-Flash-Next": 0.2}   # credits per call, whatever its length
TEMPERATURE, TOP_P = 0.6, 0.95                           # reasoning models loop under greedy decoding, so we sample
SHARD_TOKENS, CHARS_PER_TOKEN = 100_000, 2.54542         # code per worker call, and characters per token (both measured later)
MAX_AGENTS_PER_RUN = 1000                                # a backstop against a plan that loops forever
DATA = Path("data/algorithms")                           # the library, its planted bugs and the answer key
PROMPTS = json.loads((Path(__file__).parent / "prompts.json").read_text(encoding="utf-8"))      # the instructions, one text per role

"""config.py: the settings every other file reads."""
import json, os
from pathlib import Path

CAP_CREDITS = 1500.0                                     # the cap of a front door given no budget of its own
BASE_URL = os.environ.get("LLM_BASE_URL", "http://127.0.0.1:8000/v1")      # any OpenAI-compatible endpoint
API_KEY = os.environ.get("LLM_API_KEY", "")              # read once from the environment, never printed
MODELS = {"lead": "Qwen/Qwen3.8-Max", "worker": "Qwen/Qwen3.8-27B",        # the code asks for a role, not a model
          "judge": "Qwen/Qwen3.7-Plus", "fast": "Qwen/Qwen3.8-Flash-Next"}
PRICE = {"Qwen/Qwen3.8-Max": 0.5, "Qwen/Qwen3.8-27B": 0.2, "Qwen/Qwen3.7-Plus": 0.2, "Qwen/Qwen3.8-Flash-Next": 0.2}   # credits per call, whatever its length
TEMPERATURE, TOP_P = 0.6, 0.95                           # reasoning models loop under greedy decoding, so we sample
SHARD_TOKENS, CHARS_PER_TOKEN = 100_000, 2.54542         # code per worker call, and characters per token (both measured later)
MAX_AGENTS_PER_RUN = 1000                                # a backstop against a plan that loops forever
DATA = Path("data/algorithms")                           # the library, its planted bugs and the answer key
PROMPTS = json.loads((Path(__file__).parent / "prompts.json").read_text(encoding="utf-8"))      # the instructions, one text per role

Planted-Bug Benchmark

I took a public library of 849 Python files and planted one bug in each of 72 files. Each bug makes at least one of the file’s doctest examples fail, and never more than 40% of them. The copy that agents read has those examples removed.

• left out: the twelve kinds of one-line edit, the doctest proof and the leak test (the repository’s notebook has them)

"""bench.py: the files, the hidden key and the scorer."""
import ast, functools, json, re

from config import DATA

AUDITED = DATA / "audited"                                    # the copy of the library that agents read
WORK = sorted(p.relative_to(AUDITED).as_posix() for p in AUDITED.rglob("*.py"))     # every file an audit must read
KEY = json.loads((DATA / "hidden" / "answer_key.json").read_text(encoding="utf-8"))   # the 72 planted bugs, in a folder no agent tool opens

"""bench.py: the files, the hidden key and the scorer."""
import ast, functools, json, re

from config import DATA

AUDITED = DATA / "audited"                                    # the copy of the library that agents read
WORK = sorted(p.relative_to(AUDITED).as_posix() for p in AUDITED.rglob("*.py"))     # every file an audit must read
KEY = json.loads((DATA / "hidden" / "answer_key.json").read_text(encoding="utf-8"))   # the 72 planted bugs, in a folder no agent tool opens

The scorer must know which function a line belongs to.

<button data-code="# bench.py
@functools.lru_cache(maxsize=None)
def audited_tree(path):
# parsed once, then reused
return ast.parse((AUDITED / path).read_text(encoding="utf-8"))

def function_at(path, line):
"""(name, first line, last line) of the innermost function around a line, or None."""
best = None

def walk(node, prefix):
nonlocal best
for child in ast.iter_child_nodes(node):
if isinstance(child, (ast.FunctionDef, ast.AsyncFunctionDef)):
# the innermost function holding the line wins
if child.lineno <= line = best[1]):
best = (prefix + child.name, child.lineno, child.end_lineno)
walk(child, prefix + child.name + ".")
else: # methods get their class name
walk(child, prefix + child.name + "." if isinstance(child, ast.ClassDef) else prefix)
if path in WORK: # a path a model made up is never opened
walk(audited_tree(path), "")
return best” data-toggle-ms=”1200″ aria-label=”Copy code” data-nosnippet=”” class=”code-copy-btn absolute right-3 top-3 size-8 p-1.5 flex items-center justify-center bg-black/50 text-white rounded-md transition-colors duration-200 cursor-pointer hover:bg-black/70″>

# bench.py
@functools.lru_cache(maxsize=None)
def audited_tree(path):
    # parsed once, then reused
    return ast.parse((AUDITED / path).read_text(encoding="utf-8"))

def function_at(path, line):
    """(name, first line, last line) of the innermost function around a line, or None."""
    best = None

    def walk(node, prefix):
        nonlocal best
        for child in ast.iter_child_nodes(node):
            if isinstance(child, (ast.FunctionDef, ast.AsyncFunctionDef)):
                # the innermost function holding the line wins
                if child.lineno <= line <= child.end_lineno and (best is None or child.lineno >= best[1]):
                    best = (prefix + child.name, child.lineno, child.end_lineno)
                walk(child, prefix + child.name + ".")
            else:                                             # methods get their class name
                walk(child, prefix + child.name + "." if isinstance(child, ast.ClassDef) else prefix)
    if path in WORK:                                          # a path a model made up is never opened
        walk(audited_tree(path), "")
    return best

# bench.py
@functools.lru_cache(maxsize=None)
def audited_tree(path):
    # parsed once, then reused
    return ast.parse((AUDITED / path).read_text(encoding="utf-8"))

def function_at(path, line):
    """(name, first line, last line) of the innermost function around a line, or None."""
    best = None

    def walk(node, prefix):
        nonlocal best
        for child in ast.iter_child_nodes(node):
            if isinstance(child, (ast.FunctionDef, ast.AsyncFunctionDef)):
                # the innermost function holding the line wins
                if child.lineno <= line <= child.end_lineno and (best is None or child.lineno >= best[1]):
                    best = (prefix + child.name, child.lineno, child.end_lineno)
                walk(child, prefix + child.name + ".")
            else:                                             # methods get their class name
                walk(child, prefix + child.name + "." if isinstance(child, ast.ClassDef) else prefix)
    if path in WORK:                                          # a path a model made up is never opened
        walk(audited_tree(path), "")
    return best

A finding counts when it names the right file and function and lands within five lines of the bug:

match applies it to a list of findings.

0 else None
said = f.get("function", "").split(".")[-1] # the name the model gave, as a second chance
hits = [b for b in key if b["path"] == path and abs(line – b["line"])

# bench.py
def normalise_path(path):
    # ./x, /x and .../audited/x all mean x
    return re.sub(r"^(./|/|.*audited/)+", "", re.sub(r"/+", "/", str(path).strip().replace("\", "/")))

def match(findings, key=KEY, window=5):
    """A finding counts when it names the same file and function as a planted bug, within `window` lines."""
    found, wrong = {}, set()
    for f in findings:
        path, line = normalise_path(f["file"]), int(f.get("line") or 0)
        span = function_at(path, line) if line > 0 else None
        said = f.get("function", "").split(".")[-1]                                # the name the model gave, as a second chance
        hits = [b for b in key if b["path"] == path and abs(line - b["line"]) <= window
                and ((span and span[0] == b["function"]) or (said and said == b["function"].split(".")[-1]))]
        if hits:
            found.setdefault(min(hits, key=lambda b: abs(line - b["line"]))["id"], f)     # the nearest bug wins, and saying it twice counts once
        else:
            wrong.add((path, span[0] if span else f"line {line // 10}"))           # one wrong idea per place, however often repeated
    # recall over the key, precision over what was reported
    tp, fp = len(found), len(wrong)
    return {"found": sorted(found), "recall": tp / len(key), "precision": tp / (tp + fp) if tp + fp else None, "tp": tp, "fp": fp}

# bench.py
def normalise_path(path):
    # ./x, /x and .../audited/x all mean x
    return re.sub(r"^(./|/|.*audited/)+", "", re.sub(r"/+", "/", str(path).strip().replace("\", "/")))

def match(findings, key=KEY, window=5):
    """A finding counts when it names the same file and function as a planted bug, within `window` lines."""
    found, wrong = {}, set()
    for f in findings:
        path, line = normalise_path(f["file"]), int(f.get("line") or 0)
        span = function_at(path, line) if line > 0 else None
        said = f.get("function", "").split(".")[-1]                                # the name the model gave, as a second chance
        hits = [b for b in key if b["path"] == path and abs(line - b["line"]) <= window
                and ((span and span[0] == b["function"]) or (said and said == b["function"].split(".")[-1]))]
        if hits:
            found.setdefault(min(hits, key=lambda b: abs(line - b["line"]))["id"], f)     # the nearest bug wins, and saying it twice counts once
        else:
            wrong.add((path, span[0] if span else f"line {line // 10}"))           # one wrong idea per place, however often repeated
    # recall over the key, precision over what was reported
    tp, fp = len(found), len(wrong)
    return {"found": sorted(found), "recall": tp / len(key), "precision": tp / (tp + fp) if tp + fp else None, "tp": tp, "fp": fp}

Answer Ledger

I rerun model calls constantly. I store each reply under a fingerprint of its question and never pay for the same question twice.

A Reply holds one answer.

"""llm.py: replies, the ledger, one front door and one budget."""
import hashlib, json, re, sqlite3, threading, time, urllib.request, zlib
from dataclasses import asdict, dataclass

from jsonschema import Draft202012Validator

from config import API_KEY, BASE_URL, CAP_CREDITS, PRICE

@dataclass
class Reply:
    text: str = ""                 # the answer
    finish: str = None             # stop, length or error
    prompt_tokens: int = 0         # as the server counted them
    completion_tokens: int = 0
    seconds: float = 0.0
    model: str = ""
    cached: bool = False           # True when the ledger answered instead of the network

class CacheMiss(RuntimeError):
    """Replay met a question the ledger has never seen."""

class SpendRefused(RuntimeError):
    """A paid call was refused: there is no client to pay with."""

"""llm.py: replies, the ledger, one front door and one budget."""
import hashlib, json, re, sqlite3, threading, time, urllib.request, zlib
from dataclasses import asdict, dataclass

from jsonschema import Draft202012Validator

from config import API_KEY, BASE_URL, CAP_CREDITS, PRICE

@dataclass
class Reply:
    text: str = ""                 # the answer
    finish: str = None             # stop, length or error
    prompt_tokens: int = 0         # as the server counted them
    completion_tokens: int = 0
    seconds: float = 0.0
    model: str = ""
    cached: bool = False           # True when the ledger answered instead of the network

class CacheMiss(RuntimeError):
    """Replay met a question the ledger has never seen."""

class SpendRefused(RuntimeError):
    """A paid call was refused: there is no client to pay with."""

http_client is the only code that touches the network.

# llm.py
def http_client(model, messages, **params):
    """One chat completion over plain HTTP. A failed call raises, so it is never stored."""
    body = json.dumps({"model": model, "messages": messages, **params}).encode()          # the question in chat completions form
    head = {"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"}   # the key travels in a header only
    start = time.perf_counter()
    # one POST, ten minutes at most
    with urllib.request.urlopen(urllib.request.Request(BASE_URL + "/chat/completions", body, head), timeout=600) as r:
        data = json.load(r)
    choice, use = data["choices"][0], data["usage"]          # the answer and the server's token counts
    return Reply(choice["message"].get("content") or "", choice["finish_reason"], use["prompt_tokens"], use["completion_tokens"],
                 time.perf_counter() - start, model)

# llm.py
def http_client(model, messages, **params):
    """One chat completion over plain HTTP. A failed call raises, so it is never stored."""
    body = json.dumps({"model": model, "messages": messages, **params}).encode()          # the question in chat completions form
    head = {"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"}   # the key travels in a header only
    start = time.perf_counter()
    # one POST, ten minutes at most
    with urllib.request.urlopen(urllib.request.Request(BASE_URL + "/chat/completions", body, head), timeout=600) as r:
        data = json.load(r)
    choice, use = data["choices"][0], data["usage"]          # the answer and the server's token counts
    return Reply(choice["message"].get("content") or "", choice["finish_reason"], use["prompt_tokens"], use["completion_tokens"],
                 time.perf_counter() - start, model)

The fingerprint includes a sample index: samples 0 and 1 of one question get independent answers.

# llm.py
class Ledger:
    """Every answer, under a fingerprint of its question."""

    def __init__(self, path):
        # one row per answer, keyed by fingerprint
        self.db, self.lock = sqlite3.connect(str(path), check_same_thread=False), threading.Lock()
        self.db.execute("CREATE TABLE IF NOT EXISTS calls (key TEXT PRIMARY KEY, role, reply BLOB, seconds REAL, credits REAL, created REAL)")

    @staticmethod
    def key(model, messages, params, sample=0):
        blob = json.dumps([model, messages, params, sample], sort_keys=True)     # the question, exactly as it was asked
        return hashlib.sha256(blob.encode()).hexdigest()

    def get(self, key):
        with self.lock:
            row = self.db.execute("SELECT reply FROM calls WHERE key = ?", (key,)).fetchone()
        # stored replies come back marked cached
        return Reply(**{**json.loads(zlib.decompress(row[0])), "cached": True}) if row else None

    def put(self, key, reply, role="", credits=0.0):
        with self.lock:
            self.db.execute("INSERT OR REPLACE INTO calls VALUES (?,?,?,?,?,?)",
                            (key, role, zlib.compress(json.dumps(asdict(reply)).encode()), reply.seconds, credits, time.time()))
            self.db.commit()                          # a crash loses at most the calls still in flight

# llm.py
class Ledger:
    """Every answer, under a fingerprint of its question."""

    def __init__(self, path):
        # one row per answer, keyed by fingerprint
        self.db, self.lock = sqlite3.connect(str(path), check_same_thread=False), threading.Lock()
        self.db.execute("CREATE TABLE IF NOT EXISTS calls (key TEXT PRIMARY KEY, role, reply BLOB, seconds REAL, credits REAL, created REAL)")

    @staticmethod
    def key(model, messages, params, sample=0):
        blob = json.dumps([model, messages, params, sample], sort_keys=True)     # the question, exactly as it was asked
        return hashlib.sha256(blob.encode()).hexdigest()

    def get(self, key):
        with self.lock:
            row = self.db.execute("SELECT reply FROM calls WHERE key = ?", (key,)).fetchone()
        # stored replies come back marked cached
        return Reply(**{**json.loads(zlib.decompress(row[0])), "cached": True}) if row else None

    def put(self, key, reply, role="", credits=0.0):
        with self.lock:
            self.db.execute("INSERT OR REPLACE INTO calls VALUES (?,?,?,?,?,?)",
                            (key, role, zlib.compress(json.dumps(asdict(reply)).encode()), reply.seconds, credits, time.time()))
            self.db.commit()                          # a crash loses at most the calls still in flight

Every call goes through one front door, LLM.

# llm.py
class LLM:
    def __init__(self, ledger, mode="collect", client=None, scripted=None, budget=None):
        self.ledger, self.mode, self.client, self.scripted = ledger, mode, client, scripted
        self.budget = budget or Budget(CAP_CREDITS)

# llm.py
class LLM:
    def __init__(self, ledger, mode="collect", client=None, scripted=None, budget=None):
        self.ledger, self.mode, self.client, self.scripted = ledger, mode, client, scripted
        self.budget = budget or Budget(CAP_CREDITS)

chat tries, in order: the ledger, replay refusal, a free stand-in, the budget, and only then the network.

# llm.py
    def chat(self, model, messages, *, role="", sample=0, **params):
        key = Ledger.key(model, messages, params, sample)      # the same question always gets the same key
        if hit := self.ledger.get(key):
            return hit                                # free, instant and identical to the first answer
        if self.mode == "replay":
            raise CacheMiss(f"{role or model}: no stored answer, run in collect mode first")
        if self.scripted:                             # a stand-in answers for free and leaves no trace
            return self.scripted(model, messages, dict(params, sample=sample))
        if self.client is None:                       # no client means nothing can be paid for
            raise SpendRefused("no client, so nothing can be collected")
        price = PRICE.get(model, 0.2)
        self.budget.admit(price)                      # may wait here until somebody raises the cap
        try:
            reply = self.client(model, messages, **params)
        except BaseException:
            self.budget.settle(price, 0.0)            # a failed call is not billed and not stored
            raise
        self.budget.settle(price, price)
        self.ledger.put(key, reply, role, price)
        return reply

# llm.py
    def chat(self, model, messages, *, role="", sample=0, **params):
        key = Ledger.key(model, messages, params, sample)      # the same question always gets the same key
        if hit := self.ledger.get(key):
            return hit                                # free, instant and identical to the first answer
        if self.mode == "replay":
            raise CacheMiss(f"{role or model}: no stored answer, run in collect mode first")
        if self.scripted:                             # a stand-in answers for free and leaves no trace
            return self.scripted(model, messages, dict(params, sample=sample))
        if self.client is None:                       # no client means nothing can be paid for
            raise SpendRefused("no client, so nothing can be collected")
        price = PRICE.get(model, 0.2)
        self.budget.admit(price)                      # may wait here until somebody raises the cap
        try:
            reply = self.client(model, messages, **params)
        except BaseException:
            self.budget.settle(price, 0.0)            # a failed call is not billed and not stored
            raise
        self.budget.settle(price, price)
        self.ledger.put(key, reply, role, price)
        return reply

We test it with a fake client that counts calls.

<button data-code="import time
from config import MODELS
from llm import LLM, Ledger, Reply

paid = [] # one entry per call that reaches the "network"
def fake_client(model, messages, **params): # a stand-in for the network: it answers after a pause
paid.append(1)
time.sleep(0.05)
return Reply('{"answer": "ok"}', "stop", 10, 5, 0.05, model)

llm = LLM(Ledger(":memory:"), client=fake_client)
ask = lambda door, q: door.chat(MODELS["worker"], [{"role": "user", "content": q}], role="demo")
for q in ("what is 2+2?", "name a sorting algorithm", "what is 2+2?"):
print(f"{q:

import time
from config import MODELS
from llm import LLM, Ledger, Reply

paid = []                                           # one entry per call that reaches the "network"
def fake_client(model, messages, **params):         # a stand-in for the network: it answers after a pause
    paid.append(1)
    time.sleep(0.05)
    return Reply('{"answer": "ok"}', "stop", 10, 5, 0.05, model)

llm = LLM(Ledger(":memory:"), client=fake_client)
ask = lambda door, q: door.chat(MODELS["worker"], [{"role": "user", "content": q}], role="demo")
for q in ("what is 2+2?", "name a sorting algorithm", "what is 2+2?"):
    print(f"{q:<26} from the ledger: {ask(llm, q).cached}")
try:
    ask(LLM(llm.ledger, mode="replay", client=fake_client), "a new question")
except Exception as e:
    print(f"{type(e).__name__}: {e}")

#### OUTPUT ####
what is 2+2?               from the ledger: False
name a sorting algorithm   from the ledger: False
what is 2+2?               from the ledger: True
CacheMiss: demo: no stored answer, run in collect mode first

import time
from config import MODELS
from llm import LLM, Ledger, Reply

paid = []                                           # one entry per call that reaches the "network"
def fake_client(model, messages, **params):         # a stand-in for the network: it answers after a pause
    paid.append(1)
    time.sleep(0.05)
    return Reply('{"answer": "ok"}', "stop", 10, 5, 0.05, model)

llm = LLM(Ledger(":memory:"), client=fake_client)
ask = lambda door, q: door.chat(MODELS["worker"], [{"role": "user", "content": q}], role="demo")
for q in ("what is 2+2?", "name a sorting algorithm", "what is 2+2?"):
    print(f"{q:<26} from the ledger: {ask(llm, q).cached}")
try:
    ask(LLM(llm.ledger, mode="replay", client=fake_client), "a new question")
except Exception as e:
    print(f"{type(e).__name__}: {e}")

#### OUTPUT ####
what is 2+2?               from the ledger: False
name a sorting algorithm   from the ledger: False
what is 2+2?               from the ledger: True
CacheMiss: demo: no stored answer, run in collect mode first

The repeat came from the ledger for free, and replay raised CacheMiss without touching the network.

Shard Packer

Big shards look cheap because billing is per call, yet a model can miss details past the start of a long input. A shard is the code that one worker reads in a single call. numbered adds line numbers, and TOKENS sizes each file as a worker sees it.

5}| {ln}" for i, ln in enumerate(lines, 1))

TOKENS = {p: int(len(numbered(p)) / CHARS_PER_TOKEN) for p in WORK} # each file's size as a worker sees it” data-toggle-ms=”1200″ aria-label=”Copy code” data-nosnippet=”” class=”code-copy-btn absolute right-3 top-3 size-8 p-1.5 flex items-center justify-center bg-black/50 text-white rounded-md transition-colors duration-200 cursor-pointer hover:bg-black/70″>

"""shards.py: numbered lines, packed into shards."""
from bench import AUDITED, WORK
from config import CHARS_PER_TOKEN

def numbered(path):
    lines = (AUDITED / path).read_text(encoding="utf-8").split("n")
    # a path header, then numbered lines
    return f"### {path}n" + "n".join(f"{i:>5}| {ln}" for i, ln in enumerate(lines, 1))

TOKENS = {p: int(len(numbered(p)) / CHARS_PER_TOKEN) for p in WORK}      # each file's size as a worker sees it

"""shards.py: numbered lines, packed into shards."""
from bench import AUDITED, WORK
from config import CHARS_PER_TOKEN

def numbered(path):
    lines = (AUDITED / path).read_text(encoding="utf-8").split("n")
    # a path header, then numbered lines
    return f"### {path}n" + "n".join(f"{i:>5}| {ln}" for i, ln in enumerate(lines, 1))

TOKENS = {p: int(len(numbered(p)) / CHARS_PER_TOKEN) for p in WORK}      # each file's size as a worker sees it

pack fills each shard with whole files until the next one would not fit.

<button data-code="# shards.py
def pack(paths, target):
"""Whole files packed into shards of about `target` tokens, first fit."""
shards, sizes = [], []
# folder by folder, biggest files first
for folder in sorted({p.split("/")[0] for p in paths}):
for p in sorted((q for q in paths if q.split("/")[0] == folder), key=lambda q: -TOKENS[q]):
# the first shard with room takes the file
for i, size in enumerate(sizes):
if size + TOKENS[p]

# shards.py
def pack(paths, target):
    """Whole files packed into shards of about `target` tokens, first fit."""
    shards, sizes = [], []
    # folder by folder, biggest files first
    for folder in sorted({p.split("/")[0] for p in paths}):
        for p in sorted((q for q in paths if q.split("/")[0] == folder), key=lambda q: -TOKENS[q]):
            # the first shard with room takes the file
            for i, size in enumerate(sizes):
                if size + TOKENS[p] <= target:
                    shards[i].append(p)
                    sizes[i] += TOKENS[p]
                    break
            else:                                    # no room anywhere: start a new shard
                shards.append([p])
                sizes.append(TOKENS[p])
    return shards

# shards.py
def pack(paths, target):
    """Whole files packed into shards of about `target` tokens, first fit."""
    shards, sizes = [], []
    # folder by folder, biggest files first
    for folder in sorted({p.split("/")[0] for p in paths}):
        for p in sorted((q for q in paths if q.split("/")[0] == folder), key=lambda q: -TOKENS[q]):
            # the first shard with room takes the file
            for i, size in enumerate(sizes):
                if size + TOKENS[p] <= target:
                    shards[i].append(p)
                    sizes[i] += TOKENS[p]
                    break
            else:                                    # no room anywhere: start a new shard
                shards.append([p])
                sizes.append(TOKENS[p])
    return shards

My rule, fixed in advance, takes the largest size within 0.05 of the best recall, and on 283 files with 24 planted bugs it picked 100,000 tokens. A later whole-system test disagreed: a slim plan found 0.38 of the bugs at 20,000-token shards and 0.18 at 100,000.

Every workflow compared below used 100,000-token shards.

Worker Threads and Event Stream

A run is one execution of a plan, and a plan is a Python function that receives the run.

"""runtime.py: workers, threads, events, phases and background runs."""
import collections, concurrent.futures as cf, threading, time
from dataclasses import dataclass, field

from config import MAX_AGENTS_PER_RUN, MODELS, PRICE, TEMPERATURE, TOP_P
from llm import CacheMiss, SpendRefused, extract_json, validate

class RunCancelled(RuntimeError):
    """The run was cancelled."""

ABORT = (RunCancelled, SpendRefused, CacheMiss)    # only these end a run, other failures stay with the worker

"""runtime.py: workers, threads, events, phases and background runs."""
import collections, concurrent.futures as cf, threading, time
from dataclasses import dataclass, field

from config import MAX_AGENTS_PER_RUN, MODELS, PRICE, TEMPERATURE, TOP_P
from llm import CacheMiss, SpendRefused, extract_json, validate

class RunCancelled(RuntimeError):
    """The run was cancelled."""

ABORT = (RunCancelled, SpendRefused, CacheMiss)    # only these end a run, other failures stay with the worker

Each worker leaves a thread, its whole conversation.

# runtime.py
@dataclass
class Thread:
    """One worker's whole conversation, kept so a person can read it afterwards."""
    id: str
    phase: str
    label: str                                     # what the plan called it, such as "scan:logic:maths#3"
    model: str
    role: str = "worker"
    messages: list = field(default_factory=list)   # everything the worker was sent
    replies: list = field(default_factory=list)    # every Reply it got back
    status: str = "waiting"                        # waiting, running, done or failed
    started: float = 0.0                           # seconds since the run began
    ended: float = 0.0
    cost: float = 0.0                              # credits charged for its paid calls
    result: object = None
    error: str = None

# runtime.py
@dataclass
class Thread:
    """One worker's whole conversation, kept so a person can read it afterwards."""
    id: str
    phase: str
    label: str                                     # what the plan called it, such as "scan:logic:maths#3"
    model: str
    role: str = "worker"
    messages: list = field(default_factory=list)   # everything the worker was sent
    replies: list = field(default_factory=list)    # every Reply it got back
    status: str = "waiting"                        # waiting, running, done or failed
    started: float = 0.0                           # seconds since the run began
    ended: float = 0.0
    cost: float = 0.0                              # credits charged for its paid calls
    result: object = None
    error: str = None

Run keeps everything a run leaves.

# runtime.py
class Run:
    def __init__(self, title, goal="", *, llm, run_id=None, concurrency=16, max_agents=MAX_AGENTS_PER_RUN):
        self.title, self.goal, self.llm, self.run_id = title, goal, llm, run_id or title.lower().replace(" ", "-")
        self.t0, self.max_agents, self.lock = time.perf_counter(), max_agents, threading.Lock()
        # slots cap concurrency, cancelled refuses new workers
        self.slots, self.cancelled = threading.BoundedSemaphore(concurrency), threading.Event()
        # what a person reads afterwards
        self.threads, self.phases, self.events, self.dropped, self.current = [], [], [], [], None
        self.status, self.result, self.error, self._worker = "created", None, None, None

    def elapsed(self):
        return time.perf_counter() - self.t0

    def emit(self, kind, **data):
        self.events.append({"t": round(self.elapsed(), 3), "type": kind, **data})      # the run's stream: every event, in order

# runtime.py
class Run:
    def __init__(self, title, goal="", *, llm, run_id=None, concurrency=16, max_agents=MAX_AGENTS_PER_RUN):
        self.title, self.goal, self.llm, self.run_id = title, goal, llm, run_id or title.lower().replace(" ", "-")
        self.t0, self.max_agents, self.lock = time.perf_counter(), max_agents, threading.Lock()
        # slots cap concurrency, cancelled refuses new workers
        self.slots, self.cancelled = threading.BoundedSemaphore(concurrency), threading.Event()
        # what a person reads afterwards
        self.threads, self.phases, self.events, self.dropped, self.current = [], [], [], [], None
        self.status, self.result, self.error, self._worker = "created", None, None, None

    def elapsed(self):
        return time.perf_counter() - self.t0

    def emit(self, kind, **data):
        self.events.append({"t": round(self.elapsed(), 3), "type": kind, **data})      # the run's stream: every event, in order

A phase is a named stretch of the run, such as Scan or Verify.

# runtime.py
    def phase(self, title):
        with self.lock:
            if self.current:                       # closing a phase records its workers and time
                mine = [t for t in self.threads if t.phase == self.current["name"]]
                self.emit("run.phase_ended", phase=self.current["name"], agents=len(mine), seconds=round(self.elapsed() - self.current["started"], 2))
            self.current = {"name": title, "started": self.elapsed()} if title else None
            if self.current:
                self.phases.append(self.current)
                self.emit("run.phase_started", phase=title, index=len(self.phases))

    def log(self, message):
        self.emit("run.log", message=str(message)[:500])

# runtime.py
    def phase(self, title):
        with self.lock:
            if self.current:                       # closing a phase records its workers and time
                mine = [t for t in self.threads if t.phase == self.current["name"]]
                self.emit("run.phase_ended", phase=self.current["name"], agents=len(mine), seconds=round(self.elapsed() - self.current["started"], 2))
            self.current = {"name": title, "started": self.elapsed()} if title else None
            if self.current:
                self.phases.append(self.current)
                self.emit("run.phase_started", phase=title, index=len(self.phases))

    def log(self, message):
        self.emit("run.log", message=str(message)[:500])

_new_thread also holds the runaway backstop.

= self.max_agents: # over the limit: record what was refused
self.dropped.append(label)
self.emit("run.dropped", label=label, limit=self.max_agents)
if len(self.dropped) >= self.max_agents: # asking for twice the limit is treated as a runaway
self.log(f"the plan asked for {2 * self.max_agents} workers, twice its limit: cancelled as a runaway")
self.cancelled.set()
return None
t = Thread(f"{self.run_id}/t{len(self.threads) + 1:03d}", phase or (self.current["name"] if self.current else ""),
label or "worker", MODELS.get(model, model), role)
self.threads.append(t)
return t” data-toggle-ms=”1200″ aria-label=”Copy code” data-nosnippet=”” class=”code-copy-btn absolute right-3 top-3 size-8 p-1.5 flex items-center justify-center bg-black/50 text-white rounded-md transition-colors duration-200 cursor-pointer hover:bg-black/70″>

# runtime.py
    def _new_thread(self, label, model, phase, role):
        """The next worker's thread, or None when the worker limit is reached."""
        with self.lock:
            if len(self.threads) >= self.max_agents:                      # over the limit: record what was refused
                self.dropped.append(label)
                self.emit("run.dropped", label=label, limit=self.max_agents)
                if len(self.dropped) >= self.max_agents:                  # asking for twice the limit is treated as a runaway
                    self.log(f"the plan asked for {2 * self.max_agents} workers, twice its limit: cancelled as a runaway")
                    self.cancelled.set()
                return None
            t = Thread(f"{self.run_id}/t{len(self.threads) + 1:03d}", phase or (self.current["name"] if self.current else ""),
                       label or "worker", MODELS.get(model, model), role)
            self.threads.append(t)
            return t

# runtime.py
    def _new_thread(self, label, model, phase, role):
        """The next worker's thread, or None when the worker limit is reached."""
        with self.lock:
            if len(self.threads) >= self.max_agents:                      # over the limit: record what was refused
                self.dropped.append(label)
                self.emit("run.dropped", label=label, limit=self.max_agents)
                if len(self.dropped) >= self.max_agents:                  # asking for twice the limit is treated as a runaway
                    self.log(f"the plan asked for {2 * self.max_agents} workers, twice its limit: cancelled as a runaway")
                    self.cancelled.set()
                return None
            t = Thread(f"{self.run_id}/t{len(self.threads) + 1:03d}", phase or (self.current["name"] if self.current else ""),
                       label or "worker", MODELS.get(model, model), role)
            self.threads.append(t)
            return t

agent() starts one worker and returns a checked value or None.

# runtime.py
    def agent(self, prompt="", *, schema=None, model="worker", label="", system="", data=None, sample=0, role="worker", phase=None):
        """One worker: its checked JSON answer, or None. Only an exception that must end the whole run gets out."""
        if self.cancelled.is_set():
            raise RunCancelled(self.run_id)
        t = self._new_thread(label, model, phase, role)
        if t is None:
            return None
        self.emit("run.agent_started", thread=t.id, label=t.label, phase=t.phase)
        with self.slots:                           # at most `concurrency` workers talk at the same time
            t.started, t.status = self.elapsed(), "running"
            try:
                t.result = self._converse(t, system, (prompt + "nn" + (data or "")).strip(), schema, sample)
                t.status = "done" if t.result is not None else "failed"
            except ABORT:
                t.status = "failed"
                raise
            except Exception as e:                 # any other failure stays the worker's own
                t.status, t.error = "failed", f"{type(e).__name__}: {e}"[:300]
            finally:
                t.ended = self.elapsed()
        self.emit("run.agent_ended", thread=t.id, label=t.label, phase=t.phase, ok=t.status == "done", credits=round(t.cost, 2),
                  seconds=round(t.ended - t.started, 2), cached=bool(t.replies) and all(r.cached for r in t.replies))
        return t.result

# runtime.py
    def agent(self, prompt="", *, schema=None, model="worker", label="", system="", data=None, sample=0, role="worker", phase=None):
        """One worker: its checked JSON answer, or None. Only an exception that must end the whole run gets out."""
        if self.cancelled.is_set():
            raise RunCancelled(self.run_id)
        t = self._new_thread(label, model, phase, role)
        if t is None:
            return None
        self.emit("run.agent_started", thread=t.id, label=t.label, phase=t.phase)
        with self.slots:                           # at most `concurrency` workers talk at the same time
            t.started, t.status = self.elapsed(), "running"
            try:
                t.result = self._converse(t, system, (prompt + "nn" + (data or "")).strip(), schema, sample)
                t.status = "done" if t.result is not None else "failed"
            except ABORT:
                t.status = "failed"
                raise
            except Exception as e:                 # any other failure stays the worker's own
                t.status, t.error = "failed", f"{type(e).__name__}: {e}"[:300]
            finally:
                t.ended = self.elapsed()
        self.emit("run.agent_ended", thread=t.id, label=t.label, phase=t.phase, ok=t.status == "done", credits=round(t.cost, 2),
                  seconds=round(t.ended - t.started, 2), cached=bool(t.replies) and all(r.cached for r in t.replies))
        return t.result

First we pull the JSON out of a reply.

# llm.py
def extract_json(text):
    """The first JSON object in a reply, even inside a code fence."""
    t = re.sub(r"^`{3}[a-zA-Z]*s*|s*`{3}$", "", text.strip())     # `{3} is three backticks
    for start in [i for i, c in enumerate(t) if c == "{"]:
        try:
            return json.JSONDecoder().raw_decode(t[start:])[0]
        except ValueError:
            continue

# llm.py
def extract_json(text):
    """The first JSON object in a reply, even inside a code fence."""
    t = re.sub(r"^`{3}[a-zA-Z]*s*|s*`{3}$", "", text.strip())     # `{3} is three backticks
    for start in [i for i, c in enumerate(t) if c == "{"]:
        try:
            return json.JSONDecoder().raw_decode(t[start:])[0]
        except ValueError:
            continue

validate turns schema errors into short lines a model can fix.

# llm.py
def validate(obj, schema):
    if obj is None:
        return ["the reply is not a JSON object"]
    # at most eight short problems the model can fix
    errors = sorted(Draft202012Validator(schema).iter_errors(obj), key=lambda e: list(e.path))
    return [f"{'/'.join(map(str, e.path)) or 'top level'}: {e.message[:120]}" for e in errors[:8]]

# llm.py
def validate(obj, schema):
    if obj is None:
        return ["the reply is not a JSON object"]
    # at most eight short problems the model can fix
    errors = sorted(Draft202012Validator(schema).iter_errors(obj), key=lambda e: list(e.path))
    return [f"{'/'.join(map(str, e.path)) or 'top level'}: {e.message[:120]}" for e in errors[:8]]

A worker’s conversation has at most two turns.

Those turns live in _converse.

# runtime.py
    def _converse(self, t, system, user, schema, sample):
        params = {"temperature": TEMPERATURE, "top_p": TOP_P, "max_tokens": 16000}
        if schema is not None:
            params["response_format"] = {"type": "json_object"}      # JSON mode when the plan wants a schema
        if t.model == MODELS["worker"] or t.role == "scan":
            params["enable_thinking"] = False      # thinking doubled recall but took 50 times as long
        t.messages = ([{"role": "system", "content": system}] if system else []) + [{"role": "user", "content": user}]
        for turn in range(2):                      # the first answer, then at most one repair
            r = self.llm.chat(t.model, t.messages, role=t.role, sample=sample, **params)
            t.replies.append(r)
            t.cost += 0 if r.cached else PRICE.get(t.model, 0.2)
            obj = r.text if schema is None else extract_json(r.text)
            errors = [] if schema is None else validate(obj, schema)
            if not errors:
                return obj
            # the repair turn lists the problems
            t.messages += [{"role": "assistant", "content": r.text[:6000]},
                           {"role": "user", "content": "Your answer does not match the schema:n- " + "n- ".join(errors)
                            + "nSend the corrected JSON object only."}]
        t.error = "schema: " + " | ".join(errors)[:240]

# runtime.py
    def _converse(self, t, system, user, schema, sample):
        params = {"temperature": TEMPERATURE, "top_p": TOP_P, "max_tokens": 16000}
        if schema is not None:
            params["response_format"] = {"type": "json_object"}      # JSON mode when the plan wants a schema
        if t.model == MODELS["worker"] or t.role == "scan":
            params["enable_thinking"] = False      # thinking doubled recall but took 50 times as long
        t.messages = ([{"role": "system", "content": system}] if system else []) + [{"role": "user", "content": user}]
        for turn in range(2):                      # the first answer, then at most one repair
            r = self.llm.chat(t.model, t.messages, role=t.role, sample=sample, **params)
            t.replies.append(r)
            t.cost += 0 if r.cached else PRICE.get(t.model, 0.2)
            obj = r.text if schema is None else extract_json(r.text)
            errors = [] if schema is None else validate(obj, schema)
            if not errors:
                return obj
            # the repair turn lists the problems
            t.messages += [{"role": "assistant", "content": r.text[:6000]},
                           {"role": "user", "content": "Your answer does not match the schema:n- " + "n- ".join(errors)
                            + "nSend the corrected JSON object only."}]
        t.error = "schema: " + " | ".join(errors)[:240]

run_sync runs a whole plan.

# runtime.py
    def run_sync(self, plan):
        self.status = "running"
        self.emit("run.created", title=self.title)
        try:
            self.result = plan(self)
            self.status = "completed"
        except RunCancelled:
            self.status = "cancelled"
        except Exception as e:                     # includes the aborts that end a run
            self.status, self.error = "failed", f"{type(e).__name__}: {e}"[:300]
        finally:                                   # always close the last phase
            self.phase(None)
            self.emit("run.ended", status=self.status, seconds=round(self.elapsed(), 2), agents=len(self.threads),
                      credits=round(sum(t.cost for t in self.threads), 2))
        return self.result

# runtime.py
    def run_sync(self, plan):
        self.status = "running"
        self.emit("run.created", title=self.title)
        try:
            self.result = plan(self)
            self.status = "completed"
        except RunCancelled:
            self.status = "cancelled"
        except Exception as e:                     # includes the aborts that end a run
            self.status, self.error = "failed", f"{type(e).__name__}: {e}"[:300]
        finally:                                   # always close the last phase
            self.phase(None)
            self.emit("run.ended", status=self.status, seconds=round(self.elapsed(), 2), agents=len(self.threads),
                      credits=round(sum(t.cost for t in self.threads), 2))
        return self.result

parallel runs thunks, functions with no arguments, all at once. It is a barrier: it returns when the slowest is done.

In one fixed-plan scan phase, finished workers waited 37 worker-minutes for the slowest.

# runtime.py
    def parallel(self, thunks):
        thunks = list(thunks)
        out, aborts = [None] * len(thunks), []
        # one task per thunk, up to 64 threads
        with cf.ThreadPoolExecutor(max(1, min(len(thunks), 64))) as pool:
            futures = {pool.submit(fn): i for i, fn in enumerate(thunks)}
            for f in cf.as_completed(futures):
                try:
                    out[futures[f]] = f.result()      # results keep the order of the thunks
                except ABORT as e:
                    aborts.append(e)
                    self.cancelled.set()           # stop the work that has not started yet
                except Exception as e:
                    self.log(f"a parallel task failed: {type(e).__name__}: {e}")
        if aborts:                                 # the first abort ends the phase for everyone
            raise aborts[0]
        return out

# runtime.py
    def parallel(self, thunks):
        thunks = list(thunks)
        out, aborts = [None] * len(thunks), []
        # one task per thunk, up to 64 threads
        with cf.ThreadPoolExecutor(max(1, min(len(thunks), 64))) as pool:
            futures = {pool.submit(fn): i for i, fn in enumerate(thunks)}
            for f in cf.as_completed(futures):
                try:
                    out[futures[f]] = f.result()      # results keep the order of the thunks
                except ABORT as e:
                    aborts.append(e)
                    self.cancelled.set()           # stop the work that has not started yet
                except Exception as e:
                    self.log(f"a parallel task failed: {type(e).__name__}: {e}")
        if aborts:                                 # the first abort ends the phase for everyone
            raise aborts[0]
        return out

One worker on the fake client shows the events a run records:

from runtime import Run

SCHEMA = {"type": "object", "required": ["answer"]}
run = Run("one worker", llm=llm)
run.run_sync(lambda r: (r.phase("Scan"), r.agent("hello", schema=SCHEMA, label="scan#0"))[1])
t = run.threads[0]
print([e["type"][4:] for e in run.events])
print(f"{t.id} | {t.phase} | {t.label} | {t.status} | {t.cost} credits | {len(t.replies)} reply")

#### OUTPUT ####
['created', 'phase_started', 'agent_started', 'agent_ended', 'phase_ended', 'ended']
one-worker/t001 | Scan | scan#0 | done | 0.2 credits | 1 reply

from runtime import Run

SCHEMA = {"type": "object", "required": ["answer"]}
run = Run("one worker", llm=llm)
run.run_sync(lambda r: (r.phase("Scan"), r.agent("hello", schema=SCHEMA, label="scan#0"))[1])
t = run.threads[0]
print([e["type"][4:] for e in run.events])
print(f"{t.id} | {t.phase} | {t.label} | {t.status} | {t.cost} credits | {len(t.replies)} reply")

#### OUTPUT ####
['created', 'phase_started', 'agent_started', 'agent_ended', 'phase_ended', 'ended']
one-worker/t001 | Scan | scan#0 | done | 0.2 credits | 1 reply

A real worker on the library read 100,911 tokens and took 18.7 seconds.

Session Budget and Recovery

All workers draw on one meter. At the cap a worker waits until a person raises it.

admit reserves a call’s price first.

self.cap + 1e-9:
self.paused = True
self.cond.wait(timeout=1.0)
self.paused = False
self.reserved += cost” data-toggle-ms=”1200″ aria-label=”Copy code” data-nosnippet=”” class=”code-copy-btn absolute right-3 top-3 size-8 p-1.5 flex items-center justify-center bg-black/50 text-white rounded-md transition-colors duration-200 cursor-pointer hover:bg-black/70″>

# llm.py
class Budget:
    def __init__(self, cap):
        # spent is billed, reserved is held for calls in flight
        self.cap, self.spent, self.reserved, self.paused = float(cap), 0.0, 0.0, False
        self.cond = threading.Condition()

    def admit(self, cost):
        with self.cond:                               # reserve the price, or wait here until the cap is raised
            while self.spent + self.reserved + cost > self.cap + 1e-9:
                self.paused = True
                self.cond.wait(timeout=1.0)
            self.paused = False
            self.reserved += cost

# llm.py
class Budget:
    def __init__(self, cap):
        # spent is billed, reserved is held for calls in flight
        self.cap, self.spent, self.reserved, self.paused = float(cap), 0.0, 0.0, False
        self.cond = threading.Condition()

    def admit(self, cost):
        with self.cond:                               # reserve the price, or wait here until the cap is raised
            while self.spent + self.reserved + cost > self.cap + 1e-9:
                self.paused = True
                self.cond.wait(timeout=1.0)
            self.paused = False
            self.reserved += cost

settle bills it, and raise_to lifts the cap.

# llm.py
    def settle(self, reserved, actual):
        with self.cond:                               # a reservation becomes spending, or is released if the call was not billed
            self.reserved -= reserved
            self.spent += actual
            self.cond.notify_all()

    def raise_to(self, cap):
        with self.cond:                               # raising the cap wakes every waiting worker
            self.cap, self.paused = float(cap), False
            self.cond.notify_all()

# llm.py
    def settle(self, reserved, actual):
        with self.cond:                               # a reservation becomes spending, or is released if the call was not billed
            self.reserved -= reserved
            self.spent += actual
            self.cond.notify_all()

    def raise_to(self, cap):
        with self.cond:                               # raising the cap wakes every waiting worker
            self.cap, self.paused = float(cap), False
            self.cond.notify_all()

start runs a long plan on its own thread.

# runtime.py
    def start(self, plan):
        # the plan runs on its own thread
        self._worker = threading.Thread(target=self.run_sync, args=(plan,), daemon=True)
        self._worker.start()
        return self

    def wait(self):
        self._worker.join()                        # block until the plan has finished
        return self.result

    def cancel(self):
        self.cancelled.set()                       # workers already talking finish, new ones are refused

# runtime.py
    def start(self, plan):
        # the plan runs on its own thread
        self._worker = threading.Thread(target=self.run_sync, args=(plan,), daemon=True)
        self._worker.start()
        return self

    def wait(self):
        self._worker.join()                        # block until the plan has finished
        return self.result

    def cancel(self):
        self.cancelled.set()                       # workers already talking finish, new ones are refused

statusline answers “how is it going?” from any thread.

# runtime.py
    def statusline(self):
        """One line answering 'how is it going?' from any thread."""
        n = collections.Counter(t.status for t in self.threads)
        return (f"{self.run_id} | {self.status} | phase {len(self.phases)}: {self.current['name'] if self.current else '-'} | "
                f"{n['running']} running, {n['done']} done, {n['failed']} failed | {sum(t.cost for t in self.threads):.1f} credits | "
                f"{self.elapsed():.1f}s" + (" | PAUSED at the budget" if self.llm.budget.paused else ""))

# runtime.py
    def statusline(self):
        """One line answering 'how is it going?' from any thread."""
        n = collections.Counter(t.status for t in self.threads)
        return (f"{self.run_id} | {self.status} | phase {len(self.phases)}: {self.current['name'] if self.current else '-'} | "
                f"{n['running']} running, {n['done']} done, {n['failed']} failed | {sum(t.cost for t in self.threads):.1f} credits | "
                f"{self.elapsed():.1f}s" + (" | PAUSED at the budget" if self.llm.budget.paused else ""))

Forty workers get a cap of 5 credits, enough for 25 calls at 0.2 each.

from llm import Budget

paid.clear()
capped = LLM(Ledger(":memory:"), client=fake_client, budget=Budget(5.0))
run = Run("forty workers", llm=capped, concurrency=10)
run.start(lambda r: r.parallel([lambda i=i: r.agent(f"question {i}", schema=SCHEMA) for i in range(40)]))
time.sleep(1.2)
print(f"requests sent {len(paid)}n{run.statusline()}")
capped.budget.raise_to(10.0)
run.wait()
print(f"cap raised to {capped.budget.cap}: {run.status}, spent {capped.budget.spent:.1f}, requests sent {len(paid)}")

#### OUTPUT ####
requests sent 25
forty-workers | running | phase 0: - | 10 running, 25 done, 0 failed | 5.0 credits | 1.2s | PAUSED at the budget
cap raised to 10.0: completed, spent 8.0, requests sent 40

from llm import Budget

paid.clear()
capped = LLM(Ledger(":memory:"), client=fake_client, budget=Budget(5.0))
run = Run("forty workers", llm=capped, concurrency=10)
run.start(lambda r: r.parallel([lambda i=i: r.agent(f"question {i}", schema=SCHEMA) for i in range(40)]))
time.sleep(1.2)
print(f"requests sent {len(paid)}n{run.statusline()}")
capped.budget.raise_to(10.0)
run.wait()
print(f"cap raised to {capped.budget.cap}: {run.status}, spent {capped.budget.spent:.1f}, requests sent {len(paid)}")

#### OUTPUT ####
requests sent 25
forty-workers | running | phase 0: - | 10 running, 25 done, 0 failed | 5.0 credits | 1.2s | PAUSED at the budget
cap raised to 10.0: completed, spent 8.0, requests sent 40

The run stopped at exactly 25 requests, and after the cap went up it finished with forty. Next I cancel a two-phase run about halfway, then run the same plan again on the same ledger.

<button data-code="plan = lambda r: (r.parallel([lambda i=i: r.agent(f"scan {i}", schema=SCHEMA, phase="Scan") for i in range(30)]),
r.parallel([lambda i=i: r.agent(f"verify {i}", schema=SCHEMA, model="judge", phase="Verify") for i in range(30)]))
paid.clear()
ledger = Ledger(":memory:")
first = Run("crash", llm=LLM(ledger, client=fake_client), concurrency=8).start(plan)
while len(paid)

plan = lambda r: (r.parallel([lambda i=i: r.agent(f"scan {i}", schema=SCHEMA, phase="Scan") for i in range(30)]),
                  r.parallel([lambda i=i: r.agent(f"verify {i}", schema=SCHEMA, model="judge", phase="Verify") for i in range(30)]))
paid.clear()
ledger = Ledger(":memory:")
first = Run("crash", llm=LLM(ledger, client=fake_client), concurrency=8).start(plan)
while len(paid) < 25:                                  # let it get about halfway, then pull the plug
    time.sleep(0.01)
first.cancel()
first.wait()
before = len(paid)
second = Run("crash", llm=LLM(ledger, client=fake_client), concurrency=8)
second.run_sync(plan)
print(f"first run {first.status} after {before} paid calls, then the second run {second.status} with {len(paid) - before} new calls, {len(paid)} in all")

#### OUTPUT ####
first run cancelled after 30 paid calls, then the second run completed with 30 new calls, 60 in all

plan = lambda r: (r.parallel([lambda i=i: r.agent(f"scan {i}", schema=SCHEMA, phase="Scan") for i in range(30)]),
                  r.parallel([lambda i=i: r.agent(f"verify {i}", schema=SCHEMA, model="judge", phase="Verify") for i in range(30)]))
paid.clear()
ledger = Ledger(":memory:")
first = Run("crash", llm=LLM(ledger, client=fake_client), concurrency=8).start(plan)
while len(paid) < 25:                                  # let it get about halfway, then pull the plug
    time.sleep(0.01)
first.cancel()
first.wait()
before = len(paid)
second = Run("crash", llm=LLM(ledger, client=fake_client), concurrency=8)
second.run_sync(plan)
print(f"first run {first.status} after {before} paid calls, then the second run {second.status} with {len(paid) - before} new calls, {len(paid)} in all")

#### OUTPUT ####
first run cancelled after 30 paid calls, then the second run completed with 30 new calls, 60 in all

The plan needs sixty calls whether or not it is interrupted, and no call was paid for twice.

Audit Task Kit

Three kit steps call the model: scan gives a worker one shard and one lens, check gives a skeptic a batch, and report` writes the list.

The skeptic is told that most reports are false alarms and to answer real: false when unsure.

"""kit.py: the words of an audit: scan, check, vote and report."""
import collections

from bench import WORK, function_at, normalise_path
from config import MODELS, PROMPTS, SHARD_TOKENS
from shards import numbered, pack

LENSES = ["boundaries", "logic", "state"]                 # a lens points a worker at one family of mistakes, "general" at none
FINDINGS = {"type": "object", "required": ["findings"], "properties": {"findings": {"type": "array", "items": {
    "type": "object", "required": ["file", "line", "claim"], "properties": {"line": {"type": "integer"}}}}}}
VERDICTS = {"type": "object", "required": ["verdicts"], "properties": {"verdicts": {"type": "array", "items": {
    "type": "object", "required": ["report", "real"], "properties": {"real": {"type": "boolean"}}}}}}
REPORT = {"type": "object", "required": ["bugs"], "properties": {"bugs": {"type": "array", "items": {"type": "object", "required": ["ids"]}}}}

"""kit.py: the words of an audit: scan, check, vote and report."""
import collections

from bench import WORK, function_at, normalise_path
from config import MODELS, PROMPTS, SHARD_TOKENS
from shards import numbered, pack

LENSES = ["boundaries", "logic", "state"]                 # a lens points a worker at one family of mistakes, "general" at none
FINDINGS = {"type": "object", "required": ["findings"], "properties": {"findings": {"type": "array", "items": {
    "type": "object", "required": ["file", "line", "claim"], "properties": {"line": {"type": "integer"}}}}}}
VERDICTS = {"type": "object", "required": ["verdicts"], "properties": {"verdicts": {"type": "array", "items": {
    "type": "object", "required": ["report", "real"], "properties": {"real": {"type": "boolean"}}}}}}
REPORT = {"type": "object", "required": ["bugs"], "properties": {"bugs": {"type": "array", "items": {"type": "object", "required": ["ids"]}}}}

batches cuts a list into groups, and flatten` joins lists back into one.

# kit.py
def batches(items, size):
    items, size = list(items), max(1, int(size))
    return [items[i:i + size] for i in range(0, len(items), size)]

def flatten(lists):
    return [x for xs in lists if xs for x in xs]           # one list from a list of lists, skipping workers that failed (None)

# kit.py
def batches(items, size):
    items, size = list(items), max(1, int(size))
    return [items[i:i + size] for i in range(0, len(items), size)]

def flatten(lists):
    return [x for xs in lists if xs for x in xs]           # one list from a list of lists, skipping workers that failed (None)

finding builds one finding, and key_of names its place.

# kit.py
def finding(obj, source=""):
    path, line, conf = normalise_path(obj.get("file", "")), int(obj.get("line") or 0), obj.get("confidence")
    span = function_at(path, line)                         # models often name the wrong function
    # texts are cut to keep prompts small
    return {"file": path, "line": line, "function": span[0] if span else str(obj.get("function") or ""), "source": source,
            "claim": str(obj.get("claim") or "")[:400], "failing_call": str(obj.get("failing_call") or "")[:400],
            "expected": str(obj.get("expected") or "")[:200], "confidence": min(1.0, max(0.0, float(conf))) if isinstance(conf, (int, float)) else 0.5}

def key_of(f):
    """The place a finding is about: its file and function, or its file and ten-line block."""
    return f"{f['file']}::{f['function']}" if f["function"] else f"{f['file']}::line {f['line'] // 10 * 10}"

# kit.py
def finding(obj, source=""):
    path, line, conf = normalise_path(obj.get("file", "")), int(obj.get("line") or 0), obj.get("confidence")
    span = function_at(path, line)                         # models often name the wrong function
    # texts are cut to keep prompts small
    return {"file": path, "line": line, "function": span[0] if span else str(obj.get("function") or ""), "source": source,
            "claim": str(obj.get("claim") or "")[:400], "failing_call": str(obj.get("failing_call") or "")[:400],
            "expected": str(obj.get("expected") or "")[:200], "confidence": min(1.0, max(0.0, float(conf))) if isinstance(conf, (int, float)) else 0.5}

def key_of(f):
    """The place a finding is about: its file and function, or its file and ten-line block."""
    return f"{f['file']}::{f['function']}" if f["function"] else f"{f['file']}::line {f['line'] // 10 * 10}"

dedupe merges findings about one place and counts the workers behind them.

# kit.py
def dedupe(findings):
    # group by place, keep the most confident
    groups = collections.defaultdict(list)
    for f in findings:
        groups[key_of(f)].append(f)
    return [dict(max(g, key=lambda f: (f["confidence"], -f["line"], f["claim"], f["failing_call"], f["source"])),   # ties are broken by
                 reports=sum(f.get("reports", 1) for f in g)) for _, g in sorted(groups.items())]                 # content, so every run keeps the same one

# kit.py
def dedupe(findings):
    # group by place, keep the most confident
    groups = collections.defaultdict(list)
    for f in findings:
        groups[key_of(f)].append(f)
    return [dict(max(g, key=lambda f: (f["confidence"], -f["line"], f["claim"], f["failing_call"], f["source"])),   # ties are broken by
                 reports=sum(f.get("reports", 1) for f in g)) for _, g in sorted(groups.items())]                 # content, so every run keeps the same one

A place survives only when enough skeptics call it real, which is what vote checks.

= need]” data-toggle-ms=”1200″ aria-label=”Copy code” data-nosnippet=”” class=”code-copy-btn absolute right-3 top-3 size-8 p-1.5 flex items-center justify-center bg-black/50 text-white rounded-md transition-colors duration-200 cursor-pointer hover:bg-black/70″>

# kit.py
def vote(candidates, verdicts, need=2):
    # keep places that at least `need` skeptics called real
    real = collections.Counter(v["key"] for v in verdicts if v and v.get("real") is True)
    return [dict(c, votes=real[key_of(c)]) for c in candidates if real[key_of(c)] >= need]