Lesson 08. Wrap Agent Steps in Deterministic Nodes and Cap the Retries

Three public accounts of agent-built delivery lines describe the same first-version mistake. An open-source label-driven factory started with a language model as its dispatcher, and the model launched runs nobody had asked for; the author replaced it with a plain shell script that only reads labels. A vendor that sells review tooling went through its own commit history and found agents reporting a task finished while the branch held no changes at all, plus pull requests that needed four and five agent rounds with a human looking at every round (GetPullRequest, vendor blog, observed 2026-10-28). A third team, benchmarking a fix loop in its own graph runner, saw most loop runs spend every round they were allowed, so the cap and not the result set the bill (The Open Engine, vendor-hosted, observed 2026-10-28). In each case the model's work was not the expensive part. The expensive part was that nothing outside the model decided when to start, when the work was done, or when to stop. Decide now: in your line, who says a step is finished, the agent that did it or something that cannot be talked into agreeing?

This lesson gives you factory/blueprint.md - your line as a node list marking each node deterministic or agent, the retry cap and the exit for each. It serves the Run stage of the loop: the point where the line starts working unattended and you need to know what runs, what repeats, and where it ends.

What is actually going on

A line that uses agents is a graph, and the useful question is which nodes in it may use judgment. Stripe answers with what it calls a blueprint, a state machine written in code. Its nodes are either agent nodes ("Implement task", "Fix CI failures") or deterministic nodes ("Run configured linters", "Push changes") that call no model at all (Stripe, Minions part 1, observed 2026-10-28). The agent never decides to lint or push. Those steps happen because the graph reaches them. A sentence in a prompt that says "remember to lint" has been replaced by an edge that cannot be skipped. The Open Engine argues the same: prose is a request, an edge is enforced.

Deterministic nodes do three jobs. They start work (dispatch), they check work (lint, tests, a count of commits), and they finish work (push, merge). Agent nodes do the one thing code cannot: turn a fuzzy task into a diff, or a failure log into a fix. Dispatch shows why the split matters. The label-driven factory above ended up with a dumb cron script that treats labels as its only state, because a model-run dispatcher invented work. OpenAI's published orchestration spec for Codex agents reaches the same place from another side: tickets move through tracker states, and the agent must follow a defined landing procedure instead of calling merge itself (OpenAI, Symphony, observed 2026-10-28).

The same logic decides who ends a loop. One open-source workflow engine ends its implement loop when a shell test command exits clean, starting each pass with fresh context and a maximum pass count; the model supplies judgment inside the node, and the exit code supplies the verdict. An agent can say it is finished; only a node that cannot be argued with can say it is done.

Every loop that contains an agent node also needs a counter and an exit. Stripe runs lint heuristics on each push in under five seconds, lets CI auto-apply autofixes, and sends the remaining failures back to the agent once. After the second CI round it stops and hands the branch to a human, because more rounds cost tokens, compute and time for diminishing returns (Stripe, Minions part 2, observed 2026-10-28). Spotify's earlier homegrown loop capped at two layers: ten turns per session and three session retries (Spotify, observed 2026-10-28).

Read the diagram for two things: which boxes are marked AGENT, and where each loop ends.

diagramsource

drawing

The agent boxes are surrounded by code that checks them. The "no commits" edge ends the run in a failed state, and the only loop back to an agent runs once.

The rule and its condition

Use a deterministic node when you can state the step's correct result as an exit code or a yes/no read of state. Use an agent node when the step needs judgment over open-ended input and a deterministic node can check the result afterwards. Put a counter and a named exit on every loop that contains an agent node. A step with neither a checkable result nor a cap does not belong in an unattended line yet.

The condition is observable in your own system: can you write the step's pass condition as a command another process runs and reads the result of? If yes, the step is a deterministic node. If you can only write it as "the agent says it looks right", the step is unfinished. Either build a command that checks it, or route it to a human. Graph Engineering, lecture 3 covers node contracts, lecture 6 argues that a model may judge while the graph decides, and lecture 7 builds the verifier. This lesson applies them to a delivery line.

Node kind	Use it when	Who decides done	Cap and exit	Evidence
Deterministic	The pass condition is an exit code or a state read: lint, push, merge, dispatch, commit count	The code	No retries; a failure stops the run or follows a fixed edge	Stripe blueprint, first-party
Agent with a deterministic check	The step needs judgment (implement, fix CI) and a command can check the result	The check, never the agent	A fixed number of rounds, then a human	Stripe's two CI rounds, self-reported
Agent as judge	The check needs intent (does this diff match the prompt)	It may veto; code still decides pass	A veto counts against the same cap	Spotify's judge, self-reported
Agent grading itself	Never	Nobody	None	Vendor commit history: "done" with no commits

A judge gets its own row because it is easy to mistake for a gate. Spotify runs one after its verifiers, reading the prompt and the diff, to veto scope creep. The verifiers still decide pass.

Otherwise, when the repo has no runnable tests, there is nothing for a verify loop to run, and the honest exit is a human who tests by hand. Spotify's owners did exactly that for data pipelines without unit tests (Spotify, part 4, observed 2026-10-28).

Worked case

Stripe describes its Minions line in two posts (part 1 and part 2, both observed 2026-10-28). A person tags a Slack app in a thread or presses a button on a ticket. Code gathers context from the links in the thread. A coding agent works on an isolated devbox with no production data and no internet egress, which is why it can run without confirmation prompts. The run ends in a drafted pull request that a different engineer reviews. Stripe reports, self-reported and without a denominator, that over 1,300 such PRs merge per week and that the test battery holds more than 3 million tests (Stripe, part 2). That suite is far too big to run in full on every edit.

You are writing the blueprint for a line like this. The agent's first fix to a red CI run has just failed too. Decide now: what does the blueprint do next? (A) Loop until CI is green, inside a time budget. (B) Allow exactly one fix round, then stop and hand the branch to a human. (C) Ask the agent whether another attempt is likely to help.

Choose B. Work through it in order. First, mark which nodes need judgment: only "Implement task" and "Fix CI failures". Lint, push, the CI run, the PR draft and dispatch are code. Second, move cheap feedback left: lint runs on every push, in seconds, so most mistakes die before the expensive suite starts. Third, let code absorb what it can: CI applies autofixes by itself, and only failures with no autofix reach the agent, once. Fourth, set the cap at two CI rounds in total, which is Stripe's stated limit. Fifth, make the exit a person: the branch, the CI log and the run transcript go to a human operator. Stripe notes that a run that misses is still often a good starting point, so a hand-off is not a discard.

Option A costs money and hides signal. Here is the modeled cost of an uncapped loop on a small line:

Illustrative composite. A team starts 100 runs per week and 30 fail the first CI round. An uncapped loop averages 5 extra rounds on those 30 at about $3 per round, so $450 per week, and roughly a third of those runs never go green. Capped at one fix round, the same 30 runs cost $90, and the unfixed ones reach a human the same day instead of Monday. These figures are modeled, not measured.

Option C asks the agent to grade its own effort, the very thing the vendor commit history showed failing: an agent calling a task done with nothing committed (GetPullRequest, vendor blog, observed 2026-10-28). Here is the blueprint file you would write. Node names in quotes are Stripe's published ones; the rest are our labels.

The blueprint is the same shape as the ticket's life. Here is that life as states, so you can see every dead end has a name.

diagramsource

drawing

Every red path ends in Failed, Needs human or In review, and none of them ends with the agent deciding it has tried enough.

Faded case

Skip to the open case if this is familiar. Spotify describes a background agent that opens pull requests across many repositories (Spotify, part 3, observed 2026-10-28). The agent runs in a sandbox with few binaries and no credentials; it sees only the repo, edits files and calls one tool named verify. Behind that tool sit independent verifiers (build, tests, format) that switch on according to what the repo contains. The agent cannot see what they do. Before a pull request opens, a stop hook runs every relevant verifier again, and a failing verifier means no PR. The surrounding infrastructure, not the agent, opens the PR.

One more node follows the verifiers: an LLM judge that gets the original prompt and the diff and vetoes scope creep. Spotify says, self-reported, that the judge vetoes about 25% of sessions and that about half of vetoed sessions correct themselves; it also says it has no evals for the judge yet (Spotify, part 3, observed 2026-10-28). Two things in that post are worth keeping apart: what Spotify built, and what this blueprint should say about caps. The sources read for this lesson give no retry cap for the judge, so that line is yours to design.

Fill in the same table for these nodes: verify tool, stop hook, judge, PR opener. For the judge row alone, decide its kind, who decides done, its retry cap and its exit.

Open case

Illustrative composite. A team runs an overnight line. A language model reads the backlog and picks tickets. A coding agent edits the code, runs the tests itself and writes "tests pass" in its notes. If the notes say pass, a script merges to the main branch. If the agent says it needs more attempts, it keeps going until morning. Last month three tickets were closed as done with no commits behind them, and one ticket ran for nine hours.

List this line as nodes. Mark each node deterministic or agent, give every loop a cap and every cap an exit, and say which exits a human reads. The quiz grades your redesign.

Where this breaks

A deterministic wrapper only makes the line as good as its checks. Spotify names the worst failure itself: a PR that passes CI and is still functionally wrong, hard to spot across thousands of repos (Spotify, part 3, observed 2026-10-28). Nodes only ensure that checks run; they cannot make a weak check strong. Repos without runnable tests lose the verify loop altogether.

A cap bounds cost; it does not buy success. In the vendor-hosted benchmark above, 91 of 100 loop runs used every one of the 50 allowed rounds, and one task plateaued near 53% across three runs (The Open Engine, vendor-hosted, observed 2026-10-28). Stripe's two rounds is one company's choice for its own suite and no source shows two is the best number; Loop Engineering, lecture 4 on spend tiers is the place to set yours. Caps can also lose work: Spotify's earlier loop ran out of turns on edits that cascaded across files and lost the task when context filled. And every capped exit lands on a person, so the human queue grows; Agent Fleets, lesson 11 explains why review throughput is the ceiling, and Lesson 07 of this course puts exhausted retries in the lane where a human must read the change.

Volume figures need care. Stripe's weekly PR count is self-reported, with no merge rate or revert rate behind it, and humans review every one of those PRs, so it is not an unattended-merge design. Spotify's figure of over 1,500 merged agent PRs is self-reported too (Spotify, part 1, observed 2026-10-28). Its far larger count of automated maintenance PRs, more than 2.5 million, comes from classic deterministic fleet automation that mostly merges with no human, so it says nothing about agent output (Spotify, observed 2026-10-28). The vendor commit-history claims are also self-reported by a company that sells review tooling.

Key Takeaways

• Mark every node in your line deterministic or agent before you add any capability.
• Write lint, push, merge and dispatch as code, so they run on every pass.
• Cap each loop containing an agent at a small number of rounds, then name a human exit.
• Treat zero commits as a failed task, checked by code, never by report.
• An agent can say it is finished; only a node that cannot be argued with can say it is done.
• Use an LLM judge to veto, and keep pass in a deterministic check.

What You Have Now

factory/blueprint.md - your line as a node list marking each node deterministic or agent, the retry cap and the exit for each. It sits beside factory/line-map.md from Lesson 01: the line map shows stages and gates, and the blueprint shows which steps inside those stages are code and which are models. Your risk lanes from Lesson 07 read the exit column: any exit that says "human" feeds the lane where a person reads the change. It serves the Run stage, and each later lesson that adds a step must add a row here first.

Further Reading

• Stripe, "Minions" part 1 shows the blueprint idea and the thread-to-PR flow in the builder's own words.
• Stripe, "Minions" part 2 gives the lint-first order, the autofix step and the two-round CI cap with Stripe's reasoning.
• Spotify, "Feedback loops for background coding agents" describes verifiers behind one tool, a stop hook and an LLM judge.
• Spotify, "Context engineering for background coding agents" records the early turn and retry budget and the way that loop failed.
• OpenAI, "Open-source Codex orchestration: Symphony" shows a ticket-state model where the tracker, not the agent, routes work.
• The Open Engine, review loop benchmark is vendor-hosted evidence that independent checks and round caps change cost.

Exercises

1. Open factory/line-map.md from Lesson 01 and list each stage's steps. Mark every step deterministic or agent, using the test "can I write its pass condition as a command?"
2. Write factory/blueprint.md for your own line in the table shape above. Give every agent node a retry cap and every cap an exit, and mark which exits a human reads.
3. Add a red control for each cap and for the no-commits rule, using a stub agent that misbehaves. Run each stub and record whether the line stopped where the blueprint says. If a stub gets past a cap, fix the node.

Check your understanding