Lesson 09. Score the Repo Before You Switch Autonomy On

When Microsoft's .NET team started handing issues to a coding agent in its large open-source runtime repository, most early pull requests died. The agent wrote code it could not compile, because the instructions it followed and the network allow-list around its sandbox did not let it build the project. The share of its pull requests that succeeded was 38.1% until the team fixed both, and 69% afterwards (Microsoft .NET team, observed 2026-10-28), and the biggest lever the team reports was the instruction file, not a new model. Each failed pull request was still something a maintainer had to open and judge. Atlassian's engineers met the same wall one level up: tickets with thin detail, or just a link to a chat thread, rarely came back finished (Atlassian engineering, observed 2026-10-28). Now picture your own repository on the day autonomy goes live: a red default branch that people rerun until it passes, build steps that live in a wiki, a ticket queue full of one-line titles. Decide now: do you learn that from a one-minute check today, or from a week of pull requests that never had a chance?

This lesson gives you factory/readiness-score.md - a tiered checklist scored for one repo, with the tier it unlocks and the proposed-issue rule for a tier drop. It serves the Run stage of the course loop: it decides whether the line may run unattended on this repo at all, and how far.

What is actually going on

An agent that cannot ask a colleague works only with what the repository tells it. Whatever is missing, it fills with a guess, and a guess about how to build a project produces code that was never compiled. The sources agree that this is not a weak-model problem. OpenAI's team describes early progress on its agent-written product as slow because the environment was underspecified (OpenAI, observed 2026-10-28). Cloudflare reports that its first agents made plausible but wrong changes because they lacked local context: test commands, conventions and off-limits areas (Cloudflare, observed 2026-10-28). Its fix is worth copying in shape. A pipeline drafts a context file for each repository from the service catalog and opens a merge request, so the owning team reviews what the agent will be told, and it has processed about 3,900 repositories that way (Cloudflare, observed 2026-10-28).

Read those cases together and the facts a repository must state for an agent to finish a loop alone fall out. Who answers for it: a named owner. What it is and how to run it: a README. What a good change looks like: a pull request template. How a change is judged: a CI workflow, plus a default branch where that workflow is green, because a red baseline makes every agent result unreadable. How the agent checks itself before it pushes: tests it can run and build steps it can follow inside a sandbox. And what to build: a ticket specific enough to act on. Microsoft's agent runs a baseline build before it changes anything (Microsoft .NET team, observed 2026-10-28), which only helps if that build can succeed where the agent runs.

A score turns those eight facts into a decision. Each check is a one-minute test with a yes or no result and a line of evidence. Checks group into tiers, and the repo holds the highest tier whose checks all pass. The tier then caps what the autonomy map from Lesson 3 may switch on. The diagram shows where the score sits: between the repo and the lanes, with a loop back, because agent pull requests change the repo the score reads.

diagramsource

drawing

The loop back is the part teams forget. A score taken once describes the repo on rollout day, so the arrow from agent pull requests to the repo is the reason the score must be retaken on a schedule.

One caution before the rule. The score is a gate: it says the agent is not set up to fail for avoidable reasons. It does not say the agent will succeed. A readiness score is a floor under autonomy, never a forecast of success. None of the sources this course reviewed shows a score predicting agent outcomes, and the best-known levels model is published without any.

The rule and its condition

Switch autonomy on only up to the tier the repo has earned. A tier is earned when every check in it passes, and it is lost when one fails. A lost tier opens a PROPOSED issue, and a human approves the drop. Use that rule when you are about to enable an agent on a repo or lane, and again each time the repo changes underneath it. Otherwise you are measuring readiness by the agent's first failures, which is the expensive way to read the same facts.

The condition is observable in one sitting: can you run all eight tests and write PASS or FAIL beside each, with evidence? If a check cannot be tested in a minute, rewrite it until it can. Bronze, Silver and Gold are names this lesson uses. They are a convention you define, not a standard.

For comparison, Factory publishes a levels model in vendor documentation: a repo reaches the next level after passing 80% of the previous level's criteria, and Factory calls its third level, Standardized, the minimum for production-grade autonomous work (Factory docs, vendor, observed 2026-10-28). Those thresholds are Factory's own design. This lesson composes a stricter shape from the cards, and the table shows why.

Approach	Use when	Breaks when	Source
All-or-nothing tiers (this lesson composes)	A short list where each check is a prerequisite for the next step of autonomy	The list grows long enough that one trivial miss blocks a whole lane	Composed from several cards, no single source
Percentage threshold per level	Many criteria that are mostly independent	A critical check hides inside the passing share, such as a red default branch	Factory docs, vendor
One blended number	You need an org dashboard across hundreds of repos	Strong checks mask a weak one, and nobody can say what the number unlocks	None
No score, trial on a real ticket	A throwaway repo where a failed run costs nobody's time	The repo has reviewers, because every dead pull request spends their attention	Microsoft .NET team

The tier-drop half of the rule has its own loop. A script can detect that a check failed. Only a person can decide whether the failure is real, whether in-flight work should pause, and who fixes it. So automation proposes and a human approves. While a proposal sits open, new assignments in the affected lanes freeze, and silence never lifts the freeze. The diagram traces one lap.

diagramsource

drawing

The only step automation takes alone is to propose and freeze. Everything that changes the tier, or lifts the freeze, passes through a named person, which is the same author-never-holds-the-key logic you applied to merges in Lesson 4.

Worked case

Illustrative composite. A six-person billing team owns invoice-service, a Go service. It is a fictional repo used only to practise scoring. Facts on Tuesday: CODEOWNERS names the billing-platform team for every path. The README has sections for what the service does, how to run it and how to test it. A pull request template exists. One workflow runs on every pull request, and the last ten runs on main passed on their first attempt. make test passes on a fresh clone. The README build section says to fetch modules from the internal feed, and names neither the feed host nor the sandbox allow-list. Of five sampled open bug tickets, three describe expected behaviour and a file, and two say only "see the Slack thread". The lead wants the agent pulling from the bug queue on Monday.

You have three options for Monday.

Option A: switch on the bug queue as planned. Six of eight checks pass, which feels close enough.

Option B: score by the highest tier whose checks all pass. Run every test, record the evidence, and let the tier set what is unlocked.

Option C: change nothing until all eight pass. No agent touches the repo until the tickets are fixed too.

Decide now: which option, and what is the first job the agent gets?

Option A fails first. Check 7, one of the two failing checks, decides whether the agent can produce a change that compiles, and the bug queue is the lane where it will be asked to. Microsoft's .NET team saw success move from 38.1% to 69% after fixing instructions and the firewall allow-list (Microsoft .NET team, self-reported, observed 2026-10-28). The same post records what failed runs cost: one person triggered nine pull requests that each needed five to nine hours of expert review, and three were still open a week later (Microsoft .NET team, observed 2026-10-28). Counting passes tells you nothing about which check is missing.

Option C is wasteful in the other direction. The repo has earned a real tier, and the first thing worth doing with it is lift the check that blocks the next one. Waiting for tickets to improve also leaves the build gap open, since nobody is assigned to close it.

Option B is the one you build, in five steps. First, run every test and write the evidence beside it, as in the file below. Second, set the tier: checks 1 to 5 pass, so the repo holds Bronze; check 7 blocks Silver, and checks 7 and 8 block Gold. Third, write what Bronze unlocks into factory/autonomy-map.md, capped at the docs and test-only lanes with a human assigning every issue. Fourth, make the first agent job the check 7 fix in the docs lane: write down the feed host and the allow-list steps, with no credentials in the text, since the agent never holds the key (Agent Fleets, lesson 7). A deterministic test accepts the result, not a reviewer's opinion: the sandbox build either exits zero or it does not. Fifth, write the drop rule so the score keeps working after Tuesday.

The cost of Option B is an afternoon of testing and one doc task. The cost of Option A is a week of dead pull requests sitting in the reviewers' queue.

Faded case

Skip to the open case if this is familiar. The same repo has climbed to Silver. On Thursday the nightly job finds the default branch red for the second night in a row, after a dependency bump merged on Tuesday. Three agent pull requests are open in the bug lane. CI on each of them is also red, for the same reason.

You have the rule from the worked case. Your task is to write the proposed issue's key decisions: the new tier, who gets the fix issue, and what happens to the three open pull requests. Check 5 is a Bronze check, so this failure reaches below the tier the repo holds. Think about what a red baseline does to the evidence on every open pull request, and about whether the agent should be the one repairing the thing it relies on to be judged. Also decide who is told, since a reviewer who approved one of those pull requests on Wednesday did so against evidence that may now be void. Name the person, the label and the order of steps, so that the next night's job does not reopen the same issue.

Open case

A platform team runs notify-hub, a Python service. Their audit sheet reads: CODEOWNERS names an owner for every path. The README has run and test sections. A pull request template exists. A workflow runs on every pull request. Of the last ten runs on the default branch, four failed on their first attempt, all on one flaky integration test that people rerun until it passes. make test works from a fresh clone. The build steps are in a wiki page. Most bug tickets are one line plus a Slack link. The lead wants the agent to start pulling bug tickets next sprint.

Which tier does notify-hub hold today, which check decides that, and what happens to the plan? The quiz grades your answer.

Where this breaks

A readiness score is a gate, not a predictor. None of the sources this course reviewed shows that scores predict agent success. Factory's levels are vendor documentation, its own design, published without outcome data (Factory docs, vendor, observed 2026-10-28). Microsoft's 38.1% to 69% is one repository family measured before and after a setup change, on author-selected tasks, not a trial of scores.

The instruction-file evidence is mixed, so this list does not score its presence. Three preprints disagree. One measured a median runtime drop of 28.64% and fewer output tokens with comparable completion, across 10 repositories and 124 pull requests (Lulla et al., preprint, observed 2026-10-28). A second found context files tended to lower task success and raise inference cost by over 20% (ETH Zurich and LogicStar, preprint, observed 2026-10-28). A third, with 288 runs over 17 tasks and 3 repositories, saw no measurable change in correctness within a band of 10 to 15 points (Khatri, independent preprint, observed 2026-10-28). All three are small. Test a context file on your own repo.

Scores rot and can be gamed. A stub README passes a heading grep. Cloudflare's own account warns that a stale context file can be worse than none (Cloudflare, observed 2026-10-28). Review capacity is not on the list, and Microsoft's review queue shows it becomes the constraint.

Key Takeaways

• Score the repo with eight one-minute checks before any lane gets autonomy; write evidence beside every result.
• Hold the highest tier whose checks all pass, because a percentage can hide a red default branch.
• Open a PROPOSED issue on every tier drop; automation proposes and a named human approves.
• Freeze new assignments in affected lanes while a proposal is open; silence never lifts the freeze.
• A readiness score is a floor under autonomy, never a forecast of success.
• Test an instruction file on your own repo instead of scoring its presence.

What You Have Now

factory/readiness-score.md - a tiered checklist scored for one repo, with the tier it unlocks and the proposed-issue rule for a tier drop. Keep it beside factory/autonomy-map.md from Lesson 3, whose levels it now caps, and inside the protected paths from Lesson 6, because an agent that can edit its own score can pass it. The nodes in factory/blueprint.md from Lesson 8 should read the held tier before dispatching. This is the Run stage's entry gate: the check that decides whether the line may start on this repo, and the rule that stops it when the repo slips.

Further Reading

• Microsoft .NET team, "Ten months with the Copilot coding agent in dotnet/runtime": the before-and-after success rates, the firewall fix and the review bottleneck.
• Cloudflare, "The AI engineering stack we built internally": how generated context files get owner review, and why stale ones hurt.
• OpenAI, "Harness engineering": the account of an underspecified environment slowing a Codex-built product.
• Atlassian, "Vulnerability fixes with agentic pipelines": choosing jobs with a clear entry condition and a deterministic check; single-team, self-reported.
• Factory documentation, "Agent Readiness overview": a vendor's levels model, useful for its criteria list, not for outcome claims.
• ETH Zurich and LogicStar preprint on context files: the study that found context files lowering success and raising cost.

Exercises

1. Run the eight tests on one repo you own. Write PASS or FAIL beside each with a line of evidence, and state the tier it holds. Cross-check against Agent Fleets, project 1, the fleet readiness audit.
2. Draft the PROPOSED issue your nightly job would open for the check most likely to fail first in your repo. Fill every field, and name the approver and what happens if that person is away.
3. Design an A/B test for your repo's instruction file: choose ten closed tickets, run the agent with and without the file, and compare success and cost. Decide in advance what result would remove the file from your checklist.

Check your understanding