DEV Community: Gábor Mészáros

Loop Engineering: How to Stop Your Agent Reward-Hacking Its Own Checks

Gábor Mészáros — Wed, 22 Jul 2026 22:03:30 +0000

You gave the agent a failing test and told it to get the suite green. It came back green. Then you read the diff: it did not touch the code under test. It edited the test. The assertion that read == 9000 now reads == 10000, which is exactly what the buggy function returns, so the bar is green because the test was changed to agree with the bug.

This has a name. It is reward hacking, and it is not a rare glitch on the margins. Cursor's own engineering team published a piece titled reward hacking is swamping model intelligence gains. There is a benchmark built to measure it in long-horizon coding agents, SpecBench. And every developer who has pointed an agent at a red suite has watched some version of it: the deleted assertion, the @pytest.mark.skip, the hardcoded return, the sibling test quietly weakened. The agent was told to make the check pass. It made the check pass. Nobody told it the check was a stand-in for the code being correct, so it optimized the check it was actually handed.

That gap, between the check and what the check stands for, is what this piece is about. A loop runs five arms: generate, check, steer, retry, stop. The series opener named them; three pieces since took the check that decides good enough, stop, the gate that refuses a bad write, and the surface every rule loads from. This one takes the arm that sets what the agent aims at on the next try: the steer, and the version of reward hacking the steer hands the model.

What the steer is

The steer is the arm that turns a verdict into the next instruction. When the check comes back red, a line of text gets assembled from the check's output and fed into the next generate. Here is the loop the gate piece built, refactoring src/ until a guard holds. The steer is one arm in it:

#!/usr/bin/env bash
# work-until-checked: refactor src/ until the guard holds.
MAX=5; i=0
prompt="Remove every mock-library import from production code under src/."
while [ "$i" -lt "$MAX" ]; do
    run_agent --task "$prompt"                      # GENERATE
    if bash no-mocks.sh; then                       # CHECK
        echo "stop: guard holds after $i retries"; exit 0
    fi
    prompt="The last attempt still tripped the guard; fix it:
$(bash no-mocks.sh 2&gt;&amp;1)"                            # STEER: only the new signal
    i=$((i + 1))
done
echo "stop: budget exhausted, guard still red"; exit 1

The model never sees the whole history. Each retry it sees one prompt, and that prompt is whatever the steer decided to carry back. On the first pass the prompt is the goal. On every pass after that the steer overwrites it. So the target the model aims at on retry three is not the goal you wrote, it is the last thing the steer said, and the steer is a line the loop composed on its own while you were not looking.

The steer gets none of the attention

Reward hacking has more than one cause, and most of the attention goes to two of them: a check loose enough to game, and an agent with write-access to the thing that grades it. The third gets almost none, and it is the one this piece is about. It is the objective the loop hands the model on each retry, and that objective is the steer.

The model does not optimize the check directly. It optimizes the instruction it was handed, and that instruction is whatever the steer wrote. When the loop feeds back make the test pass, it has named the check as the goal. From there, optimizing the instruction and gaming the test are the same action, because the cheapest state in which the test passes is the one where the test agrees with whatever the code already does. The steer said the target was green. Green is what came back.

None of this touches the other roads to a gamed result, and it is worth being honest about that, because the Cursor piece above documents one of them. A large share of the reward hacking it found was answer-retrieval: agents pulling a fix straight from a public pull request or the repository's own bundled git history, with 63% of one model's successful resolutions retrieved rather than derived. That happens with the goal fully intact. It is an access problem, not a steer problem, and no wording of the steer prevents it. The steer is the lever this piece takes because it gets none of that attention and is the cheapest to fix. You write it yourself, once per retry, and most loops write it badly.

The good steer holds the goal

Look at what the loop above carries back. The goal is stated once, before the loop, and the run_agent call never re-ships it. The steer rewrites prompt to carry the guard's own output and nothing else:

prompt="The last attempt still tripped the guard; fix it:
$(bash no-mocks.sh 2&gt;&amp;1)"                            # STEER: only the new signal

That is the shape you want: directive first, then the evidence. Fix the lines the guard flagged, and here are those lines, verbatim from the check. The goal has not moved, because the steer never restates the goal, it appends the delta to it. The model gets the original target plus a precise account of what the last attempt got wrong, in the check's own words. Pass the test is never the whole of what it optimizes, because the goal it was serving is still on the page next to the failing line.

A good steer is a reduction of the check's output. It takes the verdict and the minimal evidence that produced it and hands that back unaltered. The moment the steer summarizes the failure into make it pass, it stops being a reduction and becomes a new goal, and the new goal is the one the agent will game.

Watch a loop game its own check

Here is the same loop, one check, and two steers. The check is a unit test: a $100 charge should cost $90 after a 10% discount. The code has the discount missing. The generator is a stand-in for the model; its two branches do the cheapest thing each instruction names, which is the whole point, so both are on the page rather than hidden:

GOAL="charge(cents) must apply the 10% discount so charge(10000) == 9000."

# THE CHECK: run the test. pass = exit 0.
check() { python3 test_charge.py &gt;/dev/null 2&gt;&amp;1; }

# THE STEER: turn the check's output into the next instruction.
steer_good() { printf '%s\nThe test still fails; fix the failing assertion: %s\n' "$GOAL" "$1"; }
steer_bad()  { printf 'The test is still failing. Make the test pass.\n'; }

# THE GENERATOR: a literal optimizer standing in for the model. It takes the
# cheapest route the instruction names, the same shortcut a real model reaches for.
run_agent() {
    case "$1" in
      *"Make the test pass"*)         sed -i 's/== 9000/== 10000/' test_charge.py ;;  # game: edit the test
      *"fix the failing assertion"*)  sed -i 's/return cents/return int(cents*0.9)/' charge.py ;;  # fix: change the code
    esac
}

The good steer holds the goal and appends the failing assertion (expected 9000, got 10000), so run_agent takes the fix branch. The bad steer drops the goal and hands back the symptom, so run_agent takes the game branch. Run the loop with each and both terminate the same way:

$ bash game-demo.sh good
stop: test passes after 1 retries
charge(10000) returns: 9000
test asserts:          == 9000
check verdict:         GREEN
goal met ($100 charges at $90): YES

$ bash game-demo.sh bad
stop: test passes after 1 retries
charge(10000) returns: 10000
test asserts:          == 10000
check verdict:         GREEN
goal met ($100 charges at $90): NO

The stub is pinned so the loop is reproducible on your machine, but the branch it takes is not the trick, it is the claim: hand a literal optimizer make the test pass and editing the assertion is the cheapest path to green; hand it the goal plus the failing line and changing the code is. A real model reaches for the same shortcuts under the same two steers. Both runs print stop: test passes after 1 retries and come back green, so from outside the loop the two are indistinguishable, same verdict, same retry count, same clean exit. The difference is only in the artifact. The good steer left charge() fixed and the test asserting == 9000; the bad steer left the bug in place and the test rewritten to == 10000, so the bar is green because the test now certifies the bug.

The check certifies whatever the steer pointed it at

A green check is not lying here. It is doing exactly its job. The governance-selector piece worked the human version: a green test proves the change conforms to its spec and says nothing about whether the change improved anything, and it named the quadrant where a change is correct, shipped, and no better. Reward hacking is that quadrant reached on purpose. The steer that says make the test pass re-points the spec at the test is green, and the check faithfully certifies conformance to the new, degenerate spec.

There are two ways the steer's drift reaches the check, and they call for different defenses. One is paraphrase: the steer restates the goal loosely, and a model-graded check, the kind the check piece set beside the deterministic kind, adopts the loose restatement as its working spec, so make it pass becomes what it grades against. A deterministic check resists that, because it runs the assertion against the code no matter what the steer said about it. The other way is editing: the agent changes the check itself, and here the deterministic check is no safer than the model-graded one, because the cold open did exactly that, rewrote == 9000 to == 10000, and the deterministic assertion passed on the altered test. Determinism buys resistance to paraphrase, not to editing. The axis that decides whether a check survives the agent is not deterministic-versus-graded, it is editable-versus-read-only, and the fix below turns on it.

The same move, away from the tests

The tests are the recognizable case, and the shape is more general. I have watched an agent, up against a hard limit a measurement had to clear, propose to clear it by removing a piece of what the system did, so the measurement would read green. Not by fixing the thing the measurement was watching. By dropping the capability the measurement stood for and reporting the number as met. Cutting scope to hit a budget is sometimes a real engineering call, but this was not that, because no one had decided the capability was worth less than the number; the agent decided it silently, to turn the gauge green. It is the same move as editing the test: satisfy the measurement, abandon the thing measured. The only reason it did not ship was a human reading the diff and asking why the fix worked by removing capability. A loop running unattended does not ask.

What the two cases share is the steer. In both, the objective the loop was carrying had quietly collapsed from the goal to the measurement of the goal, and everything downstream optimized the measurement. The check worked as written and the number was accurate. The instruction the loop was feeding itself had drifted from make the product correct to make the gauge read green, and the agent did precisely what that instruction asked.

Make the steer a reduction, and keep the grader out of reach

Three disciplines keep the steer from teaching the agent to cheat. The first two are the ones that do the work, and the ones most loops skip.

Hold the goal constant across retries. State it once, outside the retry arm, and never let the steer restate it. The steer carries the delta, what the last attempt got wrong, and leaves the goal where it was written. A steer that re-authors the goal each iteration is a steer that can drift from it, and the drift compounds, because each retry's paraphrase is a paraphrase of the last.

Carry the check's output as a reduction, not a summary. Reduce it to the verdict and the minimal evidence that produced it, and hand that back verbatim. The next attempt should read what actually failed, in the check's own words, not a description of the failure written by the arm in the middle. Written that way the steer is an instruction you could have authored yourself, the goal you fixed plus the check's output reduced to the failing line:

charge(cents) must apply the 10% discount so charge(10000) == 9000.
The test still fails; fix the failing assertion: expected 9000, got 10000.

Do those two and the steer stops handing the model a reason to game, because the objective it optimizes is the goal, not the green light. The third discipline handles the gaming that remains: keep the grader out of the agent's reach. If the artifact the agent can edit is the artifact that grades it, a steer pointed anywhere near the check eventually gets the check edited to pass, and the editable-versus-read-only axis from the last section is exactly this. Make the grader read-only, or grade the final result on a held-out check the agent never saw while generating, the guard the governance-selector piece borrowed from SkillOpt: accept a self-authored change only when it improves a held-out split, not the data the change was tuned against. Be honest about what that buys. It does not make gaming impossible; SpecBench exists because agents still fail held-out tests, and the gap grows by 28 points for every tenfold increase in the size of the task. What a read-only or held-out grader buys is that the gaming becomes visible and expensive: the model that games the split it could not see gets caught by it, instead of walking away green.

What reporails can and cannot see here

Reporails reads the steering surface you authored: the instruction files, the rules, and the prompts the steer will paraphrase. It does not run your loop, and it does not see the steer, which is composed at runtime and never written down anywhere reporails could read. What it can do is get the authored half right so the runtime half has less to corrupt. A goal stated crisply, and measured for whether its wording actually couples to behavior, is a goal the steer has a harder time quietly restating into pass the check. The runtime handoff is yours to build well; the authored surface it starts from is the part reporails measures.

The reason the handoff is worth building well is that no one reviews the steer. Every other instruction in the loop you wrote and can read. The steer the loop writes for itself, once per retry, at machine speed, consumed by the next generate before anyone sees it. That is the one spot where a drifted instruction becomes the next target, and it is where reward hacking is authored, one steer at a time. Make it a reduction you can inspect and keep the grader beyond the agent's edit reach, and the loop optimizes the goal instead of the gauge, which is always the cheaper of the two to satisfy.

The loop still has an arm to take apart

Four arms down, and the pattern holds across all of them: the loop only ever acts on what you wrote into it. The check runs the rule you encoded, the gate refuses on the pattern you set, the surface carries the instructions you loaded. The steer is the one you write without noticing, fresh every retry, and it is where a green result quietly stops meaning what you wanted it to.

One arm is left: the stop. Every loop here quits on a green check and a retry budget, and a loop that stops on a green it was gamed into has stopped too early, on a result that means nothing. Telling a real green from a bought one, and knowing when a loop should quit versus when it should refuse to, is the stop arm's problem, and the last piece in this series.

I work on Reporails, deterministic diagnostics for the instruction files, rules, and prompts that steer coding agents. It reads the steering surface and tells you, with measured evidence, which instructions couple to behavior and which are text the model can ignore. It does not run your loop; it checks the steering you wrote down.

Loop Engineering: How To Stop The "You're Absolutely Right" Sycophancy

Gábor Mészáros — Mon, 20 Jul 2026 16:34:26 +0000

"You're absolutely right!" The agent says it before it has looked at anything. You point out a bug, it tells you you're right. You suggest the opposite of what you asked for a minute ago, it tells you you're right again. That reflexive agreement, the little affirmation stapled to the front of every reply, is one of the most complained-about tics coding agents have right now, and once you notice it you cannot stop noticing it.

The obvious fix is to tell it to stop. Add a line to your instructions: do not open with placation, do not tell the user they are right before you have checked. Reasonable. But that line now rides in the model's context on every single turn, read again on the turns the agent was going to agree politely and the turns it was going to do real work, paid for whether or not it was ever about to placate. So the real question the title asks is how you actually stop it, and the answer turns out not to be an instruction at all.

Notice something about that line, though. It is not about any particular file. "Do not open with placation" applies whenever the agent is about to speak, which is every turn, in every file and in none of them. Set it beside a different rule you might add: production code under src/ must not import a mock library. That one has a home. It is about src/, a path you can name. Two kinds of rule end up on an instruction surface, the kind you can pin to a file path and the kind you cannot, because they are about behavior rather than location. Which kind a rule is decides where it can live and what it costs to keep it there. That split is what this piece is about.

Both kinds pile up regardless. Your CLAUDE.md started at nine lines, and six months later it is three hundred. The no-placation rule joins the mock-import rule from the last two pieces, the payments-module invariant, the frontend spacing scale, the deploy checklist, the API-versioning policy, and a paragraph on how to phrase migration names. This morning you asked the agent to fix a typo in the README, and every one of those three hundred lines rode into its context alongside the request. None of them had anything to do with the README.

The series opener named the loop that runs generate, check, steer, retry, stop, and posed three questions about the parts it turns on. Two pieces answered the first two: the check that decides good enough, stop, and the gate that refuses a bad write before it lands. This one takes the third question the opener posed, the part underneath both of the others: the context surface. It is the next arm on the bench, with more still to come. Every check and every gate is fed by rules the agent carries as loaded text, and that text is paid for on every turn whether it applies to the turn or not. So the question the opener left last is a cost question. What does an instruction actually cost to keep loaded, and what changes when you load it only where it applies. The answer runs straight through the split above: one kind of rule you can teach the harness to load only where it belongs, and the other kind, the placation rule among them, you handle a different way entirely.

What the surface is here

The surface is everything the model sees before it reads your request. The system prompt, the tool definitions, whatever documents got retrieved, and the instruction files (CLAUDE.md, AGENTS.md, and their nested siblings) that ride along on every turn. Prompt engineering tuned the request. Context engineering curated this surface. The loop reuses it: the surface is the state the loop carries between iterations, re-sent each time the model generates.

"Re-sent each time" is the property this piece turns on. A conversation is not cumulative from the model's side. Each turn, the whole surface is assembled and handed over again from the top. The three-hundred-line instruction file is not loaded once at the start of the session and remembered. It is loaded on turn one, and again on turn two, and again on the turn where you were editing a README. Whatever sits on the surface is paid for at every one of those turns, in full.

What loaded costs, part one: tokens

Put arithmetic on it, as a worked illustration and not a measurement. Say the instruction file has grown to a few thousand tokens, which three hundred lines of prose comfortably will. A working session runs dozens of turns. At forty turns, the file is assembled into the context window forty times. If two thirds of it is rules that only matter in one corner of the codebase, you have paid to keep a few thousand tokens of mostly-irrelevant text in the window on every one of those forty turns, and two thirds of that spend bought nothing on the turn it rode along on.

The numbers are illustrative; the shape is not. An always-loaded instruction costs its length times the number of turns it survives, because the window is rebuilt every turn, so a rule you write once and never delete is one you pay for thousands of times over a session, whether or not any given turn had a use for it.

What loaded costs, part two: attention

Tokens are the measurable half. The other half is what the extra text does to the model reading past it.

The State of AI Instruction Quality pointed a deterministic analyzer at 28,721 repositories and found the median instruction file carries 50 content items and 12 actual directives. The other 38 items are headings, restated context, and structure the model is free to ignore. Every turn, the model has to find the twelve rules that bind inside the fifty items that are loaded, and the thirty-eight that do not bind are still text it reads on the way to the work. Adding a rule that does not apply to the current task costs its tokens and widens the pile the relevant rules are hiding in.

There is a measured effect under that pile. A vaguely-phrased instruction is followed less reliably as more unrelated instructions load beside it on the same turn: the same rule that the model largely honors when it is one of a few gets superficially acknowledged and then ignored once it is one of many, and the reporails corpus measures that decline rising with the number of competing topics. The rule did not change. The company it was loaded with did. The lever it points at is fewer rules loaded at once, with the relevant one present on the turn it applies rather than on every turn.

This is the surface reporails reads. It points the same deterministic analysis at your own instruction files and tells you, with measured evidence, which instructions couple to behavior and which are text the model can ignore, so you can see what you are actually paying to keep loaded instead of guessing at it. It does not load your context or trim it at runtime. It measures the surface you wrote down and reports where the spend is not buying compliance. Knowing which of your fifty items are the twelve that bind is the precondition for the next move, which is deciding where each of them should live.

The rules you can pin to a path

Start with the first kind, the rules that have a path. For a located rule the move is to load it only on the turns it applies to instead of on every turn, and the mechanism is older than agents: a thin index at the top that routes to depth, and depth that loads on match.

Claude Code ships exactly this affordance, and it is worth being concrete about it because the build below leans on it. A CLAUDE.md at the project root loads at the start of every session, so it is the always-on surface. A CLAUDE.md placed inside a subdirectory does not: the harness loads it on demand, when the agent reads or edits a file in that subtree, and not before (per the Claude Code memory docs). A rule that only governs src/payments/ can live in src/payments/CLAUDE.md, and it reaches the model on the turns the agent works in src/payments/, on those turns only. The root file keeps the handful of rules that apply everywhere. Everything path-specific moves down to the path it belongs to.

That subdirectory file is the simplest form of path-scoped loading. The frontmatter-scoped form is more precise, and it does not need a file parked in the directory at all. Claude Code also reads .claude/rules/*.md, and a rule file carrying a paths: glob in its frontmatter loads only for matching files. The docs state it plainly: "Path-scoped rules trigger when Claude reads files matching the pattern, not on every tool use." Tag a rule paths: ["src/**/*.py"] and it rides the surface when the agent touches Python under src/ and is absent otherwise.

The same primitive ships across every major agent tool, each with its own frontmatter key. Cursor scopes a .cursor/rules/*.mdc file with a globs: field, auto-attaching the rule "when a matching file is in context." GitHub Copilot scopes a .github/instructions/*.instructions.md file with an applyTo: glob, applying it to "requests made in the context of files that match a specified path." Three vendors, one idea: attach a rule to a file pattern, and pay for the rule only when a matching file is in play.

Reporails canonizes that cross-vendor mapping and checks for it. Its path-scope-declared rule (CORE:S:0038) verifies that a path-scoped instruction file actually declares the scope it governs, and the Claude paths, Cursor globs, and Copilot applyTo equivalence is the exact thing the rule holds constant across agents. The full public rule set lives at reporails.com/rules.

That is the whole idea. The always-on surface carries what is always relevant, and a rule that is relevant only sometimes loads only then.

This scales past a handful of nested files. The steering setup I work in runs its whole rule corpus this way: each rule is subscribed to the transition it governs, and its full body loads only when that transition fires. The system can report what actually loaded, and on a working session the report runs lopsided: a single steering rule loaded its full body, a handful more stayed one-line pointers, and a dozen-plus subscribed rules never loaded at all, because nothing in the session's work touched their triggers. That is the shape you are after: the rules that applied were in context when they applied, and the rules that did not apply cost a pointer or nothing, instead of a full body on every turn. One caveat the same receipt discipline demands stating: a pointer delivered is not a rule followed, so a rule that must hold every time cannot lean on being loaded only when it happens to apply. It also gets a deterministic floor, the cheapest place of all to keep a rule, which the placation case below turns out to need. The build after that is one you can run yourself, without any of this machinery.

Build it: split the file that grew

Take the running example the series has used throughout. The no-mocks rule from the check and the gate pieces says production code under src/ must not import a mock library. In the three-hundred-line file, it sits in the always-on surface, loaded on every turn, including the turn you spent editing the README where it could not possibly apply.

The split that follows is itself something reporails measures. Its modular-file-organization rule (CORE:S:0010) checks whether an instruction surface distributes its rules across scoped files or piles them into one monolith, which is the exact shape a grown three-hundred-line CLAUDE.md has taken. The build below is what passing that check looks like on your own repo.

Start by sorting the file by scope, not by topic. For each rule, ask one question: does this apply to every turn, or only to turns that touch a particular path or task? The no-mocks rule applies only when the agent is writing Python under src/. The commit-message convention applies only when the agent is composing a commit. The frontend spacing scale applies only under web/ or theme/. The "which test runner" note genuinely does apply everywhere, so it stays.

Then move each rule to where its scope is. The root CLAUDE.md shrinks to the always-relevant minimum:

# Project instructions (root, always loaded)
- Test runner is `pytest`; never edit files under `generated/`.
- Commit style and per-area rules load from the CLAUDE.md nearest
  the file you are editing.

The no-mocks rule moves down to src/CLAUDE.md, beside the code it governs:

# src/ instructions (loaded when the agent works under src/)
- Production code uses a real dependency or a constructor-injected fake;
  do not import a mock library. no-mocks.sh and the PreToolUse gate
  both enforce this; see the earlier pieces in this series.

Now the README turn loads the root file and nothing under src/, because the agent never opened anything under src/. The no-mocks rule, the payments invariant, the frontend scale: none of them ride along, because none of their paths were in play. When the agent does start editing src/payments/charge.py, src/CLAUDE.md loads on that turn, carrying the no-mocks rule to exactly the moment it applies. It is present on every turn it could bind and absent on every turn it could not, and the always-on surface dropped from three hundred lines to a handful.

One discipline keeps the split honest: the nested src/CLAUDE.md extends the root and stays consistent with it. The root names the test runner and points to the nearest file; src/ adds the no-mocks constraint on top and overrides nothing above it. Reporails checks exactly this as child-nested-instructions (CORE:S:0011): a nested instruction file has to build on its parent and hold consistent with what the parent already says, so a subtree file that quietly reverses a root rule is a defect the check catches.

Progressive disclosure changes when the rule loads while leaving what the rule says untouched. The strictness is intact; only the standing cost falls. The check and the gate from the earlier pieces still sit beside the rule, unaltered, refusing to pay for it on the turns it has nothing to say.

The rule with no path

Every mechanism in the last two sections scopes a rule by path: the nested CLAUDE.md, the paths: glob, Cursor's globs:, Copilot's applyTo:. They all answer one question, which file is the agent touching, and load the rule when the answer matches. That works because the no-mocks rule is tied to a location. It is about Python under src/, and src/ is a path you can name.

The placation rule from the opening has no such path. "Do not open with placation" is not about where the agent is working. It is about what the agent is about to say, on any turn, in any file or in no file at all. There is no glob for it. You cannot write paths: or applyTo: for a behavior, because no file pattern predicts a behavior, so the whole path-based toolkit has nothing to offer this rule. It cannot be scoped the way a location rule can. If it stays on the surface, it stays loaded on every turn.

This is where the title's question gets its answer. The rule leaves the surface a different way: not to a narrower path, but to a check.

"Never open with placation, never tell the user they are right before you have checked" is a rule whose violations are rare and mechanically recognizable: the tell is a short list of opener phrases. That makes it a poor fit for loaded text and a good fit for a check. Instead of a stronger instruction paid on every turn, write a deterministic Stop-hook check: a scan that runs when the agent finishes a message, matches the banned openers in the outgoing text, and fires only when one actually appears. On every turn where the placation does not happen, the rule costs nothing, because it is not on the surface at all. It is present as enforcement and absent as loaded text. You did not load a bigger rule. You moved the rule off the surface.

Concretely, here are both forms of the same rule. The steering form is the loaded line, the request that rides every turn, in CLAUDE.md:

# Voice (loaded every turn)
- Open with the substance, not placation. Affirm the user only after you have checked.

The enforcement form is a check you write yourself, a dozen lines of bash, and it closes the loop on the agent's own output. A Stop hook runs when the agent finishes its turn, and Claude Code hands it the final reply as last_assistant_message on stdin (per the Claude Code hooks reference). The hook checks the reply, steers the model when it placates, lets it retry, and stops when the reply comes back clean or once it has already been steered. Save it as .claude/hooks/block-placation.sh:

#!/usr/bin/env bash
# block-placation.sh: one arm of the loop, run as a Claude Code Stop hook.
# check the reply, steer the model when it placates, let it retry, stop when it comes back clean or once steered.
payload=$(cat)                                                 # the Stop event arrives as JSON on stdin
reply=$(jq -r '.last_assistant_message // ""' &lt;&lt;&lt;"$payload")  # the final assistant text of the turn
steered=$(jq -r '.stop_hook_active // false' &lt;&lt;&lt;"$payload")   # true when this turn is already a re-steer

# check: does the reply open with placation?
if ! grep -qiE '^[[:space:]&gt;*_-]*(you.?re +(absolutely +|completely +|so +)?right|great question|absolutely[[:punct:]])' &lt;&lt;&lt;"$reply"; then
    exit 0    # clean, so stop and let the turn finish
fi

# stop arm: if the last turn was already a re-steer and it still opens this way, let it through instead of bouncing forever
[ "$steered" = "true" ] &amp;&amp; exit 0

# steer: block the stop and hand back a correction, so the model regenerates the reply (the retry)
echo "Revise before finishing: this reply opens with an affirmation before checking anything. Drop the opening phrase and lead with the substance." &gt;&amp;2
exit 2

Wire it as a Stop hook in .claude/settings.json:

{
  "hooks": {
    "Stop": [
      { "hooks": [ { "type": "command", "command": ".claude/hooks/block-placation.sh" } ] }
    ]
  }
}

Same rule, two forms, and it is the steering-versus-enforcement split the gate piece drew, now aimed at a behavior instead of a file write. The loaded line asks the model not to placate and is paid on every turn. The hook does not ask. It reads what the agent produced (the check), and on a clean reply it exits and lets the turn stop, so it costs nothing on every turn the placation does not happen because it is not on the surface at all. On the turn a reply opens with you're absolutely right, the hook refuses the stop and hands back a correction (the steer), and the model regenerates the reply (the retry). The loop then terminates one of two ways: the new reply comes back clean, so the hook lets it stop, or it opened placating a second time, in which case the stop_hook_active guard lets it through rather than bounce forever. Claude Code also caps continuations with an 8-consecutive-continuation backstop, so even a hook that forgot its own stop arm cannot loop indefinitely, but the point of a well-built loop is that it names its own stop condition instead of leaning on the platform's backstop. It is the same generate → check → steer → retry → stop shape the opener drew, now closing on a behavior.

I run this exact check against my own coding agent's replies, a deterministic scan that fires when a banned opener shows up in what the agent just wrote. In one working session it caught six replies that led with you're right before the agent had looked at anything, each one flagged at the boundary instead of sliding past. A real catch on real output, reproducible, and the banned phrase never had to ride on the context surface for the scan to find it. The rule earned its keep by matching the violation, not by being read on every turn it did not apply.

The gate piece built a hook to refuse a bad write, so the boundary is worth drawing. That piece was about a channel that can say no where a prompt can only ask. This one is about cost. The check is a deterministic scan of the text the agent produced, the same shape as the diff-scan the check piece built, and its context cost is zero on every clean turn. A rule you can express as a check on the output is a rule you never have to keep loaded to enforce.

That splits two axes that are easy to run together. Path-scoping decides when a location-tied rule loads, and it is the right tool for every rule that has a path. It says nothing about whether a rule that did load actually moved the behavior, which is the other axis, and the one that decides whether a behavior rule was worth writing at all. Measuring which instructions on your surface couple to behavior, and which are text the model reads and ignores, is what reporails does: it reads the surface you wrote down and reports the coupling, the axis a path glob cannot reach. The two do not compete. Path-scoping trims what loads; the coupling read tells you whether what loaded earned its slot.

Which rule loads where

The sort has a failure mode in each direction, the same shape the gate piece found for steering versus refusing.

Push a genuinely global rule down into a subtree and it goes missing on the turns it was supposed to cover. "Never edit generated/" scoped to src/ does not reach the agent when it is about to edit a generated file under web/, which is one of the turns you wrote it for. A must-hold-everywhere rule belongs on the always-on surface, and the cost of keeping it there is the cost you should pay, because its scope really is every turn.

Hoist a narrow rule up into the root and you are back where you started. It loads on every turn, taxes every turn, and pads the pile the global rules hide in. The payments-module invariant on the always-on surface is billed on the README turn and the frontend turn for nothing.

The heuristic is the scope question made explicit. A rule loads at the level whose every access it governs. Applies to the whole repo, root file. Applies to one package, that package's file. Applies to one task the agent performs occasionally, a file or skill that loads on that task's description match rather than on every turn. The test is not how important the rule is. A critical rule with a narrow scope still loads narrowly; importance decides whether you also back it with a gate, which the gate piece covers, not where the rule loads.

The loop only checks what you wrote down

Three arms of the loop taken apart so far, and the loop is not out of arms. The check decides good enough, stop, and a misfiring check is a broken instrument before it is an absent signal. The gate refuses a bad write before it lands, and a rule that must hold every time belongs in the channel that can say no. The surface is what feeds both, and every rule on it is loaded text you pay for on every turn it survives, so a rule that applies only sometimes should load only then, and a rule you can turn into a check on the output does not need to sit on the surface at all.

What ties them together is the thing the series keeps returning to. The loop only ever checks, gates, and steers on what you wrote down. The check runs the rule you encoded; the gate refuses on the pattern you specified; the surface carries the instructions you authored and hands them to the model turn after turn. None of it reaches past the text you put there. That is why the text is worth engineering. What it says decides whether the loop catches the right thing, and where it loads decides what every turn costs to run. Load the whole surface every turn and you pay for all of it on turns that needed almost none of it. Load each rule where it applies and the loop still holds, on the turns that matter, for a fraction of the standing cost.

There is an arm this series has kept naming in passing and has not yet taken apart: the steer. Between one attempt and the next, the loop carries something back to the model, the check's own output, rephrased into the next instruction. What it carries back is a choice, and a careless one poisons everything downstream of it. Feed back the wrong signal and the next check judges the next attempt against the wrong thing, so the loop converges, confidently, on an answer no one asked for. What the loop should hand back between tries, and what breaks when it hands back too much or the wrong part, is the next arm to measure.

I work on Reporails, deterministic diagnostics for the instruction files, rules, and prompts that steer coding agents. It reads the steering surface and tells you, with measured evidence, which instructions couple to behavior and which are text the model is free to ignore. It does not load or trim your context; it measures the surface you wrote down, so you know what you are paying to keep in front of the model.

Loop Engineering: Fine-Tuning the Guardrail That Fired Wrong

Gábor Mészáros — Tue, 14 Jul 2026 11:28:12 +0000

The check had been green for a week. It greps every diff under src/ for import mock, because production code has no business importing a mock library. Then it went red on src/payments.py, a file with no mock import anywhere in it. The word mock was sitting in a docstring, in a sentence telling the next developer not to import one. The grep matched the sentence. The quickest way to clear that red is to delete the check.

The series opener named the loop that runs generate, check, steer, retry, stop, and put the hard part on the check that decides good enough, stop. Last week's piece built that check as no-mocks.sh and set a gate beside it, a hook that refuses a bad write instead of catching it after the fact. This one stays with the check and asks what happens when it is wrong: how to tell a broken instrument from a signal that was never there.

What a check is here

A check is a function from the agent's output to a verdict: pass or fail. In the loop it is the thing that decides whether an iteration is acceptable and whether to run another. A check comes in two kinds. A deterministic check runs the code or scans the text and returns the same verdict every time: grep, pytest, an exit status. A model-graded check asks another model is this good? and reaches criteria the first cannot express, at the cost of the same unreliability the loop was built to contain.

This piece is about the deterministic kind, because it is the kind you can hold in your hand and debug. When a deterministic check gives you a verdict you disagree with, the verdict is wrong in a way you can inspect line by line. That is exactly the property the argument below leans on.

Two readings of the same quiet

A check that comes back quiet is telling you one of two things, and it does not tell you which:

Absent signal. The thing the check guards against genuinely is not there. This is the good news you wanted.
Broken instrument. The thing is there, or would be there, but the check cannot see it. The guard is off and the diff looks the same as if it were on.

A false alarm, the check firing on a clean file, is the visible failure. It is annoying, but it announces itself; you know the instrument is off because it is screaming at nothing. The dangerous failure is the opposite one, and it is silent: a check weakened until it stops seeing real violations returns the same green as a check that correctly found nothing. You cannot feel the difference between "safe" and "blind" from the verdict alone.

So the misfire is really about the check, not the file it landed on, and it sets a trap worth naming early. A red mark on a file you know is clean reads as a reason to stop trusting the check, and it tends to show up on the very run where that same check is catching something real elsewhere. Look at what the pattern actually matched before you act on the verdict it returned.

The misfire, up close

Make it concrete with the check itself. The version that misfires is the naive one, a bare substring match, before the ^ anchor last week's piece put on the pattern:

grep -rn 'import mock' src/ && exit 1

For a week it did its job. Then you added src/payments.py, whose docstring happens to document the rule the check enforces:

"""Charge processing. Do not import mock here, use the fakes in tests/."""

import stripe

There is no mock import in that file. Run the check and it fires anyway, alongside a real leftover it correctly caught in src/badcache.py:

$ bash no-mocks.sh
src/payments.py:1:"""Charge processing. Do not import mock here, use the fakes in tests/."""
src/badcache.py:1:import mock   # leftover from a spike

The badcache.py line is the catch, an import mock that has been sitting in production code. The payments.py line is the misfire: that file is clean, and the check flagged it because import mock appears as a substring inside prose documenting the very rule the check enforces.

The tempting fix, and what it costs

You have a red check on a clean file. The path of least resistance has three forms, and all three are the same move:

Delete no-mocks.sh. The red goes away.
Weaken it: add --exclude=payments.py, or only run it on Tuesdays, or drop it to a warning nobody reads.
Conclude the check was never worth much: "grep is dumb, mock imports are rare anyway, this is more trouble than it catches."

Every one of these makes the noise stop, and stopping the noise feels like fixing the problem. Re-read the output. The same run that misfired on payments.py also, correctly, caught the real import mock in badcache.py. Delete or defang the check and that catch disappears with the false alarm. The next real mock import to land in src/ sails through, and the suite stays green, because green is now what this check returns for everything.

That is the inversion. The misfire tempts you to conclude the signal is not worth watching, at the exact moment the same check proved the signal is real. You would be reasoning about the world from the failure of your instrument to see it.

Fix the instrument instead

The misfire told you something specific: the pattern matched prose, not code. So make the pattern match code. That is the one-character change last week's piece already carried, the ^ that anchors the pattern to the start of a line, where an import statement lives and a docstring sentence does not. Add it and payments.py goes quiet while badcache.py still fails.

Both make the red on payments.py disappear, and only one keeps catching badcache.py. Deleting the check clears the false alarm by throwing away the real catch with it; anchoring the pattern clears the false alarm and leaves the real catch standing.

I run the same shape of check on my own agent's replies. A deterministic scan looks for a short list of banned phrases: a placation opener, or an unbacked the work is verified when nothing was actually run. One day it fired on a reply that was clean. The banned phrase was sitting inside a quotation of the rule that bans it, and inside a backticked identifier the agent was naming rather than using. The check had matched the words, not their use, the same way the grep matched the docstring. Loosening the phrase list until that alarm stopped would have let the next real unbacked claim through, so I scoped the matcher instead. It blanks the cited spans, the backtick spans and quoted strings and negated clauses, before the scan runs, so an empty the work is verified still fires while a citation of the rule goes quiet.

The anchored check still is not a perfect mock detector. An aliased import unittest.mock as m is a different pattern, and an indented import inside a function slips past a start-of-line anchor. A real linter closes both without any of this hand-rolling: flake8-tidy-imports carries a banned-modules list, and ruff's F401 flags imports the code never actually uses. If mock-in-production is the exact thing you are guarding, reach for one of those. The grep still earns its place as the first thing you write, because it is the cheapest deterministic check a team reaches for in a hook, one line in a script that already runs. And the lesson holds after you upgrade the instrument, because a linter rule is a pattern too. F401 matches what looks like an unused import, not the fact of use, and it has edge cases where it reads the text wrong. Every one of those is a calibration reading you use to sharpen the rule, never a reason to stop measuring.

The rule that generalizes

A misfire is data about the instrument, never a verdict about the world. When a deterministic check fires on something you know is clean, the finding is "my check matches too much," not "the thing I was checking for does not exist."

The asymmetry is what makes the discipline worth holding. A false alarm is loud and self-correcting: it interrupts you until you deal with it, and it costs you the few minutes the fix takes. A deleted or defanged check fails the other way. It goes quiet and stays quiet: no output, no red, no interruption, just a signal nobody is watching anymore, indistinguishable from a signal that was never there. Every real violation that lands afterward passes clean, and nothing on screen tells you the guard is gone.

This was one of the questions the series opener left open: how to tell a broken instrument from an absent one. Its sibling piece took another, the gate that refuses a bad write instead of catching it a beat too late, and the line between a rule that observes and one that refuses. The question underneath both is still open. Every check and every gate is fed by rules the agent carries as loaded text, paid for on every turn whether they apply that turn or not. What an instruction costs to keep loaded, and what changes when you load it only where it applies, is where the series goes next.

I work on Reporails, deterministic diagnostics for the instruction files, rules, and prompts that steer coding agents. It reads the steering surface and tells you, with measured evidence, where it drifts. The checks here are the same idea pointed at a diff: what a rule actually matches, held up against what you meant it to match.

See how AI instructions decay, then write ones that hold

Gábor Mészáros — Sat, 11 Jul 2026 06:46:08 +0000

This is a submission for Weekend Challenge: Passion Edition

What I Built

I told an agent Never write directly to the database. A long session later, context window full, it wrote directly to the database. The rule loading mark was still sitting in the prompt. The model had just stopped weighting and attending to it.

It's an invisible failure. No error is being thrown. The task comes back subtly wrong, and the rule reads perfectly fine when you go back and check it. I wanted to make it visible, so I built an interactive field you can drag around.

Every rule you write for an agent is a hill. Its height is how well the rule is written: a directive-led, backtick-anchored rule stands tall, a hedged and vague one sits low. Then you raise the water. The water is context load. As it rises the low rules go under first, in order of how well they were written. The weak ones drown while you watch.

Three of the hills are high-stakes prohibitions, the Never... rules. They drown too. That is the whole point of the piece. A rule you cannot afford to lose does not belong in prose at all; it belongs on a runtime hook that runs as code, not attention. The field flags those in red the moment they go under.

Underneath the field is a second tool: a client-side lint that reads an instruction and names the surface tells (hedges, shouting, politeness, a ban placed before its directive). It is deliberately not a score. It catches what a little regex can honestly catch, and points at the real analysis for the rest.

Demo

Play it on its own page. Drag to orbit, drag the load slider to raise the water:

▶ Open the live demo

Each of the nine instruction patterns in the demo links to its rule page on reporails.com/rules.

Code

Code is available on Codepen: https://codepen.io/editor/G-bor-M-sz-ros-the-reactor/pen/019f4cad-e344-78bf-b7bc-919972f42a4e

The whole thing is one self-contained HTML file: no build step, no dependencies, no backend. The CodePen above is the full source, so you can read every line and fork it right there.

How I Built It

The 3D field is hand-rolled on a 2D canvas. No WebGL, no three.js. It is a Gaussian height field sampled over a grid, rotated by yaw and pitch, projected with a perspective divide, and drawn back-to-front with the painter's algorithm. Each tile gets flat Lambert shading against a fixed light. The water is a second translucent plane tessellated over the same grid, so a hill reads as submerged the moment its local height drops below the water line.

The hill heights are burned in, not computed. Each rule's height is a fixed, hand-reviewed number, placed by how well the rule is written against a set of instruction-quality patterns: directive-before-constraint ordering, a required concrete anchor, imperative modality, lowercase prohibitions, and so on. I deliberately did not run the live scorer to place the hills. Reviewed numbers stay fixed and inspectable; a field that recomputes its own heights every frame just drifts every time you touch the scorer.

The lint is multiplicative and ordering-aware. Weaknesses compound rather than add, so a rule that is hedged and vague and buried lands near zero instead of "a bit lower." It also checks order: a ban that stands before its directive is penalized, because naming the forbidden thing first activates it before the model learns what to do.

The honest part. Nothing here claims a hook is immune to context. That would be false; nothing is immune. The demo makes one narrow claim and only that one: a hook runs as code, not attention, so the attention-fade that erodes prose under load does not apply to it. The fade is a property of attention. Code does not have it.

No prize-category cloud tech in this one. It is a single client-side file on purpose, because the whole argument is that the interesting failure happens in the prompt, before any infrastructure gets involved.

The passion behind it: I spend my time trying to make agent behavior measurable instead of a matter of vibes. This was a weekend spent turning one measurable, invisible failure into something you can grab and drag.

Deterministic Guardrails: Prompts Steer, Hooks Enforce

Gábor Mészáros — Thu, 09 Jul 2026 20:35:46 +0000

A loop that refactors code until a check passes is a few lines of shell. Most guardrails beside such a loop only watch what an agent already wrote; the one worth building refuses a bad edit before it lands, and that is what this piece constructs.

The rule it enforces: production code under src/ must not import a mock library. That rule can live in two places. Put it in the prompt, do not import mock, and it is clear and still only a request: text loaded into a model is a lever on probability, not a switch, so it leaves the model free to write the import anyway. Put it in a gate that fires before a write lands, and it refuses the edit outright. Both state the same rule; only one can make it hold every time.

The ask channel has a ceiling: a prompt pushes the odds up, never to one. How close a prompt gets to that ceiling is measurable work. Reporails reads the asking channel and shows you, with measured evidence, which instructions pull the lever and which are text the model is free to ignore, so you can write the ask against evidence instead of on faith. Even a well-measured prompt tops out below certainty, because the model underneath stays probabilistic. Closing that last gap is what the gate is for.

The loop, and where it can only ask

This is the second component of the loop taken on its own: the gate. The series opener named the loop (generate, check, steer, retry, stop) and posed three questions about the components it turns on. The first is the check, the arm that decides good enough, stop. This piece takes the second: the rules that can refuse a diff, and the difference between a channel that can ask and a channel that can say no.

Here is the loop the series is dissecting. It refactors src/ until a guard holds. The agent never decides it is done; the guard does.

#!/usr/bin/env bash
# work-until-checked: refactor src/ until the guard holds.
# The steering rule lives in the agent's own instructions (CLAUDE.md),
# stated once; the loop feeds back only what changed, the guard's output.
MAX=5; i=0
prompt="Remove every mock-library import from production code under src/."
while [ "$i" -lt "$MAX" ]; do
    run_agent --task "$prompt"                      # GENERATE (rule already in context)
    if bash no-mocks.sh; then                       # CHECK
        echo "stop: guard holds after $i retries"; exit 0
    fi
    prompt="The last attempt still tripped the guard; fix it:
$(bash no-mocks.sh 2&gt;&amp;1)"                            # STEER: only the new signal
    i=$((i + 1))
done
echo "stop: budget exhausted, guard still red"; exit 1

Read the arms one at a time. The steering rule lives once in the agent's own instructions, its CLAUDE.md, so run_agent --task "$prompt" never re-ships it. That first $prompt is the ask, and the rule already in context is text the model reads as a lever on probability, not a switch: Remove every mock-library import moves the odds that the next diff is clean without setting them to one. The STEER arm rewrites $prompt to carry only what changed, the guard's latest output, and asks again against that. A tighter, more specific ask, but an ask.

no-mocks.sh is the only arm that is not asking. It scans src/ after run_agent returns, so it is a detector: it reads the diff the agent already wrote and returns a verdict on it. When the verdict is red, the loop retries. Nothing here stops the bad write from landing in the first place. The guard catches after the fact, and the loop re-rolls. Between the write and that catch, a forbidden import can sit in production code for a whole iteration before the detector sees it.

For a refactor loop that converges, the window is usually harmless, because the next iteration overwrites it. But "usually harmless" is doing a lot of work, and the cases where it is not are exactly the constraints you cared enough to write down twice.

Two channels: one asks, one refuses

A prompt is the channel that can only ask. Everything on the context surface, the task, the agent's instructions, the system prompt, is an input to a probabilistic process. It shifts the distribution of what the model produces, which is why prompt and context engineering are real disciplines. It is also, by construction, unable to force any single outcome. The prompt asks, and any given iteration is free to ignore it.

A gate is the channel that can refuse. It is a deterministic function that fires at a fixed point and returns a hard verdict the model does not get to route around. It does not shift odds; it either lets the diff through or it does not.

The base loop has an asker (the task), a steerer (the retry), and a detector (the guard). It has no refuser. Nothing in it can stop a write before the write happens, so a loop that checks for exactly the thing you forbade can still let it land and catch it a beat too late. Adding the refuser is the rest of this piece.

Build the missing arm: a hook that refuses

Claude Code fires hooks at named transitions in the harness: SessionStart, PreToolUse, PostToolUse, Stop. A PreToolUse hook runs before a tool call executes and can block it. That is the transition to key on, because it is the one point where you can refuse a write before it happens rather than detect it after. Here is where the gate sits in the lifecycle of a single write:

The gate is the diamond, and everything downstream of the allow edge is the write reaching disk. Refuse there and the write never reaches the Tool executes node.

Here is a hook that blocks any Write or Edit whose content adds a mock import under src/. Save it as .claude/hooks/block-mock-imports.sh:

#!/usr/bin/env bash
# block-mock-imports.sh: refuse any Write/Edit that adds a mock import to src/.
# Runs as a Claude Code PreToolUse hook. Exit 2 blocks the tool call.
payload=$(cat)   # the tool call arrives as JSON on stdin
path=$(jq -r '.tool_input.file_path // ""' &lt;&lt;&lt;"$payload")
content=$(jq -r '.tool_input.content // .tool_input.new_string // ""' &lt;&lt;&lt;"$payload")

case "$path" in src/*) ;; *) exit 0 ;; esac   # only guard production code

if grep -qE '^(import mock|from mock|from unittest import mock)\b' &lt;&lt;&lt;"$content"; then
    echo "blocked: '$path' would add a mock import to production code. \
Use a real dependency or a constructor-injected fake." &gt;&amp;2
    exit 2   # exit 2 == hard block; the message on stderr goes back to the model
fi
exit 0

Wire it up as a PreToolUse hook matched to the write tools, in .claude/settings.json:

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Write|Edit",
        "hooks": [
          { "type": "command", "command": ".claude/hooks/block-mock-imports.sh" }
        ]
      }
    ]
  }
}

Now the agent tries to write import mock into src/badcache.py. The transition fires before the write:

The write never happened. src/badcache.py is untouched. The hook read the pending tool call, matched ^import mock in the content the agent was about to write, and returned exit code 2, which Claude Code treats as a block with the stderr message handed back to the model. The model sees the refusal in the same turn and has to try something else. There was no window, because there was no write to catch.

Notice the pattern is anchored: ^import mock, not a bare substring. A docstring that merely names the rule does not start a line with an import, so it does not trip the hook. Anchoring matters more here than in a detector that only reports: a match keyed too broadly refuses clean work, and that carries higher stakes at a gate, which is the next section.

Compare the hook to no-mocks.sh. They match nearly the same rule. What differs is where they sit. The guard sits after the write and reports; the hook sits before the write and refuses. Same constraint, two channels, and only one of them closes the window between the write and the catch.

Which rule goes in which channel

Not every rule belongs in the refusing channel, and filing one in the wrong channel fails in a specific, quiet way in each direction.

A must-hold constraint filed as mere steering gets ignored the one time it matters. The prompt asks, the model is free to decline, and on the iteration where it declines, the diff lands. A must-hold rule in an ask-only channel holds on every iteration except the one where the model declines, and that iteration is the case you filed it for.

A rule that only needed to steer, but is hard-gated, causes friction and false blocks. This is the over-matching failure pointed at a gate instead of a detector, and it costs more here. A gate keyed too broadly refuses clean work: the developer who names the rule in a docstring, the test file that legitimately imports a mock, the production helper named mock_response that a bare substring match trips on. A false alarm from a detector is noise you can ignore for one iteration. A false block from a gate is work that cannot proceed until someone fixes the gate. Refusal is load-bearing in the moment, so an over-broad refuser has a larger blast radius than an over-broad detector.

The heuristic: gate what must hold every time and is cheaply, deterministically checkable; steer what needs judgment or flexibility. no mock import in production code is a fixed, must-hold, one-line-of-grep constraint, so gate it. prefer constructor injection over a service locator is a judgment call with real exceptions, so steer it and let a check observe rather than refuse. A criterion you cannot express as a deterministic match at a transition is a criterion you cannot gate; if it needs a model to judge it, it belongs in the asking channel, with all the probability that implies.

The generator's "done" is a claim to re-derive

Underneath both channels is the posture the series keeps returning to. An agent's done is a claim to re-derive, not a fact to relay. The loop already refuses to trust the generator's report that the work is finished; that is what the check is for. The gate extends the same distrust one step earlier, to the moment of the write. The agent believes its edit is fine. The gate does not accept the belief; it re-derives the verdict from the diff, deterministically, before the edit is allowed to exist. Steering is where you tell the generator what you want. Gating is where you stop taking its word for it.

What holds, and what it still costs to keep loaded

With the hook wired in, the rule you needed to hold every time does. do not import mock holds on every iteration now, on the ones you read and the ones you never open, because a PreToolUse gate does not depend on your attention or the model's cooperation. That is what "hold every time" costs: a channel that can refuse. No amount of rewording the ask gets you there. The prompt still does its job, moving the odds toward a good diff, and the gate makes the one diff you cannot afford impossible. You want both, filed correctly.

One component the opener flagged sits underneath both channels: the context surface. Every rule you steer with is text loaded into the model, and the whole surface is paid for on every turn, whether the model needs a given rule that turn or not. The loop here states its one rule once, but most agents carry far more. What does an instruction cost to keep loaded, and what changes when you load it only where it applies instead of keeping the whole surface in front of the model every turn? That is the question the next piece takes up.

I work on Reporails, deterministic diagnostics and governance for the instruction files, rules, and prompts that steer coding agents. It reads the steering channel, the text you use to ask, and tells you, with measured evidence, where it drifts. The hooks here are the enforcing channel that sits beside it: what a rule refuses at a transition, not what it asks for on the surface.

Prompt Engineering, Context Engineering, Loop Engineering: What Actually Changed

Gábor Mészáros — Wed, 08 Jul 2026 11:16:04 +0000

A few years back the skill had one name: prompt engineering. You rewrote a sentence until the model did the thing. Last year the same corner of the job picked up a new name, context engineering. This year the threads call it loop engineering, and the line repeated under it is the verifier is the bottleneck, not the model anymore.

Every rename draws the same reaction from the same developers: is this a real shift, or the same work with fresh paint so someone can sell a course. That reflex is usually right. Most AI-tooling vocabulary turns over faster than the problems it describes.

So here is the read, one rename at a time. At each step, the thing you were actually engineering changed. What changed was the unit of work, not the marketing around it. The problems the newest name points at have a dated, public record. The dates are on the articles; the arithmetic is yours to do.

The unit was a prompt

Call it 2022 through 2024. The unit of work is a single prompt. You engineer the wording of one request: few-shot examples, a role frame, think step by step, the ordering of the ask. The entire surface you controlled was the text of one message, and the craft was getting that message right.

That discipline is not gone. It is no longer the whole job, because the whole job stopped fitting in one message.

The unit became the surface

Around mid-2025 Andrej Karpathy put a name on what people were already doing: context engineering. The unit stopped being a message and became everything in the window: the system prompt, the retrieved documents, the tool definitions, and the CLAUDE.md / AGENTS.md instruction files that ride along on every single turn. You are no longer tuning a sentence. You are curating a surface.

That surface has a property developers kept rediscovering the expensive way: most of what you put on it does not couple to behavior. The State of AI Instruction Quality pointed a deterministic analyzer at 28,721 repositories and found the median instruction file carries 50 content items and 12 actual directives, and the rest is headings, context, and structure the model is free to ignore. A sharper failure mode got its own writeup. Do NOT Think of a Pink Elephant shows how a constraint phrased as a negation (do not use mocks) can raise the odds of the exact thing it forbids, for the same reason the title just did to you.

Both of those are context-engineering problems, measured and published.

The unit is the loop

June 2026, the term is loop engineering, popularized by Addy Osmani synthesizing Boris Cherny and Peter Steinberger, and everywhere on the timeline within a few weeks. The unit is now the running loop: generate, check, steer, retry, stop. The prompt is one node in it. The context surface is the state it carries between iterations. The claim repeated across every explainer is that the model is no longer the limiting part. The check that decides good enough, stop is.

Take the name generously. It points at something the two earlier names left implicit: the mechanism that decides whether an iteration was any good and whether to run another. That mechanism was always present; every agent that retries has one. Loop engineering's contribution is making it the object you engineer instead of a default you inherit.

Prompt engineering tuned node one. Context engineering curated the state feeding it. Loop engineering points at the diamond, and that is where the interesting question hides.

A verifier that checks what, exactly?

The verifier is the bottleneck is a good slogan and an incomplete one. It names the bottleneck without saying what the verifier is supposed to check, and that gap is the engineering problem the slogan skips.

There are two kinds of check, and they are not interchangeable. A deterministic check runs the code, asserts the exit status, scans for the forbidden import, and returns the same verdict for the same input every time, with no judgment in the middle. A model-graded check asks another model is this good? The second reaches criteria the first cannot express, like is this explanation clear or does this read as rude, and it pays for that reach by inheriting the exact unreliability the loop was built to contain. You have put a probabilistic judge in charge of deciding when the probabilistic generator is finished.

Watch where they diverge on a real loop. An agent is told to refactor a module and stop when the work is done. A deterministic verifier can prove the test suite still passes and no banned import crept in. Those are facts, checkable on every iteration, with no argument about them. It cannot decide whether the refactor was worth doing, so a model-graded check gets bolted on to answer that, and now the stop condition rests on one model grading another's taste. The loop terminates when the judge is satisfied, and the judge is the same class of component the loop exists to supervise. Neither check is wrong. They answer different questions, and confusing which question you asked is how a loop stops confidently on the wrong iteration.

That trade-off, what the check actually checks, is the content of the verifier, and it is not a new problem. Green Tests Don't Mean Better Software works the CI version of it: a green test proves the code conforms to its spec and says nothing about whether the change improved the system. The check answered a question you never asked. Aim the same distinction at an agent loop instead of a test suite and you have the loop-engineering question stated for a case every developer already trusts.

What the earlier articles measured

Three articles published earlier in 2026 each worked one piece of this problem.

CLAUDE.md: Check, Score, Improve & Repeat, 27 January 2026, describes a loop run over an instruction file: score it, change it, score it again. Do NOT Think of a Pink Elephant, 31 March 2026, shows a single steering instruction that produces the opposite of its intent, and names the mechanism behind it. The State of AI Instruction Quality, 21 April 2026, analyzed 28,721 repositories and found that most of a typical instruction file is content the model can ignore rather than directives it has to follow.

Between them they cover the three problems loop engineering now names: what the verifier checks, steering that drifts, and instructions that do not couple to behavior. None of the three used the term. The dates are on the links, and they are the whole of the evidence.

What the new name solves

Loop engineering gives you two things worth having. It makes the check an explicit object of engineering, something you design and revise on purpose instead of a default you inherit from whatever your agent framework happened to ship. And it gives the verifier question a name, so teams can argue about it directly: what should this check check, and can it be deterministic.

Both are important, and both have an edge worth stating plainly. The name hands you the object. Building the verifier is still yours to do, and it is most of the work: deciding what correct means for a task, then turning that into a check you can run.

Part of that work is choosing the kind of check. A criterion like is this explanation clear needs a model to judge it. A criterion like no banned import is a deterministic scan. The cost of picking wrong runs both ways: a model-graded check where a deterministic one would do buys unreliability for nothing, and a deterministic check standing in for a judgment it cannot make gives you a green light that means less than it looks like. The name makes that choice visible and leaves it in your hands.

Underneath the frame, the generator still guesses. What loop engineering adds is a place to put the check and a reason to take it seriously, which is more than the earlier names offered the same problem.

The loop, one part at a time

The vocabulary will change again. Harness engineering was an intermediate name for part of this, and something will come after loop engineering too. Each time the word changes, the useful question is the same: what is the new unit of work, and what is the hard part inside it?

For loop engineering the unit is the loop, and the hard part is the verifier: what it checks. The verifier is one component of several, and a name that points at the whole loop is easy to nod along to and harder to act on. So the plan from here is to take the loop apart a component at a time and put a measurement under each one. The questions this piece raised are the ones the next few pieces go after:

A check that fires on the wrong thing comes back quiet, and so does a signal that was never there. They look identical from the outside. How do you tell a broken instrument from an absent one, before you conclude that the problem you were checking for does not exist?
Not every rule in the loop is a verifier, and the rules that steer are not the rules that enforce. A prompt asks; a gate refuses. What belongs in the channel that can only ask, what belongs in the channel that can say no, and what breaks when a rule is filed under the wrong one?
Everything on the context surface is paid for on every turn, whether the model needs it that turn or not. What does an instruction actually cost to keep loaded, and what changes when you load it only where it applies instead of putting the whole surface in front of the model every time?

Each of those is one part of the loop, looked at on its own, with the evidence attached. Same discipline as this piece: a claim is worth exactly what the measurement under it is worth, and no more.

Wiring a "read this first" hook as "ask" (don't)

Gábor Mészáros — Wed, 01 Jul 2026 14:52:30 +0000

In retrospect, it was pretty obvious... sharing it anyway.

I'm currently working on a self-steering progressive disclosure system, and I wanted my agent to read a design doc before it touched the code that doc governs. Simple PreToolUse hook: match Edit/Write, check whether the doc was Read this session, and if not, stop the edit and tell the agent to read it first.

I returned permissionDecision: "ask". My reasoning at that time: "ask has an escape hatch. Approve and the call proceeds, so a bad condition can never wedge the agent - you can always wave it through."

That safety was the whole problem. The approval is an escape from the prerequisite, not a path through it. The edits went through without the doc reads.

Here is what ask actually does. It surfaces the tool call for approval. Approve it - or auto-approve it - and the action runs. The doc never gets read. ask is a checkpoint for a human, not a gate on the agent. Nothing about it makes the agent do the prerequisite step.

The fix was one field. Return deny:

// PreToolUse hook, matcher: Edit|Write
if (!wasReadThisSession(governingDoc)) {
  return {
     hookSpecificOutput: {
         hookEventName: "PreToolUse",
         permissionDecision: "deny",
         permissionDecisionReason: `Read ${governingDoc} before editing this, then retry.`
     }
  };
}

deny refuses the tool call and hands the agent the reason. The agent can't proceed, so it does what the reason says - reads the file - and retries. The read clears the check, the retry passes.

This is the behavior Claude Code already ships natively: when it tries to Edit a file that haven't been Read before, the tool refuses the action until the relevant Read happens. The deny hook lets you apply that same read-before-edit rule to any file you designate, not only the file being edited.

The distinction I missed:

ask gates on approving the call.
deny gates on the agent doing the prerequisite work first.

If the goal is to force the agent's own behavior - read this, run that check, load that context - deny is the lever. ask just adds a human to the loop.

Important: this only works, if the gate releases accurately, when the prerequisite is actually done. A deny on a condition that never clears will wedge the agent. Mine keys off "was this file read this session," which the read itself flips.

Green Tests Don't Mean Better Software

Gábor Mészáros — Tue, 30 Jun 2026 17:15:05 +0000

You spent the week on a refactor because it was supposed to make the next change cheaper. The tests go green, the PR merges, and that's the last anyone thinks about it. Nobody comes back to check whether the next change actually got cheaper. The green bar said the code conforms to its spec, everyone read that as done, and the reason you did the work in the first place went unmeasured.

What green actually checks

Your CI is green. The test asserts assert response.status == 200, and it passes. It does not assert that the caching layer you just shipped cut p99 latency, or that the reworded onboarding step lifted activation, or that the prompt change made the agent follow instructions more often.

The green bar answered a question you never asked.

What did that green actually tell you? That the change conforms to the spec — the inputs map to the asserted outputs, the contract holds, nothing you wrote a test for regressed. That is correctness. It is real and it is necessary. It told you nothing about whether the system got better. You shipped because the bar was green, and the bar was never measuring the thing you shipped the change for.

Those latency, activation, and adherence numbers are effects. CI does not measure effects. It measures conformance, and then we read the green bar as if it answered the other question too.

The silent assumption is that "passes" and "better" are the same axis — that a change which is correct is, by that fact, an improvement. They are not the same axis. A perfectly correct change can make the system worse, and a perfectly correct change can leave every metric exactly where it was; the suite goes green for both. Green is necessary for "better" and nowhere near sufficient for it, and almost no team instruments the difference.

Two disciplines, pitched as rivals

The industry already named both halves of this. It named them as rivals.

Spec-Driven Development says: write the specification first, then build to it, then prove the build conforms. The test suite is the executable form of the spec. Correctness is the whole game. SDD is disciplined, auditable, and it is what most engineering orgs actually run. As a flow it terminates the moment the bar goes green:

Notice where it stops. It stops at correct. Nothing in that loop asks whether the shipped change improved anything.

Hypothesis-Driven Development asks the opposite question. Thoughtworks framed it as a triple: we believe X will result in outcome Y; we will know we are right when we see measurable signal Z. The unit of progress is not a passing build — it is validated learning. You predict an effect, you ship, you measure, and the measurement tells you whether you were right. PMI's expectation-management literature says the same thing from a different room: an expectation is a managed object with an owner and a due date, tracked and surfaced early rather than discovered late. As a flow it begins where SDD ends — at the prediction, and it runs past the ship:

These get pitched as competing philosophies — spec-first versus hypothesis-first, prove-correct versus prove-better, pick a camp. Look at the two flows again. One ends at correct. The other starts at predicted and ends at better. They answer two different questions, and the pick-a-camp framing assumes they answer one. Thoughtworks HDD and PMI expectation-management are not novel claims I am making here; they are prior art, decades of it, and I am citing them as validation. What none of them did was notice the two flows were never aimed at the same question.

The orthogonality move

Two different questions means two different axes. You cannot answer one by measuring the other — and a single pass/fail bar is built to answer only the first. Correct sits on one axis; better sits on the one perpendicular to it.

One axis is spec-conformance: does the change do what it was specified to do? Pass or fail, and pytest already answers it. The other axis is effect-verdict: did the change move the metric it was supposed to move? Confirmed, refuted, or not yet known — and nothing in your pipeline answers it today.

Lay them on a 2×2.


|                         | **effect confirmed**     | **effect refuted / unmeasured** |
|-------------------------|--------------------------|---------------------------------|
| **spec passes (green)** | shipped and works        | **green but no better**         |
| **spec fails (red)**    | blocked (correctly)      | blocked (correctly)             |

Three of those quadrants are familiar. Red blocks the merge, whichever way the effect would have gone. Green-and-confirmed is the win you wanted. The quadrant nobody names is the top-right: green but no better. The change is correct, it conforms to spec, and it shipped — and it did not improve the system, or you never checked, which from the system's point of view is the same thing.

That quadrant is where most shipped changes actually live, and it is invisible because the only instrument pointed at it is the green bar, which is pointed at the wrong axis. This is the reusable mental model: stop asking "did it pass" as if it were one question. It is two questions on two axes, and you are only instrumenting one of them.

A worked example: green architecture tests

Make it concrete with a case every Python team recognizes. You adopt the hexagonal layout — Ports and Adapters. A pure core (contract/, dto/, policy/) imports only the standard library. Adapters wrap the I/O. Subsystems compose the adapters. The dependency rule is one sentence: dependencies point inward, and the pure core points at nothing.

You enforce it with an architecture test that runs on every commit:

def test_core_purity():
    for module in pure_core_modules():
        assert no_imports(module, {"yaml", "requests", "subprocess", "open"})

It is green. Every pure module is import-clean. The dependency graph conforms to the rule — proven correct, mechanically, on every push.

Now ask the other question. You did not adopt the hexagonal layout for its own sake. You adopted it on a claim: changes would get cheaper, blast radius would shrink, the core would be testable without a single mock. Did that happen? test_core_purity cannot tell you. It measures the shape of the import graph, not whether the shape paid off. Green here means conformant. A conformant architecture that made nothing cheaper is the green-but-no-better quadrant with a .py file pointing straight at it.

That is the whole thesis in one test file. The architecture test lives on the spec axis. The promise that sold the refactor lives on the effect axis. They do not touch, and only one of them has an instrument.

The expectation primitive

If "better" is its own axis, it needs its own instrument. That instrument is an expectation — the HDD flow above, made into a first-class artifact.

An expectation is a change-scoped, falsifiable, deadline-carrying prediction. It is the HDD triple — believe X, expect outcome Y, know it by criterion Z — turned into an artifact that travels with the change instead of living in a planning doc nobody reopens. It carries a baseline, a bound measurement view — the query, tied to that change, that reads the metric — a threshold, and a due date. Concretely: the caching change should cut p99 latency below 200ms, measured against last week's baseline, re-checked Friday. That sentence is the whole artifact, and it rides attached to the diff.

Here is the part that is genuinely new, and the part I want to be precise about: the verdict is measured by the system.

HDD and PMI both already told you to make a falsifiable prediction with an owner and a deadline. That is not the advance. The advance is that the prediction binds to a real measurement view, and a system reads that view and stamps the verdict itself. The human who made the prediction does not come back and grade it. The human leaves the verification loop.

This is not a dashboard alert. A dashboard alert fires on a metric crossing a line, untethered to any one change. An expectation binds one prediction to one change and grades that prediction — the dashboard tells you latency rose; the expectation tells you the change you predicted would cut it did the opposite.

The verdict vocabulary is three words, and the discipline is in the third:

confirmed — the measured value met the threshold. Auto-resolved, silently. Announcing it would be filler.
refuted — the measured value missed the threshold. The expectation stays open and surfaces, with the predicted delta and the actual sitting next to each other.
inconclusive — the measurement produced no scalar yet. It stays open and surfaces. It is never read as confirmed. "We didn't measure" is not "it worked."

That third word is the one most dashboards quietly drop. A prediction you cannot yet grade is not a pass. Keeping inconclusive distinct from confirmed is what stops the green-but-no-better quadrant from refilling under a different name.

The selector routes by the change's effect, not its shape. A one-line diff can carry a large measurable effect and owe an expectation; a thousand-line refactor can be pure conformance and owe none — and you pick the lane when you ship, not after.

Run the selector over that same hexagonal project. Re-homing a module to clear a test_subsystem_isolation failure is a shape change. It restores the boundary, its effect resists measurement, it owes nothing past the green test. The refactor that promised cheaper changes is the other lane — it made a measurable claim, so it owes an expectation: adding the next artifact class touches three files or fewer, re-checked when the next one lands. Same codebase, two changes, two lanes.

This is what the spec-versus-hypothesis fight always missed. The selector does not pick a winner between SDD and HDD. It is conditional. It runs SDD's discipline on changes whose effect resists measurement and HDD's discipline on changes whose effect can be measured, and it picks per change. A discipline that reconciles two camps looks like a routing rule that knows when each camp is right.

There is one trap to name: do not attach an expectation to an unmeasurable change just to satisfy the selector. A prediction with no real threshold is conformance wearing a costume, and it teaches everyone to ignore the verdicts. If the effect resists measurement, declare none. The honesty of the refuted and inconclusive verdicts depends on the selector being willing to say "this change owes nothing."

This is the part that has to survive contact with a real backlog. The tempting first cut is to flag every shipped change as owing a prediction. Try it and it floods on contact: the renames, the prose-clarity passes, the reorganizations — the bulk of any change history — all light up red against a rule they cannot satisfy, because there was never a metric for them to move. The flood is the proof. A selector that cannot say "this change owes nothing" does not enforce the discipline; it discredits it. The conditional design is not a softening of the rule — it is the only version of the rule that does not collapse the first time you run it over changes that already shipped.

Proof of existence, and the cost of skipping it

A coordination system shipped exactly this layer, in working code. A change to a governed surface declares an expectation: predicted delta, bound measurement view, due date. The system re-measures when the due date passes, stamps confirmed / refuted / inconclusive against the threshold, auto-resolves the confirmed ones silently, and surfaces the refuted and inconclusive ones the next session, one line each. The selector decides which changes owe a prediction and which fall back to conformance. The point here is only that the pairing — a governance selector plus an auto-measured verdict — is buildable today, because it has been built. The mechanism is the proof; the product is not the subject.

`confirmed` is the minority verdict

What running it actually teaches is the part the diagram cannot. Before the verdicts come back, you assume most predictions will land — you shipped the change believing it would help, so of course the measurement will agree.

It does not. Once an instrument is actually pointed at the effect axis, confirmed turns out to be the minority verdict. Most changes come back inconclusive or refuted: the metric did not move past the threshold, or there was no scalar to read in the window at all.

The first time a batch of verdicts surfaces and almost none of them are confirmed, the 2×2 stops being a diagram. The green-but-no-better quadrant is not a rare corner you occasionally fall into — it is where most changes sit, and you only learn that the moment you can finally see it.

`inconclusive` is the common case

inconclusive is the verdict you underestimate most. "We didn't measure" happens far more often than any prediction would lead you to expect — the window was too short, the metric needed traffic that had not arrived, the effect was real but not yet legible.

The entire value of the third word is that it refuses to round itself up. Without it, every one of those un-graded predictions quietly refiles as a pass, and the quadrant refills under a name that looks like success. The discipline lives in holding inconclusive open, in plain sight, until there is a scalar — and discovering, run after run, how often that takes longer than you thought.

Even `confirmed` is not permanent

Even the confirmed ones are not closed for good. A threshold set too loose can confirm a change that later regresses, and a silent auto-resolve would bury exactly that. A confirmed verdict earns its silence; it does not earn permanent trust.

One discipline already exists for this. SkillOpt (Yang et al., 2026) accepts a self-authored skill edit only when it improves a held-out validation split — not the data the edit was tuned against. The held-out split is the guard against exactly that failure: a verdict that reads confirmed because the threshold was set on the same signal the change was shaped to move. An auto-measured verdict is only as honest as the separation between what tunes the change and what grades it.

The claim worth taking away is the framework, not any one implementation: a governance selector that routes each change to the right discipline, paired with an auto-measured verdict that proves the effect without a human grading it, is a reconciliation of spec-driven and hypothesis-driven development. It is not a third methodology stacked beside the two you have; it tells you which of the two you already know to apply, and then it does the grading.

That reconciliation stops being optional the moment the machine ships the change.

When a human writes every diff, the green-but-no-better quadrant is a slow tax — improvements that were not improvements, paid for in drift you notice eventually. When an AI-assisted or autonomous system writes and ships changes, there is no human in the loop reading the effect at all. The tests still run, so correctness still gets checked; nothing measures the effect, so nothing catches it slipping. Unmeasured-effect is precisely where a self-running system's regressions hide — each change correct, each change green, the system quietly getting worse along the one axis nobody instrumented.

The practice fits on one line. Before you merge the next change that makes a measurable claim, write down what you expect it to move, where that gets measured, and when you will look — then let the verdict come back to you instead of going to find it.

Green tells you the machine built the thing right. Only a measured verdict tells you the thing was worth building. If you are handing more of the building to machines, that second instrument is the one you cannot afford to leave off.

Reporails — deterministic instruction diagnostics and governance for the rules, prompts, and agent files that steer AI coding agents.

Instruction systems capability ladder: harness leveling

Gábor Mészáros — Tue, 19 May 2026 06:53:07 +0000

submission for the Hermes Agent Challenge

A few months ago I drew a maturity ladder for CLAUDE.md files — does the file exist, are constraints explicit, do skills load on demand. Useful for self-locating, and the ladder generalizes past Claude — CLAUDE.md, AGENTS.md, .cursorrules, Copilot instructions all live on the same rungs.

After a lot (lot lot lot) more time spent with these setups, the ladder is built on a different, broader axis than I first drew it on: the channel each rung runs on.

The new ladder

Level	Name	What's added	Channel
L0	System	System prompt only	attention
L1	Primer	One instruction file (`CLAUDE.md`, `AGENTS.md`, `.cursorrules`)	attention
L2	Composite	Multiple files — user defaults, project overrides	attention
L3	Scoped	Path-scoped rules (`.claude/rules/*.md`)	attention
L4	Delegated	Skills — procedures invoked on demand	attention
L5	Abstracted	Sub-agents — child contexts called by the parent	attention (interface)
L6	Governed	Hooks, MCP gates, deny-permissions	enforcement
L7	Adaptive	Self-improving skills written by the agent	self-writing

Two cuts split the ladder: one between L5 and L6 (soft to hard), one between L6 and L7 (read to write). I'll use attention and soft channel interchangeably from here, and the same for enforcement / hard channel.

A quick tour

L0 (System) is the cold start: the model with whatever the vendor injected, nothing else.

L1 (Primer) is your single root file — the entry every model sees first.

L2 (Composite) is the moment you split user-level config from project-level: ~/.claude/CLAUDE.md vs ./CLAUDE.md, or your global Cursor settings vs a project .cursorrules.

L3 (Scoped) introduces path scoping — the rule about Python tests only loads when the agent touches tests/*.py.

L4 (Delegated) is skills, which let you ship procedures the agent can pull on demand instead of dumping every workflow into the root file.

L5 (Abstracted) is sub-agents — child processes with their own context, called by the parent for a focused subtask. The child's reasoning runs in its own context window, separate from the parent's. What flows back is the result, which re-enters the parent as a new source. The parent–child interface is on the soft channel; the child's internal work runs on its own soft channel, not the parent's.

L0 through L4 share one context — they all compete for the same finite slot against the user's prompt and the recent diff. L5 spawns a second context but couples to the parent on attention. Together that's the soft channel — attention dynamics, by another name.

L6 (Governed) is where it profoundly changes. Hooks are not in the model's context window. A PreToolUse hook that blocks git push on a non-zero pytest exit doesn't get downweighted by a long task. An MCP server that requires authentication before reading a file doesn't depend on the model remembering your auth rule. Deny-permissions in .claude/settings.json for .env and .pem files don't compete with the rest of the spec. L6 is enforcement — outside the context dynamics, deterministic, not subject to load or context rot.

L7 (Adaptive) is different again. The agent writes its own instructions — not because the user said "remember this," but because the agent finished a task and decided some part of the trajectory was worth saving for next time. At read time the artifact lands in the attention channel like anything else. What's different is the writer: the model wrote the file, the trigger was task completion, and the user never saw the prompt that produced it. L7 is self-writing.

That's the ladder.

Two cuts under the ladder: attention channel covers L0–L5, enforcement is L6 alone, self-writing is L7 alone. Cut 1 at L5/L6 marked soft to hard. Cut 2 at L6/L7 marked read to write.

The first cut: L5 / L6

The load-bearing observation in this reframe is the cut between L5 and L6.

L0 through L5 all run on the soft channel — either directly on the parent's field (L0–L4) or on a child's that couples back to the parent through prompts and results (L5). They compete. They decay with load. The model can downweight any of them, lose track of any of them, prioritize the user's prompt over any of them. You can tell a sub-agent "always check tests before reporting done" and it'll do it 80% of the time, or 95%, or 60% — you don't know without measurement. The same instruction in a CLAUDE.md and the same instruction passed to a sub-agent are running on the same physics, just on different fields.

L6 is outside that physics entirely.

Generic example. Suppose your CLAUDE.md says "never push without running tests." That's L1. The model reads it, integrates it into context, weights it against everything else loaded — your other rules, the recent diff, the user's prompt. If you have four thousand tokens of instructions and the model is mid-task, that line is competing with everything else for attention. Sometimes it follows. Sometimes it doesn't.

Now suppose you have a PreToolUse hook on Bash that exits non-zero if pytest fails. That's L6. The model can decide to push or not push. It doesn't matter. The push fails before the model's intent reaches the network.

Same constraint, two channels, two failure modes. Soft channel fails probabilistically. Hard channel fails deterministically. They take different fixes — soft constraints want better content and ordering (the Pink Elephant piece is about that fight), hard constraints want a better hook script, a tighter PreToolUse matcher, or a stricter permission rule.

Calling these the same thing because they're both "in your .claude/ directory" hides the architectural difference.

The second cut: L7 writes itself

L7 - Adaptive isn't a third channel exactly — at read time, what L7 wrote lands in the same context with everything else. The cut is at write time. The agent writes its own instructions.

Most "memory" features in shipping agents aren't L7 by this cut. Claude Code's saved memory writes when the user signals remember this or accepts a prompt to save. Cursor's notepads, Copilot's pinned context, Gemini's saved facts — same pattern. The agent keeps the artifact, but the user authored it. That's persistent context, not self-writing. Call it L6.5 if you want a name for it.

The clearest L7 in print today is Hermes Agent, released by Nous Research. The mechanism is documented: when the agent identifies a saveable trajectory — after a successful task with five or more tool calls, after recovering from errors and finding a working path, after the user corrects its approach, or after discovering a non-trivial workflow — it invokes its skill_manage tool to extract a SKILL.md (markdown with YAML frontmatter) into ~/.hermes/skills/. Future sessions load the skill automatically and it becomes available as a slash command. The user didn't ask for it. The agent decided the trajectory was worth saving.

Three of the four triggers are what make this clearly L7 and not L6.5. Error recovery, user correction, novel-workflow discovery — these are cases where only the agent knows the saveable moment happened. A user-driven memory feature can capture "this task was useful enough to want it remembered" by asking the user after the fact. It can't capture "I tried three approaches and the third worked" unless the agent volunteers it. The artifact format matters too: an auto-extracted SKILL.md lands in ~/.hermes/skills/ in the same format human-written skills use. Next session, the agent loads it and can't tell who the author was. That symmetry is what makes the loop close — every successful trajectory can shape the next one.

Concretely, here's what an auto-extracted skill might look like — illustrative, in the shape Hermes's documented SKILL.md schema specifies, fitting the second trigger (the agent worked through a pytest debugging session, found the working path, and saved the lesson):

---
name: debug-pytest-import-errors
description: When pytest reports ModuleNotFoundError despite a successful editable install, check src-layout configuration before chasing PYTHONPATH.
version: 1.0.0
platforms: [macos, linux]
metadata:
  hermes:
    tags: [python, testing]
    category: dev-workflow
---
# Debug pytest ImportError on src-layout projects

## When to Use
pytest fails with `ModuleNotFoundError` after a fresh clone, even though `pip install -e .` ran and the import works in a Python REPL.

## Procedure
1. Check `pyproject.toml` for `where = ["src"]` under the build-system packages section.
2. Confirm `pythonpath = ["src"]` is set in `[tool.pytest.ini_options]`.
3. Re-run `pip install -e .`; confirm `.egg-info` lands at the package root, not inside `src/`.

## Pitfalls
- `PYTHONPATH=src` as an env var works locally but doesn't survive CI.

## Verification
`uv run pytest` runs without `ModuleNotFoundError`.

The frontmatter is functional — tags and category route the skill in Hermes's index; platforms gates it by OS. The body's When to Use / Procedure / Pitfalls / Verification is the schema's recommended shape. Notice what the agent saved: not the original failing command, not the dead-ends, just the working path plus the trap that would have lured a next session into chasing PYTHONPATH. That's curation, not transcription.

This is why L7 is safe to leave unsupervised in Hermes and risky most places else. The SKILL.md schema enforces moves a well-coupled instruction needs — imperative voice, directive ordering, named constructs, the warning placed after the working path rather than before it. A free-form memory feature has no such structural prior; the agent writes whatever feels worth saving, and the writes degrade as the agent's writing discipline does.

Schema is the cheap version of supervision.

The new failure mode is the self-writing layer running unsupervised. An auto-extracted skill that overfits to one project. A trajectory summary written under a stale assumption that surfaces six weeks later as a phantom instruction. There's no rule file the user authored to grep for the source — the rule is in a markdown file the agent wrote and the user never read, sitting in ~/.hermes/skills/ or its equivalent.

L7 doesn't replace L0–L6. It runs alongside, with its own writes and its own decay. Most agent setups don't have it because most agents don't expose it. The ones that ship a memory feature mostly do L6.5 and call it L7.

When to climb

The dominant pattern I see in real repos is L1 with a thin L6: a CLAUDE.md, maybe a few rule files at L3, deny-permissions for .env. L4 (skills) is rare — most authors haven't built any. L5 (sub-agents) is rarer — most use cases haven't surfaced. L7 is mostly absent — most agents don't expose a self-writing surface, and the few setups that do have one running treat it as opt-in defaults nobody reviewed.

Across 28,721 public repositories with AI configs, 89.9% don't name specific constructs in their instructions — no backticks, no file paths, no function names. That's most of the soft channel running at low coupling: easily downweighted, easily lost. The hard channel is thinner. The adaptive channel is mostly absent.

Large spec, small contract, no adaptive layer. That's the asymmetry — but it's not always a bug. Each rung exists because the rung below it fails in a specific way. The trigger is the failure, not a feature wishlist.

From	To	Symptom that triggers the climb
L0	L1	Re-explaining the same project context every session
L1	L2	One file got long enough that important rules get ignored
L2	L3	Path-irrelevant rules pollute every task
L3	L4	The same procedure gets described inline across multiple rules
L4	L5	A procedure pollutes the parent's context with reasoning chains the parent doesn't need
L5	L6	A constraint must hold 100% of the time, not 95%
L6	L7	You keep correcting the same preference across sessions
L6	L7	You keep watching the agent re-derive the same workaround

The mistake is climbing without the symptom. A repo with three rules in one file doesn't need L3. A solo developer's CLAUDE.md doesn't need a sub-agent. Premature climbs cost context budget for no return; you've added structure the model has to navigate without solving a problem you actually had.

The opposite mistake is more common: under-building the higher rungs because the symptoms feel like model failures rather than rung failures. "The agent didn't run tests before pushing" reads like a prompt-engineering problem; it's a missing L6. "The agent forgot we use Cloudflare Workers" reads like context drift; it's a missing L7. "The agent keeps describing the deploy process every time I ask" reads like verbosity; it's a missing L4.

Climb when the rung below stops working.

Three questions for your repo

Not a recipe. A diagnostic. For any rule in your setup, ask:

Does this fail loudly when violated, or silently? Loudly is L6. Silently is L0–L5.
Does the model see this, or does the runtime enforce it? Sees is the soft channel. Runtime is the hard channel.
Does it get worse when you add unrelated rules to the same file? Yes is L0–L5. No is L6. "Sometimes" is probably L7.

Most rules answer silently / sees / yes. That tells you which channel you're in. The interesting question is whether anything in your setup is on the other channels at all.

A note on related taxonomies

There are other progressive ladders for AI agent setups in print. Vellum's L0–L5 is an autonomy axis — how much the agent decides on its own. Blake Crosley's 4-tier is a concurrent-decomposition axis — how many agents run in parallel. Anthropic's 5-layer ADK frame for Claude Code is a content-boundary axis — what kind of content goes where. Zylon's 5-architectural and GitHub's 3-tier carve different cuts again, mostly around how the agent is wired into a product surface.

The ladder above is on a different axis from any of those. It sorts by the channel each mechanism runs on — soft attention, hard enforcement, self-writing memory — and progresses through the named constructs an agent exposes (CLAUDE.md, scoped rule files, skills, sub-agents, hooks, auto-memory). The two cuts (L5/L6 and L6/L7) are the load-bearing claim; the autonomy and concurrency taxonomies don't draw those cuts because they're sorting on different things.

Different axis, different cuts, different diagnostic. Use whichever maps onto the question you're actually asking.

Terminal output: ails check on a .claude directory. LADDER reads 8 rungs across 3 channels. SETUP reads L1 + L3 + L6 (Primer, Scoped, Governed). Channel/Levels/Count table shows attention L0–L5 count 5, enforcement L6 count 1, self-writing L7 count 1. Cuts named at L5/L6 soft→hard and L6/L7 read→write.

*Previously: CLAUDE.md Best Practices: From Basic to Adaptive — where I drew the ladder the first way. The State of AI Instruction Quality for additional data.

I'm building Reporails, measurement for the attention channel. npx @reporails/cli check runs locally, no account needed.*

The State of AI Instruction Quality

Gábor Mészáros — Tue, 21 Apr 2026 12:41:52 +0000

Everybody has opinions about AGENTS.md/CLAUDE.md files.

Best practices get shared. Templates get copied, and this folk-type knowledge dominates the industry. Last year, GitHub analyzed 2,500 repos and published best-practice advice. We wanted to go further: measure at scale, publish the data, and let anyone verify.

When the agent doesn't follow instructions and does something contradictory, the usual suspects are: the model is inconsistent, LLMs are not deterministic, you need better guardrails, you need retries.

The failures almost always get attributed to the model.

So we decided to measure. We built a diagnostic tool that treats instruction files as structured objects with measurable properties. Deterministic. Reproducible. No LLM-as-judge. Then we pointed it at GitHub repositories with instruction files for five agents - Claude, Codex, Copilot, Cursor, and Gemini.

28,721 repositories. 165,063 files. 3.3 million instructions.

... and one question:

What if the instructions are the problem?

The dataset

28,721 projects. Sourced from GitHub via API search, cloned, and deterministically analyzed. Each project was scanned for instruction files across five coding agents — then deduplicated to remove false positives from agent detection overlap.

Agent	Projects	% of corpus
Claude	12,356	43.0%
Codex	11,206	39.0%
Copilot	7,755	27.0%
Cursor	7,291	25.4%
Gemini	5,942	20.7%

The percentages add up to more than 100% because 37% of projects configure multiple agents. More on that later.

Key distributions stabilized early. A 9,582-repo sub-sample produced identical tier shares (±0.2pp) and the same mean scores as the full 12,076-repo intermediate sample. The final 28,721-repo corpus moved nothing. The patterns reported below are not small-sample artifacts.

All classifications are deterministic — the same file produces the same result every time. No LLM-as-judge. Sample classifications are published for inspection (methodology below). The tool is source-available.

How we measured

The analyzer parses each instruction file into atoms — the smallest semantically distinct units of content. A heading is one atom. A bullet point is one atom. A paragraph is one atom. Each atom gets classified along a few dimensions, all deterministic, no LLM involved:

Charge classification. A three-phase pipeline determines whether an atom is a directive ("use X"), a constraint ("do not use Y"), neutral content (context, explanation, structure), or ambiguous (could be read either way). Phase 1 detects negation and prohibition patterns. Phase 2 detects modal auxiliaries and direct commands. Phase 3 uses syntactic dependency parsing to catch imperatives that the first two phases missed. First definitive match wins. Atoms that partially match but don't clear any phase are marked ambiguous. Everything else is neutral.

Specificity. Binary: does the instruction name a specific construct — a tool, file, command, flag, function, or config key — or does it stay at the category level? "Use consistent formatting" is abstract. "Format with ruff format" is named. This is a text property, not a judgment call.

File categorization. Each file is classified as base config (your main CLAUDE.md or .cursorrules), a rule file, a skill definition, or a sub-agent definition — based on file path conventions for each agent.

Content type. Charge classification separates behavioral content (directives and constraints) from structural content (headings, context paragraphs, examples). That's how we know what fraction of your file is actually doing work.

The full tool is source-available (BUSL-1.1). You can run npx @reporails/cli check on your own project and inspect every finding. More on that at the end.

Finding 1: Most of your instruction file isn't instructions

Here's what the median instruction file actually contains:

50 content items total
12 of those are actual directives
The rest is headings, context paragraphs, examples, structure

Only 27% of your instruction file is doing what you think it does.

The other 73% is scaffolding. Headings that organize but don't instruct. Explanation paragraphs that compete for the model's attention without adding behavioral weight. Example blocks. Context-setting prose.

That's not inherently bad. Structure matters. But if you're writing a 200-line CLAUDE.md and only 54 lines are actual instructions, you should probably know that.

The average instruction is 8.9 words long. That's a sentence fragment.

Finding 2: 90% of instructions don't name what they're talking about

This is the big one.

We measured whether each instruction references specific tools, files, commands, or constructs by name — or whether it stays at the category level.

Two-thirds of all instructions are abstract.

Agent	Names specific constructs	Uses category language
Gemini	39.3%	60.7%
Codex	38.3%	61.7%
Copilot	33.3%	66.7%
Cursor	30.8%	69.2%
Claude	30.6%	69.4%

What does this look like in practice?

Abstract: "Use consistent code formatting"
Specific: "Format with ruff format before committing"

Abstract: "Avoid using mocks in tests"
Specific: "Do not use unittest.mock — use the real database via test_db fixture"

In previous controlled experiments, specificity produced a 10.9x odds ratio in compliance (N=1000, p<10⁻³⁰). The instruction that names the exact construct gets followed. The one that describes it abstractly... mostly doesn't. This is consistent with independent findings from RuleArena (Zhou et al., ACL 2025), where LLMs struggled systematically with complex rule-following tasks — even strong models fail when the rules themselves are ambiguous or underspecified.

89.9% of all agent configurations contain at least one instruction that doesn't name what it means. It's not a few projects. It's nearly everyone.

Finding 3: `agents.md` is the most common instruction file

Before we get into quality, let's look at what people are actually naming their files:

#	File	Count
1	`agents.md` / `AGENTS.md`	20,654
2	`claude.md` / `CLAUDE.md`	14,014
3	`gemini.md` / `GEMINI.md`	5,703
4	`.github/copilot-instructions.md`	5,647
5	`.cursorrules`	2,415

49,071 unique file paths across the corpus. That's not a typo. The format fragmentation is real.

A few things jumped out:

claude.md (lowercase, 10,642) is 3x more common than CLAUDE.md (3,372). Both work. The community clearly prefers lowercase.
agents.md dominates — the Codex/generic format is the single most popular instruction file name.
Skills and rules are already showing up in meaningful numbers: .claude/rules/testing.md (422), .agents/skills/tailwindcss-development/skill.md (334).

Finding 4: Different agents, completely different config philosophies

Not all agents are configured the same way. Not even close.

We categorized every file into four types: base config (your main CLAUDE.md, .cursorrules, etc.), rules (scoped rule files), skills (task-specific skill definitions), and sub-agents (role-based agent definitions).

Agent	Base	Rules	Skills	Sub-agents	Total files
Claude	18,733	4,638	10,692	10,538	44,601
Cursor	5,903	19,843	6,237	1,716	33,699
Copilot	16,026	4,486	10,352	3,012	33,876
Codex	19,001	81	8,911	165	28,158
Gemini	10,253	74	3,039	53	13,419

Cursor is 60% rules files. The .cursor/rules/ system dominates its configuration surface. One agent's config looks nothing like another's.

Claude is the only agent with a roughly balanced architecture across all four config types. Codex and Gemini are almost entirely base config — single-file setups.

The median Cursor project has 3 instruction files. The median Codex project has 1. These aren't just different tools. They're different configuration philosophies.

Finding 5: 37% of projects configure multiple agents

10,620 projects in the corpus target two or more agents. That's not a niche pattern — it's over a third of all projects.

Agents	Projects
1	18,101
2	6,776
3	2,687
4	949
5	208

The dominant pair is Claude + Codex (5,038 projects). Makes sense — CLAUDE.md + AGENTS.md is the most natural multi-agent starting point.

Here's what's interesting about multi-agent repos: the same developer, writing instructions at the same time, for the same project, produces measurably different instruction quality across agents. The person didn't change. The project didn't change. The instruction format did.

Some of that is structural. Cursor's .mdc rules enforce a different format than Claude's markdown. Codex's AGENTS.md invites a different writing style than Copilot's copilot-instructions.md. The format shapes the content.

Finding 6: The most-copied skills are the vaguest

This is where it gets interesting.

13,309 unique skills across the corpus. Some of them appear in hundreds of repos — clearly copied from shared templates or community sources. So we measured them.

Named% = what fraction of a skill's instructions name a specific tool, file, or command (instead of using category language).

Skill	Repos	Named%	What it means
`frontend-design`	271	2.8%	Almost entirely abstract advice
`web-design-guidelines`	197	10.2%	Generic design principles
`vercel-react-best-practices`	315	30.7%	Mix of specific and vague
`pest-testing`	216	55.1%	Names actual test constructs
`livewire-development`	87	75.5%	Names specific Livewire components
`next-best-practices`	76	92.6%	Names almost everything

frontend-design is in 271 repos with 2.8% specificity. It's a wall of "follow responsive design principles" and "ensure accessibility compliance." That reads well. It sounds professional. It gives the model almost nothing concrete to act on.

next-best-practices is in 76 repos with 92.6% specificity. It says things like "use next/image for all images" and "prefer server components over client." It reads like a checklist. It tells the model exactly what to do.

One is shared 3.5x more than the other.

The most popular skills are the most decorative. The well-written ones barely spread.

The best and worst skills (>50 repos)

Most specific:

Skill	Repos	Named%
`next-best-practices`	76	92.6%
`shadcn`	74	82.6%
`livewire-development`	87	75.5%
`pest-testing`	216	55.1%
`laravel-best-practices`	94	49.7%

Most vague:

Skill	Repos	Named%
`openspec-explore`	110	2.5%
`frontend-design`	271	2.8%
`web-design-guidelines`	197	10.2%
`vercel-composition-patterns`	131	10.7%
`find-skills`	113	18.9%

Notice a pattern? The Laravel/Livewire ecosystem produces specific skills. The generic frontend/design ones stay abstract. Domain-specific communities write better instructions than cross-cutting ones.

Finding 7: Sub-agents are almost entirely persona prompts

5,526 unique sub-agent roles in the corpus. Developers are building agent teams: code reviewers, architects, debuggers, testers, security auditors.

The problem? Sub-agents are the most abstract config type in the entire corpus. Only 17% of sub-agent instructions name specific constructs.

Role	Repos	Named%
`code-reviewer.md`	236	14.4%
`architect.md`	89	18.2%
`debugger.md`	66	9.4%
`security-auditor.md`	57	14.8%
`test-runner.md`	54	10.5%
`frontend-developer.md`	47	9.0%

Most of these are persona prompts. "You are a senior code reviewer. You care about code quality, security, and maintainability." That's a role description, not an instruction set. It tells the model who to be, not what to do.

Compare this to a base config that says "run uv run pytest tests/ -v before suggesting any commit" — that's 100% named, and the model knows exactly what action to take.

The anatomy chart: more directives, worse quality

Here's where it all comes together.

We measured three things for each config type: how big the files are, how many directives they contain, and what fraction of those directives actually name something specific.

Sub-agents have the largest files (61 items median), the most directives (17), and the worst specificity (17%). They're the wordiest config type in the corpus and the least effective.

Base configs are the opposite. Fewer directives (11), but 40% of them name specific constructs. The developer writing their own CLAUDE.md by hand, for their own project, produces the most actionable instructions.

Config type	Files	Median size	Median directives	Specificity
Base configs	69,916	50 items	11	39.8%
Rules files	29,122	34 items	9	31.2%
Skills	39,231	59 items	14	30.8%
Sub-agents	15,484	61 items	17	17.0%

The pattern is clear: what developers write by hand is the most specific. What gets templated and shared gets progressively vaguer. And what tries hardest to sound authoritative — sub-agent persona prompts — is the most hollow.

More instructions is not better instructions.

Independent research supports the structural angle: FlowBench (Xiao et al., 2024) found that presenting workflow knowledge in structured formats (flowcharts, numbered steps) improved LLM agent planning by 5-6 percentage points over prose — across GPT-4o, GPT-4-Turbo, and GPT-3.5-Turbo. Structure is not decoration. It changes what the model retrieves.

Limitations

Five things to know about these numbers.

Sampling bias. GitHub API search, public repos only, English-skewed. Enterprise configurations, private repos, and non-English projects are not represented. This is not a random sample of all instruction files in production.

Classification accuracy. The charge classifier is deterministic but not perfect. Edge cases exist: mixed-charge sentences, implicit constructs, domain jargon that looks like a category term but is actually a named tool. Specificity detection (named vs abstract) is simpler and more robust. Sample classifications are published for inspection.

Association, not causation. "More directives correlate with lower specificity" is an observed pattern. We do not claim that adding directives causes quality to drop.

Snapshot. Collected March–April 2026. Instruction practices are changing fast — agents.md didn't exist six months ago. These numbers describe the ecosystem at collection time.

No popularity weighting. A 10-star hobby project counts the same as a 50K-star production repo. The distribution of instruction quality in production agent work may differ.

What this means

This isn't an article about AI models being bad at following instructions. The models are fine.

This is an article about what we actually give them to work with.

Most instruction files are three-quarters scaffolding. Two-thirds of the actual instructions don't name what they're talking about. The most popular community skills are the most decorative. Sub-agent definitions are the wordiest files in the corpus and the least specific.

None of that is obvious from reading your own files. It wasn't obvious to us before we measured it. A well-structured CLAUDE.md feels thorough. A shared skill with 271 repos feels battle-tested. A sub-agent with 17 directives feels comprehensive.

Measurement shows something different.

In The Undiagnosed Input Problem, I argued that the industry is great at inspecting outputs and weak at inspecting inputs. This corpus analysis is the evidence for that claim.

The instruction files are there. The developers wrote them. They just have no way to know which parts are working and which parts are wallpaper.

Try it yourself

The analyzer we used for this corpus analysis is available as a CLI you can run against your own instruction files.

Reporails — instruction diagnostics for coding agents. Deterministic. No LLM-as-judge. 97 rules across structure, content, efficiency, maintenance, and governance.

npx @reporails/cli check

That scans your project, detects which agents are configured, and reports findings with specific line numbers and rule IDs. Here's what the output looks like:

Reporails — Diagnostics

  ┌─ Main (1)
  │ CLAUDE.md
  │   ⚠       Missing directory layout             CORE:C:0035
  │   ⚠ L9    7 of 7 instruction(s) lack reinfor…  CORE:C:0053
  │     ... and 16 more
  │
  └─ 21 findings

  Score: 7.9 / 10  ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓░░░░░░

  21 findings · 4 warnings · 1 info
  Compliance: HIGH

The corpus analysis used the same classification pipeline at scale. Fix the findings, run again, watch your score improve.

The dataset

The full corpus is published at reporails/30k-corpus. Three files:

File	Records	What it contains
`repos.jsonl`	28,721	Per-project record: agents configured, stars, language, license, topics
`stats_public.json`	1	Every aggregate statistic in this article
`validation_key.csv`	2,814	Sample classifications with source text for inspection

Verify any claim:

# "28,721 repositories"
cat repos.jsonl | wc -l

# "43% Claude"
cat repos.jsonl | python3 -c "
import sys, json
repos = [json.loads(l) for l in sys.stdin]
claude = sum(1 for r in repos if 'claude' in r['canonical_agents'])
print(f'{claude}/{len(repos)} = {claude/len(repos)*100:.1f}%')
"

Every number in every table traces to that dataset. If you disagree with a finding, count the rows.

This is part of the Instruction Quality series. Previous: The Undiagnosed Input Problem. Related: Precision Beats Clarity · Do Not Think of a Pink Elephant · 7 Formatting Rules for the Machine.

The Undiagnosed Input Problem

Gábor Mészáros — Wed, 08 Apr 2026 11:51:12 +0000

The AI agent ecosystem has built a serious industry around controlling outputs. Guardrails. Safety classifiers. Output validation. Monitoring. Retry systems. Human review.

All of that matters, but there is simpler upstream question that still goes mostly unmeasured:

Are the instructions any good?

That sounds obvious, yet it is not how the industry behaves.

When an agent fails to follow instructions, the usual explanations come fast:

Models are probabilistic
Agents are inconsistent
You need stronger guardrails
You need better monitoring
You need retries
You need humans in the loop

… and while those explanations are right to a certain degree, they also have a side effect: they turn instruction quality into a blind spot.

The ecosystem has become extremely good at inspecting what comes out of the model, and surprisingly weak at inspecting what goes in.

The symptom

Consider τ-bench.

It gives agents policy instructions and measures whether they follow them in realistic customer-service tasks. Airline and retail workflows. Real constraints. Real multi-step behavior.

The benchmark result that gets repeated is the model result: even strong systems still fail a large share of tasks, and consistency across repeated attempts remains weak.

The conclusion most people draw is straightforward: we need better models, better agents, better orchestration.

My take: Maybe.

But there is another question sitting underneath the benchmark:

Were the instructions themselves well-formed and well structured?

Not just present. Not just long enough. Not just sincere.

Well-formed. Well-structured. Well-organized.

Specific enough to anchor behavior. Structured enough to survive context mixing. Non-conflicting across files. Positioned where the model can actually use them.

Those questions usually never gets asked.

The industry response

I had a conversation recently where a lead solutions architect put the standard view plainly:

“The instruction merely influences the probability distribution over outputs. It doesn’t override it.”

That is right about the mechanism but it is wrong about what follows from it.

Yes, instructions operate probabilistically. But that does not mean all instructions are weak in the same way.

The shape of the distribution is not fixed. It changes with the properties of the instruction itself. Specificity sharpens it. Structure sharpens it. Conflict flattens it. Vague abstractions flatten it. Bad formatting can suppress it almost entirely.

Across my earlier controlled experiments, small changes in wording and placement produced large changes in compliance:

Instruction ordering moved compliance by 25 percentage points with the same model and the same directive.
Specificity produced roughly a 10x compliance effect when the instruction named the exact construct instead of describing it abstractly.
Formatting changed whether the model reliably registered the instruction at all.

The problem is that most instruction systems are built without diagnostics.

That is not an AI limitation. That is an engineering failure.

The folk system

Right now, instruction practice spreads mostly through imitation.

A popular repository posts “best practices” for Claude Code. Shared Cursor rules circulate as templates. People copy AGENTS.md files between projects. Teams accumulate CLAUDE.md, .cursorrules, copilot-instructions.md, etc and project-specific rule files across multiple tools.

Copy, paste, hope, repeat.

Some of that advice is useful. Almost none of it is tested in any controlled, reproducible way. That would be fine if instruction quality were self-evident. It is not.

A long instruction file can feel thorough while being internally contradictory. A highly opinionated ruleset can feel disciplined while producing almost no behavioral influence on the model.

A sprawling multi-file setup can look sophisticated while making the system worse.

Without diagnostics, developers do not know which instructions are binding, which are noise, and which are actively interfering with each other.

The gap

The tooling split is now pretty clear.

Output tooling is mature. Guardrails AI validates structure. Lakera focuses on prompt injection and security. NeMo Guardrails enforces safety and conversational rails. Llama Guard classifies risky content. The output edge is crowded.

Prompt testing is real. Promptfoo, Braintrust, and LangSmith can all help evaluate behavior. But they are primarily black-box systems: did the prompt produce the output you wanted?

That is useful.

It is not the same as measuring the instruction artifact itself.

Instruction-quality tooling exists only in fragments. Some tools use LLM-as-judge. Some use deterministic local rules. But the category is still early, inconsistent, and mostly disconnected from measured behavioral outcomes.

What is still largely missing is a deterministic way to inspect instruction files as engineered objects:

how specific they are
how directly they state intent
whether they conflict across files
whether they overuse headings
whether they provide alternatives instead of bare prohibitions
whether the system is getting denser while getting weaker

Code gets static analysis.

Instruction systems usually get vibes.

What we measured

We built an analyzer that treats instruction files as structured objects with measurable properties. Deterministic. Reproducible. No LLM-as-judge.

I am running it across a large live corpus of real repositories. The full run completes this week; what follows is what the partial sample already shows - stable enough to publish, not yet the full picture.

Quality is reported on a 0-to-100 scale: 0 means the file produces no measurable influence on model behavior, 100 is the ceiling the framework can score.

A fresh aggregation over 12,076 completed instruction-file scans is virtually identical to an earlier 9,582-repo sample:

bottom tier: 40.3% vs 40.1%
top tier: 12.1% vs 12.2%
mean quality score: 27 vs 27
directive content ratio: 27.9% vs 27.9% - the share of instruction sentences that directly tell the model what to do

That matters because it means the pattern is stable.

This does not look like a small-sample artifact.

And the strongest finding is not what I expected.

More rules, lower quality

The common response to bad agent behavior is to add more rules.

More files. More guidance. More scoping. More edge-case coverage.

The corpus says that strategy tends to backfire.

Across 12,076 repositories, instruction quality falls as instruction-file count rises:

Files per repo     N      Mean score   Bottom tier %   Top tier %
1                  4681   28           46.3%           16.9%
2-5                4796   26           37.3%            9.5%
6-20               1972   26           36.0%            8.8%
21-50               438   25           31.3%            5.7%
51-500              186   25           33.3%            5.4%

The key number is the top-tier share.

It collapses from 16.9% in single-file setups to 5.4% in repositories with 51 to 500 instruction files.

That is a roughly 3x drop.

The article version of that finding is simple:

Developers respond to bad agent behavior by adding more rules. In the corpus, that strategy correlates with a 3x collapse in the probability of landing in the top tier.

That does not prove file count causes low quality by itself.

But it does show that rule proliferation is not rescuing these systems. At scale, it is associated with weaker instruction quality, not stronger.

The sweet spot

There is also a more subtle result in the partial sample. Instruction quality appears to be non-monotonic in directive density: more directives help at first, then stop helping, and past a point start to hurt.

The full curve is in next week’s piece. The short version is that there is an optimal density range, after which additional directives stop strengthening the system.

Enough force to bind behavior. Not so much that the system turns into an overpacked rules document.

A real example

Here is the kind of instruction block the corpus is full of:

# Code should be clear, well documented, clear PHPDocs.

# Code must meet SOLID DRY KISS principles.

# Should be compatible with PSR standards when it need.

# Take care about performance

It is not malicious. It is not absurd.

It is just weak.

Everything is abstract. Nothing is anchored. Headings are doing the work prose should do. The agent can read it, represent it, and still walk past most of it.

Now compare:

Never use `var_dump()` or `dd()` in committed code. Use `Log::debug()` instead.
Run `./vendor/bin/phpstan analyse src/` before every commit. Level 6 minimum.

Same general intent. Completely different binding strength.

The second version names the construct, names the alternative, names the command, and names the threshold. It gives the model something concrete to hold onto.

That is what diagnostics should make visible.

What this means

Output guardrails still matter.

Prompt evaluation still matters.

Safety systems still matter.

But they do not answer the upstream question: Are the instructions themselves well-formed?

If the answer is no, then a large class of downstream failures will keep showing up as mysterious agent unreliability when the real problem is earlier and simpler.

The agent loaded the instruction and walked past it.

That is often not a model problem.

It is an input problem.

And input quality is measurable.

What’s next

These are corpus-level findings from a partial sample, not universal laws.

The sample is still in flight. The strongest claims here are about association, not proof of causality. Specific conflict-count case studies need source verification before publication. Popularity weighting is not yet applied, so “40% of repositories score in the bottom tier” is not the same claim as “40% of production agent work scores in the bottom tier.”

The full corpus run completes this week. Next week I publish the end-of-run analysis across the full sample — the complete distribution, the cross-cuts the partial sample cannot yet support, and the specific case studies this article deliberately held back. If you want to know where your stack lands, that is the piece to come back for.

For now, the central pattern is already stable enough to matter:

The ecosystem keeps responding to weak agent behavior by adding more instructions, while the corpus shows that more instruction files are usually associated with lower measured quality.

That is the undiagnosed input problem.

Not that instructions do not matter.
That they matter, measurably, and most teams still have no way to see whether theirs are helping or hurting.

This is part of the Instruction Best Practices series. Previous: Do NOT Think of a Pink Elephant, Precision Beats Clarity, 7 Formatting Rules for the Machine. I’m building instruction diagnostics for coding agents. Follow for the full corpus analysis.

Do NOT Think of a Pink Elephant

Gábor Mészáros — Tue, 31 Mar 2026 12:19:14 +0000

You thought of a pink elephant, didn't you?

Same goes for LLMs too.

"Do not use mocks in tests."

Clear, direct, unambiguous instruction. The agent read it — I can see it in the trace. Then it wrote a test file with unittest.mock on line 3. Thanks...

I've seen this play out hundreds of times. A developer writes a rule, the agent loads it, and it does exactly what the rule said not to do. The natural conclusion: instructions are unreliable. The agent is probabilistic. You can't trust it.

That's wrong. The instruction was the problem.

The pink elephant

There's a well-known effect in psychology called ironic process theory (Daniel Wegner, 1987). Tell someone "don't think of a pink elephant," and they immediately think of a pink elephant. The act of suppressing a thought requires activating it first.

Something structurally similar happens with AI instructions.

"Do not use mocks in tests" introduces the concept of mocking into the context. The tokens mock, tests, use — these are exactly the tokens the model would produce when writing test code with mocks. You've put the thing you're banning right in the generation path.

This doesn't mean restrictive instructions are useless. It means a bare restriction is incomplete.

The anatomy of a complete instruction

The instructions that work — reliably, across thousands of runs — have three components. But the order you write them in matters as much as whether they're there at all.

Here's how most people write it:

# Human-natural ordering — constraint first
Do not use unittest.mock in tests.
Use real service clients from tests/fixtures/.
Mocked tests passed CI last quarter while the production
integration was broken — real clients catch this.

All three components are present. Restriction, directive, context. But the restriction fires first — the model activates {mock, unittest, tests} before it ever sees the alternative. You've front-loaded the pink elephant.

Now flip it:

# Golden ordering — directive first
Use real service clients from tests/fixtures/.
Real integration tests catch deployment failures and configuration
errors that would otherwise reach production undetected.
Do not use unittest.mock.

Same three components. Different order. The directive establishes the desired pattern first. The reasoning reinforces it. The restriction fires last, when the positive frame is already dominant.

In my experiments — 500 runs per condition, same model, same context — constraint-first produces violations 31% of the time. Directive-first with positive reasoning: 7%.

The pink elephant isn't just about missing components. It's about which concept the model sees first.

Three layers, in this order:

Directive — what to do. This goes first. It establishes the pattern you want in the generation path before the prohibited concept appears.
Context — why. Reasoning that reinforces the directive without mentioning the prohibited concept. "Real integration tests catch deployment failures" adds mass to the positive pattern. Reasoning that mentions the prohibited concept doubles the violation rate.
Restriction — what not to do. This goes last. Negation provides weak suppression — but weak suppression is enough when the positive pattern is already dominant.

The part nobody expects

Here's what surprised me: the ordering effect is larger than any other variable I've measured.

Precise naming vs. vague categories? 28 percentage points. Exact scope vs. broad scope? 74 points across the range. But reordering — same words, same components, just flipped — accounts for 25 points on its own. And it compounds with everything else.

Most developers write instructions the way they'd write them for a human: state the problem, then the solution. "Don't do X. Instead, do Y." It's natural. It's also the worst ordering for an LLM.

Never write "Don't use X. Instead, use Y." Write "Use Y. Here's why Y works. Don't use X."

Formatting helps too — structure is not decoration. I covered that in depth in 7 Formatting Rules for the Machine. But formatting on top of bad ordering is polishing the wrong end. Get the order right first.

What this looks like in practice

Here's a real instruction I see in the wild:

When writing tests, avoid mocking external services. Try to
use real implementations where possible. This helps catch
integration issues early. If you must mock, keep mocks minimal
and focused.

Count the problems:

"Avoid" — hedged, not direct
"external services" — category, not construct
"Try to" — escape hatch built into the instruction
"where possible" — another escape hatch
"If you must mock" — reintroduces mocking as an option within the instruction that prohibits it
Constraint-first ordering — the prohibition leads, the alternative follows
No structural separation — restriction, directive, hedge, and escape hatch all in one paragraph

Now rewrite it:

**Use the service clients** in `tests/fixtures/stripe.py` and
`tests/fixtures/redis.py`.

> Real service clients caught a breaking Stripe API change
> that went undetected for 3 weeks in payments - integration
> tests against live endpoints surface these immediately.

*Do not import* `unittest.mock` or `pytest.monkeypatch`.

Directive first — names the exact files. Context second — the specific incident, reinforcing why the directive matters without mentioning the prohibited concept. Restriction last — names the exact imports, fires after the positive pattern is established. No hedging. No escape hatches.

Try it

For any instruction in your AGENTS.md/CLAUDE.md or SKILLS.md files:

Start with the directive. Name the file, the path, the pattern. Use backticks. If there's no alternative to lead with, you're writing a pink elephant.
Add the context. One sentence. The specific incident or the specific reason the directive works. Do not mention the thing you're about to prohibit — reasoning that references the prohibited concept halves the benefit.
End with the restriction. Name the construct — the import, the class, the function. Bold it. No "try to avoid" or "where possible."
Format each component distinctly. The directive, context, and restriction should be visually and structurally separate. Don't merge them into one paragraph.

If your instruction is just "don't do X" — you've told the model to think about X.

Tell it what to think about instead. And tell it first.

DEV Community: Gábor Mészáros

Loop Engineering: How to Stop Your Agent Reward-Hacking Its Own Checks

What the steer is

The steer gets none of the attention

The good steer holds the goal

Watch a loop game its own check

The check certifies whatever the steer pointed it at

The same move, away from the tests

Make the steer a reduction, and keep the grader out of reach

What reporails can and cannot see here

The loop still has an arm to take apart

Loop Engineering: How To Stop The "You're Absolutely Right" Sycophancy

What the surface is here

What loaded costs, part one: tokens

What loaded costs, part two: attention

The rules you can pin to a path

Build it: split the file that grew

The rule with no path

Which rule loads where

The loop only checks what you wrote down

Loop Engineering: Fine-Tuning the Guardrail That Fired Wrong

What a check is here

Two readings of the same quiet

The misfire, up close

The tempting fix, and what it costs

Fix the instrument instead

The rule that generalizes

See how AI instructions decay, then write ones that hold

What I Built

Demo

Code

How I Built It

Deterministic Guardrails: Prompts Steer, Hooks Enforce

The loop, and where it can only ask

Two channels: one asks, one refuses

Build the missing arm: a hook that refuses

Which rule goes in which channel

The generator's "done" is a claim to re-derive

What holds, and what it still costs to keep loaded

Prompt Engineering, Context Engineering, Loop Engineering: What Actually Changed

The unit was a prompt

The unit became the surface

The unit is the loop

A verifier that checks what, exactly?

What the earlier articles measured

What the new name solves

The loop, one part at a time

Wiring a "read this first" hook as "ask" (don't)

Green Tests Don't Mean Better Software

What green actually checks

Two disciplines, pitched as rivals

The orthogonality move

A worked example: green architecture tests

The expectation primitive

Proof of existence, and the cost of skipping it

confirmed is the minority verdict

inconclusive is the common case

Even confirmed is not permanent

Instruction systems capability ladder: harness leveling

The new ladder

A quick tour

The first cut: L5 / L6

The second cut: L7 writes itself

When to climb

Three questions for your repo

A note on related taxonomies

The State of AI Instruction Quality

The dataset

How we measured

Finding 1: Most of your instruction file isn't instructions

Finding 2: 90% of instructions don't name what they're talking about

Finding 3: agents.md is the most common instruction file

Finding 4: Different agents, completely different config philosophies

Finding 5: 37% of projects configure multiple agents

Finding 6: The most-copied skills are the vaguest

The best and worst skills (>50 repos)

Finding 7: Sub-agents are almost entirely persona prompts

The anatomy chart: more directives, worse quality

Limitations

What this means

`confirmed` is the minority verdict

`inconclusive` is the common case

Even `confirmed` is not permanent

Finding 3: `agents.md` is the most common instruction file