DEV Community: Finley Li

Mutate the Grader, Not the Code: A Weekly Self-Audit for AI C++ Patch Evals

Finley Li — Mon, 09 Nov 2026 20:44:36 +0000

Example scenario (illustrative, not a reported incident): a team's eval harness has returned PASS on every AI-generated C++ patch for three weeks. Then a patch replaces a bounded comparison with a modulo on a signed value. The unit grader stays green because no golden case feeds a negative operand. The sanitizer grader stays green because it runs the same test binary, which never reaches the branch. Nothing in the report is lying — the harness simply stopped being able to see that class of bug.

Green means "the cases we wrote still pass". It does not mean "the grader can still detect this failure". Those are different properties, and only the second one decays silently.

The technique below audits the second property directly: seeded faulty patches, known as mutants, are pushed through the real harness, and every grader that fails to reject them is reported as blind. No model names, quotas, or benchmark numbers are assumed; the artifact is a plain script and a manifest.

1. Three ways a grader goes blind

Exit-code-only grading. The harness records returncode != 0 as a failure and nothing else. A patch that compiles, runs, and prints a wrong number passes. So does one that emits a new warning nobody promoted to an error.

Baseline-normalized output. Warnings and diagnostics are diffed against a stored baseline file. When the baseline is regenerated after a noisy commit, the diff quietly absorbs the next real diagnostic.

Assertions the patch can edit. The grader greps test output for a string, and the test file lives in the same tree the model may modify. The patch adjusts the expected string; the grader confirms the adjustment.

Each of these is a grader weakness, not a case weakness. Adding golden cases will not fix any of them, which is why pruning and shuffling the case set does not surface the problem either.

2. The artifact: a mutant manifest plus an audit loop

The audit needs three inputs: a clean git worktree, a list of small patches with known faults, and a harness entry point that can report per-grader status as machine-readable output. A manifest declares, for each mutant, which graders must reject it.

# mutants/manifest.yaml (illustrative)
mutants:
  - id: signed-modulo-clamp
    diff: mutants/signed_modulo_clamp.diff
    why: replaces a bounded comparison with a modulo
    must_fail: [unit_clamp, prop_clamp_bounds]
  - id: unchecked-alloc-size
    diff: mutants/unchecked_alloc_size.diff
    why: drops a size guard before memcpy
    must_fail: [asan_guard]
  - id: silent-float-narrowing
    diff: mutants/silent_float_narrowing.diff
    why: narrows double to float without a cast warning gate
    must_fail: [warnings_as_errors]

The must_fail list is the contract. A mutant that survives a listed grader is a finding; a mutant that fails extra graders is fine and often informative.

3. Step-by-step: wiring the loop

Step 1 — Give the audit an isolated worktree

Dirty state is the most common cause of false "invalid diff" verdicts. A dedicated worktree keeps the audit off the developer's working tree.

git worktree add ../eval-audit HEAD
python tools/mutant_audit.py \
  --manifest mutants/manifest.yaml \
  --worktree ../eval-audit \
  --report audit-$(date -u +%Y%m%d).md

Step 2 — Apply, grade, reset

The runner below is illustrative and has not been executed as part of this article. It expects python -m evals.grade --json to print {"graders": {"<id>": {"status": "pass"|"fail"}}} on stdout.

#!/usr/bin/env python3
"""Illustrative grader self-audit. Not executed for this article."""
import json, pathlib, subprocess, sys
import yaml

ROOT = pathlib.Path(__file__).resolve().parent.parent

def sh(cmd, cwd):
    return subprocess.run(cmd, cwd=cwd, capture_output=True, text=True)

def reset(worktree):
    sh(["git", "checkout", "--", "."], worktree)
    sh(["git", "clean", "-fdq"], worktree)

def apply(worktree, diff_path):
    diff_path = str(diff_path)
    if sh(["git", "apply", "--check", diff_path], worktree).returncode != 0:
        return False
    return sh(["git", "apply", diff_path], worktree).returncode == 0

def grade(worktree):
    """Returns (graders_dict, error). Never treats a crash as a grader failure."""
    p = sh([sys.executable, "-m", "evals.grade", "--json"], worktree)
    try:
        return json.loads(p.stdout)["graders"], None
    except (json.JSONDecodeError, KeyError):
        return None, f"no JSON from harness (exit {p.returncode})"

def audit(manifest_path, worktree):
    manifest = yaml.safe_load(pathlib.Path(manifest_path).read_text())
    rows = []
    for m in manifest["mutants"]:
        reset(worktree)
        if not apply(worktree, ROOT / m["diff"]):
            rows.append({"id": m["id"], "verdict": "invalid-diff", "survived": []})
            continue
        graders, err = grade(worktree)
        if graders is None:
            rows.append({"id": m["id"], "verdict": "infra-error", "survived": [], "detail": err})
            continue
        survived = [g for g in m["must_fail"]
                    if graders.get(g, {}).get("status") != "fail"]
        rows.append({
            "id": m["id"],
            "verdict": "caught" if not survived else "blind",
            "survived": survived,
        })
    reset(worktree)
    return rows

Three verdicts matter, and they are not interchangeable. caught is a healthy grader. blind is the signal the audit exists for. infra-error means the harness itself broke — it must never be counted as a pass, otherwise a crash in the grader loop reads as perfect detection.

Step 4 — Report kill rate per grader, not per run

A single global number hides the useful part. Aggregate by grader id so a weakness localizes:

from collections import defaultdict

def kill_rates(rows):
    seen, killed = defaultdict(int), defaultdict(int)
    for r in rows:
        if r["verdict"] not in ("caught", "blind"):
            continue
        for g in r.get("must_fail", []):
            seen[g] += 1
            if g not in r["survived"]:
                killed[g] += 1
    return {g: killed[g] / seen[g] for g in seen}

A grader below roughly its own historical rate is worth a look, and a grader at zero is definitionally blind. The thresholds are a team convention; they are not a standard, and picking one without a baseline of stable weeks is guesswork.

Step 5 — Triage survivors into three buckets

Not every survivor is a bug, and treating them all as one leads teams to abandon the audit.

Weak grader. The mutant is a real fault and the grader missed it. Fix the grader or add the missing assertion. This is the case worth real engineering time.
Equivalent mutant. The patch compiles and looks wrong but is behaviorally identical on every reachable path. Retire it from the manifest with a note explaining why.
Missing case. The grader is sound but no golden case reaches the affected code. This is a case-set gap, so it overlaps with coverage work rather than duplicating it.

Record the bucket in the report. A manifest that grows without retiring equivalent mutants becomes noise within a few months.

4. Generating candidate mutants without hand-writing all of them

Hand-authored mutants are high quality and slow to produce. A model can draft candidates from the function under test plus the current grader list, and a human then decides which are real faults versus equivalents. The cheap part of that loop is generation, and the expensive part is review — which is exactly where the human should stay.

This is also the only place MonkeyCode participates in the workflow described here. Its free model access covers the candidate-generation step, and the free server option is a place to run the nightly audit so it does not consume CI minutes or a developer's laptop. The project's outreach materials describe both options; specific quota figures and server hardware change, so read the current numbers on the official page rather than planning capacity from any blog post, this one included.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A generation prompt that respects the constraint the audit needs — compiles cleanly, changes behavior — reads roughly like this:

Given the function below and these existing graders [list], propose 5 patches that
(a) compile without new errors, (b) change observable behavior on at least one
reachable input, (c) plausibly evade at least one listed grader. For each, state
the input class that exposes the fault and which grader should catch it.

Two rules keep model-generated mutants honest. Every candidate is compiled before it enters the manifest, since a patch that fails to build is not a mutant. And every candidate is reviewed for equivalence by a human, since a model that believes two behaviors differ is not evidence that they do.

5. Verify the audit itself

An audit that always prints caught is worse than no audit. Plant a known failure before trusting the report: replace one grader's body with return True in a scratch branch, run the audit, and confirm that grader's kill rate drops to zero and the run exits non-zero.

# in the audit entry point
blind = [r["id"] for r in rows if r["verdict"] == "blind"]
errors = [r["id"] for r in rows if r["verdict"] == "infra-error"]
if blind or errors:
    print_report(rows, kill_rates(rows))
    sys.exit(1)

If the planted failure is not reported, the problem is the audit, not the graders. This self-check is the difference between a dashboard and a measurement.

6. Limits, and who should skip this

The audit assumes deterministic graders and a harness that exposes per-grader status. Flaky timing or network checks make survivors ambiguous, and the first fix should be determinism, not more mutants.

It also assumes someone owns the manifest. A stale mutant set stops finding things while continuing to print green — the same failure mode it was built to detect.

Skip this if golden cases barely exist yet, if the harness only reports pass/fail with no per-grader granularity, or if a single engineer owns both the harness and every patch under test. The cost lands on the harness owner, and the benefit only appears when grader coverage and patch volume are both large enough for silent blindness to be plausible.

Closing

Graders rot quietly. A short weekly run that shoves known-bad patches through the real harness converts that decay into a number someone can act on. Teams already running a patch eval loop can start with three mutants and one grader contract; that is enough to learn whether the report is describing the model's patches or the harness's blind spots.

Coverage-Coupled Grading: Flag AI C++ Patch Lines That No Golden Case Ever Ran

Finley Li — Sun, 08 Nov 2026 19:27:20 +0000

A patch landed in our C++ service last quarter with a green board: 412 golden cases passing, no flakes, no sanitizer noise. Three days later a nightly batch job wrote garbage into a column that had never been read by a test. The changed line was real, the tests were real, and they simply never met.

That failure mode is the one I now design against: every golden case passes because no case ever executes the line the model touched. Coverage is not correctness. But coverage that is joined to the diff is a cheap, honest alarm — and it catches a class of silent regression that pass/fail grading cannot see.

This article is the harness I use for that join. It assumes you already grade AI-written C++ patches with golden cases and want one more gate before a patch is accepted.

1. The problem statement, stated precisely

A golden case gives you a boolean. A patch gives you a set of changed lines. The dangerous region is the intersection of "changed" with "never executed by any case" — unverified behaviour that still reports success.

The gate is therefore not "coverage must go up." It is narrower and much harder to game:

For every line an AI patch adds or modifies, at least one golden case must execute that line — or a human must sign off on the gap by name.

Two properties matter. It is per-line, so a large refactor cannot be waved through by an unrelated coverage increase. And it is per-case, so you learn which case owns the line, which is the first question anyone asks during review.

2. Build an instrumented binary once

Clang source-based coverage is the least painful option for per-case attribution. Build a second binary tree; never grade against the release build (optimisation and inlining move the line map underneath you).

cmake -S . -B build-cov \
  -DCMAKE_BUILD_TYPE=Debug \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_CXX_FLAGS="-fprofile-instr-generate -fcoverage-mapping -O0 -g" \
  -DCMAKE_EXE_LINKER_FLAGS="-fprofile-instr-generate"
cmake --build build-cov -j"$(nproc)"

One binary, one build, reused for every case. Rebuilding per case is where people give up on this idea.

3. Run each golden case in isolation

Per-case attribution needs one profile per case. The %p in the filename keeps forked children from overwriting each other.

rm -rf prof && mkdir -p prof
while read -r case_id; do
  LLVM_PROFILE_FILE="prof/case-${case_id}-pid%p.profraw" \
    ./build-cov/tests --case "$case_id" || echo "FAIL $case_id" >&2
done < cases.txt

Keep cases.txt under version control — it is part of the grader, not a scratch file. The harness lives as scripts, not prose checklists, because prose drifts and scripts get diffed.

4. Join coverage to the diff

The script below is the whole gate. It parses git diff -U0 for changed line numbers in the new file, exports per-case executed lines from llvm-cov, and emits three buckets: covered, uncovered, unmapped.

#!/usr/bin/env python3
# covgate.py — flag changed lines that no golden case executed.
# Usage: python3 covgate.py <repo-root> <instrumented-test-binary>
import json, re, subprocess, sys, pathlib, collections

ROOT = pathlib.Path(sys.argv[1])
BIN  = sys.argv[2]                      # e.g. build-cov/tests
PROF = ROOT / "prof"
BASE = "HEAD"                           # base rev of the model's patch
exec_sh = lambda cmd: subprocess.run(
    cmd, cwd=ROOT, text=True, capture_output=True, check=True).stdout

RAW = re.compile(r"^case-(.+)-pid\d+\.profraw$")

def per_case_profiles():
    cases = collections.defaultdict(list)
    for raw in sorted(PROF.glob("case-*-pid*.profraw")):
        m = RAW.match(raw.name)
        if m:
            cases[m.group(1)].append(raw)
    return cases

def executed_lines(raws, tag):
    merged = PROF / f"merged-{tag}.profdata"
    exec_sh(["llvm-profdata", "merge", "-sparse",
             *[str(r) for r in raws], "-o", str(merged)])
    doc = json.loads(exec_sh(["llvm-cov", "export", BIN,
                              "-instr-profile", str(merged)]))
    lines = collections.defaultdict(set)
    for unit in doc["data"]:
        for f in unit["files"]:
            for seg in f["segments"]:
                line, _col, count, has_count = seg[0], seg[1], seg[2], seg[3]
                if has_count and count > 0:
                    lines[f["filename"]].add(line)
    return lines

HUNK = re.compile(r"^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@")

def changed_lines():
    diff = exec_sh(["git", "diff", "-U0", "--no-color", BASE,
                    "--", "*.cpp", "*.cc", "*.h", "*.hpp"])
    out, cur = collections.defaultdict(set), None
    for ln in diff.splitlines():
        if ln.startswith("+++ b/"):
            cur = ln[6:]
        m = HUNK.match(ln)
        if m and cur:
            start, n = int(m.group(1)), int(m.group(2) or 1)
            out[cur].update(range(start, start + n))
    return out

norm = lambda p: p.replace("\\", "/")

def main():
    cases = per_case_profiles()
    if not cases:
        sys.exit("no .profraw files — run step 3 first")

    union = collections.defaultdict(set)
    for cid, raws in cases.items():
        for f, ls in executed_lines(raws, cid).items():
            union[norm(f)] |= ls

    report = {"covered": [], "uncovered": [], "unmapped": []}
    for rel, lines in changed_lines().items():
        keys = [k for k in union if k.endswith("/" + norm(rel))]
        if not keys:
            report["unmapped"].append({"file": rel, "lines": sorted(lines)})
            continue
        hit = set().union(*(union[k] for k in keys))
        for line in sorted(lines):
            bucket = "covered" if line in hit else "uncovered"
            report[bucket].append({"file": rel, "line": line})

    json.dump(report, sys.stdout, indent=2)
    sys.exit(1 if report["uncovered"] else 0)

if __name__ == "__main__":
    main()

Exit code 1 means "at least one changed line is unexecuted." That single bit is what you wire into the grader.

5. Read the three buckets with different eyes

Not every bucket means the same thing, and treating them uniformly is how this gate gets switched off in week two.

Bucket	Meaning	Default action
`covered`	Some case executed the line	Accept; log which case for review context
`uncovered`	Changed, executable, never run	Reject, or require a named human waiver
`unmapped`	Changed file absent from coverage data	Investigate: header-only code, uninstrumented target, or a file the binary never links

The unmapped bucket is the one people skip and later regret. A header that is compiled into a different translation unit than the test binary's will look unmapped forever; if you never look, the gate silently has a hole exactly where template-heavy code lives.

6. Where free model access and a free server actually fit

The gate above only works if you run it on many candidate patches, because a gate that blocks one patch a week is theatre. Fan-out is the cost driver: k candidates per failing case, each needing a build, a case sweep, and a coverage export.

I use MonkeyCode's free model access for the wide, disposable part of that fan-out — the candidates I expect to reject — and reserve paid capacity for the narrow set that survives. The operator states a free allowance of roughly 10 million tokens and a free server option as of this writing; confirm the current terms on the project's own page before you build a budget around them, since allowances change.

The free server matters for a less obvious reason: coverage exports are I/O-heavy and bursty, and they do not belong on the same box as latency-sensitive CI. A separate small host running the sweep on a schedule keeps the main pipeline honest. The operator describes MonkeyCode as an open-source coding-agent project; check the repository for the licence and current feature set rather than trusting a blog post, including this one.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

7. Limitations, stated before someone finds them for me

Coverage is not assertion. A line executed and never checked is still unverified. This gate catches "never ran," not "ran and proved nothing."
Debug builds lie about hot paths. -O0 changes inlining and template instantiation; a line covered in build-cov may not exist in the release binary. Keep a separate release-split check rather than merging the two concerns.
segments is an approximation. Region-based line attribution can mark a line executed when only part of its expression ran. Treat the output as a strong hint plus an audit trail, not a proof.
Diff base is a policy choice. HEAD works when the patch is uncommitted; otherwise pin the merge base explicitly and record it in the report.
Header instrumentation explodes build time. On a heavy template codebase expect the instrumented build to dominate everything else in this workflow.

Do not adopt this if you have no isolated per-case runner, no instrumented build target, or a test suite where most cases share one giant fixture — you will get uncovered noise on lines that are genuinely exercised, and the gate will lose its credibility in a week.

8. What to do next

The order matters more than the tooling. First get per-case profiles, then get the diff join, then add the gate to the grader only after you have hand-checked twenty reports against reality. A gate nobody has audited is just another number that goes green.

If you want to try the fan-out half cheaply, run this harness against a handful of candidate patches from MonkeyCode's free tier on a spare host, and compare the uncovered list to what a human reviewer actually flagged. That comparison — not the pass rate — is the number worth tracking.

Green Tests, Slow Code: Budget the Clock in Your AI C++ Patch Eval Harness

Finley Li — Sat, 07 Nov 2026 18:09:12 +0000

An eval suite can be green and still ship a regression. The patch returns the right values, compiles clean, and clears every golden case — it just does the work about three times slower than the code it replaced. The numbers in this section are synthetic, chosen so the failure mode is easy to see; the gate further down is the real artifact.

The cause is boringly common. A std::map swapped in where a sorted std::vector was enough. A defensive copy added inside a hot callback. A std::string rebuilt per element because the AI model completed a loop that reads better than it runs. Every assertion stays true. Every reviewer nods.

Behavioral golden cases answer one question: does the patch produce the specified output? They say nothing about what that output cost. That is the third silent-regression class in an eval harness, after state leaks that ordering hides and grader drift that rubric canaries catch. This post adds a budget for it.

Why a slower patch survives review

A patch that is correct but slower passes every gate that only inspects results. Code review catches some of it, but review attention is finite and the interesting cases are subtle: a container change buried in a 400-line diff, an extra allocation behind an if that is only true on the hot path.

Benchmarks do not catch it either, in most repos, because they are run manually, weekly, or never. The regression lands between two benchmark runs, and by the time someone profiles, the commit that caused it is three sprints back in the log.

The fix is not a better benchmark. It is a budget that runs on every candidate patch and fails the patch, not the build. Treat wall-clock cost as another graded property, in the same loop that already grades correctness.

The artifact: a ratio budget gate

Two design decisions make this workable on ordinary hardware, including shared servers.

First, budgets are ratios against a baseline build, not absolute milliseconds. A patch that is 15% slower than the code it replaced is a regression regardless of whether the machine is fast or loaded.

Second, baseline and candidate runs are interleaved, not batched. Thermal drift, noisy neighbors, and page-cache state then affect both builds roughly equally.

Step 1 — Express budgets as ratios, not milliseconds

Keep the budgets in a versioned JSON file next to the golden cases. Absolute ceilings stay as a backstop; they catch a patch that is slow in absolute terms even when the baseline was slow too.

{
  "noise_floor_ms": 20,
  "default_k": 5,
  "cases": [
    { "id": "ring_push_pop", "arg": 2000000, "baseline_ratio_max": 1.15, "absolute_ms_max": 400 },
    { "id": "index_build",  "arg": 500000,  "baseline_ratio_max": 1.25 }
  ]
}

A 15% band on a microbenchmark is generous on purpose. Tighten it case by case only after you have measured the spread of repeated runs on the machine that will host the gate.

Step 2 — Build one bench binary, used twice

Baseline and candidate must be compiled from the same source except for the patch, with the same compiler and the same flags. A different -O level turns a performance gate into a toolchain diff.

The bench skeleton below takes a case id and a size, runs the workload, and returns a value so the optimizer cannot delete the work. Internal timing is printed for debugging; the gate re-times from outside, because it needs end-to-end numbers that include process startup consistently on both sides.

// bench.cpp -- one binary, one case per argv; baseline and candidate share it
#include <chrono>
#include <cstdio>
#include <cstdlib>
#include <string_view>

int run_ring_push_pop(long n);
int run_index_build(long n);

int main(int argc, char** argv) {
  if (argc < 3) return 2;
  std::string_view id{argv[1]};
  long n = std::strtol(argv[2], nullptr, 10);

  auto t0 = std::chrono::steady_clock::now();
  int sink = (id == "ring_push_pop") ? run_ring_push_pop(n) : run_index_build(n);
  auto ms = std::chrono::duration<double, std::milli>(
                std::chrono::steady_clock::now() - t0).count();

  std::printf("%s %.3f %d\n", argv[1], ms, sink); // debug output; gate re-times
  return sink == 12345 ? 0 : 0;
}

Build it twice: once at the merge base (bench_base), once with the candidate patch applied (bench_cand).

Step 3 — Interleave runs and compare medians

The driver runs k rounds. Each round measures baseline and candidate back to back, so machine-level drift is shared.

#!/usr/bin/env python3
"""perf_gate.py -- ratio-based performance gate for AI-written C++ patches.

exit 0 = all budgets met, 1 = regression, 2 = inconclusive (below noise floor)
"""
import argparse, json, statistics, subprocess, sys, time
from pathlib import Path


def one_run(binary, case_id, arg):
    t0 = time.perf_counter()
    subprocess.run([binary, case_id, str(arg)], check=True,
                   stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
    return (time.perf_counter() - t0) * 1e3


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--baseline", required=True)
    ap.add_argument("--candidate", required=True)
    ap.add_argument("--budgets", default="perf_budgets.json")
    args = ap.parse_args()

    cfg = json.loads(Path(args.budgets).read_text())
    floor = cfg.get("noise_floor_ms", 20)
    rows, regressed, skipped = [], 0, 0

    for case in cfg["cases"]:
        cid, arg = case["id"], case["arg"]
        k = case.get("k", cfg.get("default_k", 5))
        base, cand = [], []
        for _ in range(k):                      # interleave: drift hits both
            base.append(one_run(args.baseline, cid, arg))
            cand.append(one_run(args.candidate, cid, arg))

        b, c = statistics.median(base), statistics.median(cand)
        if b < floor:
            skipped += 1
            rows.append((cid, b, c, c / b, "SKIP", "below noise floor"))
            continue

        ratio, limit = c / b, case["baseline_ratio_max"]
        abs_fail = "absolute_ms_max" in case and c > case["absolute_ms_max"]
        status = "REGRESS" if (ratio > limit or abs_fail) else "pass"
        regressed += status == "REGRESS"
        rows.append((cid, b, c, ratio, status, f"limit {limit:.2f}"))

    print("| case | base ms | cand ms | ratio | status | note |")
    print("|---|---|---|---|---|---|")
    for cid, b, c, r, s, note in rows:
        print(f"| {cid} | {b:.1f} | {c:.1f} | {r:.2f} | {s} | {note} |")

    if regressed:
        return 1
    return 2 if skipped else 0


if __name__ == "__main__":
    sys.exit(main())

Run it with ./perf_gate.py --baseline ./bench_base --candidate ./bench_cand. Output is a Markdown table that pastes straight into a pull request comment.

Step 4 — Respect the noise floor and the inconclusive state

A case whose baseline median lands under noise_floor_ms is dominated by process startup and scheduling, not by the code under test. Marking it SKIP is more honest than letting a 0.4 ms case produce a confident 2.5× verdict.

The distinct exit code matters. 2 means the gate could not decide, which needs a human or a bigger input size — not a silent pass. If most cases return 2, the workload sizes are too small: raise arg until the baseline clears the floor.

Step 5 — Quarantine regressions, do not delete cases

When a case fails, the patch is held for review; the case stays. Deleting the case because the slow patch was accepted anyway trains the harness to approve whatever it last saw.

Keep a small history file with the case id, the accepted ratio, and the commit that set the new baseline. The next time the same case fails, the reviewer can see whether the budget was renegotiated deliberately or drifted upward one patch at a time.

A representative gate output

The table below is synthetic, from the illustrative slowdown described at the top.

| case          | base ms | cand ms | ratio | status  | note       |
|---------------|---------|---------|-------|---------|------------|
| ring_push_pop | 41.9    | 117.6   | 2.81  | REGRESS | limit 1.15 |
| index_build   | 96.2    | 101.0   | 1.05  | pass    | limit 1.25 |

Two rows, two different messages. One patch would have merged on correctness alone; the gate caught the container substitution before review.

Where free model access and a free server fit

Candidate patches have to come from somewhere, and a budget gate needs a place to run every night. MonkeyCode's free model access supplies the first: generate several candidate patches for the same issue and let the gate decide which ones are merely different and which ones are worse. Its free server option supplies the second: a host for the recurring perf_gate.py run instead of a laptop that sleeps.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Availability terms are the operator's claim, not a benchmark: free model access with a stated token allowance (10M tokens as of this writing), plus a free server option. Quotas, the model roster, and server specifications change, so read them from the live dashboard rather than from any post, including this one. If the free tier's constraints do not fit, check whether self-hosting or a paid runner does before building the workflow around it.

One practical caveat matters more than the rest. A shared or modest server is a noisy measurement host. That is exactly why this gate uses ratios, interleaved rounds, and an absolute ceiling as a backstop instead of trusting a single millisecond reading. The design does not assume quiet hardware; it assumes the noise is shared.

Limitations and who should skip this

Microbenchmarks are not production workloads. A ratio budget proves that a specific loop got slower, not that users will notice. Service-level latency needs service-level measurement.

The gate also misses scaling regressions that only appear at larger inputs. A patch that is 1.05× slower at n = 500000 may be 4× slower at n = 5000000. Add a second size per case and compare growth, not just the ratio, if that class matters to the codebase.

Caching can game the gate as easily as it games tests: a patch that memoizes results looks fast in a repeat-k run. Behavior cases still have to cover the uncached path. Budgets extend the grading surface; they do not replace it.

Skip this approach if the project already has hardware-isolated performance CI — use that instead of rebuilding it. Skip it too if the baseline build is not reproducible, or if the workload takes under a millisecond, because the gate will spend its life returning 2.

A short checklist before wiring it in

One bench binary, compiled twice from the same flags and compiler.
Ratio budgets in version control, next to the golden cases.
Interleaved rounds, median of at least five.
An honest SKIP for cases under the noise floor, and a non-zero exit code for it.
A budget history file, so a renegotiated limit is a decision rather than an accident.

Correctness gates protect the output. A clock budget protects the cost of producing it — and it is the cheapest place to find out that a green patch made the hot loop three times slower. If the workflow fits, the free tier is a reasonable place to generate the candidate patches this gate will judge.

Benchmark Budgets Are Part of the Grader: A Paired Cost and Latency Gate for AI C++ Patches

Finley Li — Fri, 06 Nov 2026 16:43:18 +0000

Consider a pull request that turns a 12-line C++ change into a 40-line patch. Every golden case passes. CI is green. Two days later the review queue tells a different story: the patch needed three retries per case, pushed p95 compile time up by a third, and burned a quarter more tokens than the model it replaced. The grader never noticed, because it only asked whether the tests passed.

That gap is the subject of this article. The latest benchmarking challenge wrap-up is a useful reminder that leaderboards are snapshots, not budgets. For AI-generated C++ patches, a passing score has a second dimension: tokens, compile time, and wall-clock time per accepted patch.

The failure mode: a green eval with a red budget

A correctness-only harness treats every passing patch as equal. A patch that compiles in 200 ms and one that compiles in 2 s both get a check mark. A patch that uses 800 tokens and one that uses 8,000 tokens both count as one success.

That is fine for a smoke test. It is not fine for a merge gate. Token allowances are finite, CI minutes are finite, and reviewer attention is finite. When a model swap improves pass rate by two points but doubles the tail latency, the eval has found a trade-off; the harness should report it instead of hiding it.

The gate below is deliberately small. It does not replace correctness tests, prompt lock files, or flaky-case triage. It sits after them and answers one narrow question: did the candidate patch set get more expensive per accepted patch?

The artifact: a paired budget gate

The gate reads two JSONL files, one for the baseline and one for the candidate. Both must share a suite hash and the same case IDs. For each case, the script takes the median over repeats, requires every repeat to pass, and computes candidate-to-baseline ratios. It gates the 95th percentile of per-case wall and compile ratios, the median token ratio, and the 90th percentile of per-case token ratios. Queue wait is reported separately as a warning because shared capacity is noisy.

#!/usr/bin/env python3
'''budget_gate.py: paired cost and latency gate for AI C++ patch evals.'''
import argparse
import json
import sys
from collections import defaultdict
from pathlib import Path
from statistics import median

REQUIRED = {
    'run_id', 'case_id', 'variant', 'suite_hash', 'server_class',
    'passed', 'prompt_tokens', 'completion_tokens',
    'compile_ms', 'exec_ms', 'queue_wait_ms',
}
LIMITS = {
    'p95_wall_ratio': 1.20,
    'p95_compile_ratio': 1.25,
    'median_tokens_ratio': 1.15,
    'p90_case_tokens_ratio': 1.50,
    'queue_wait_p95_ms': 30_000,
    'min_paired_cases': 20,
}

def pct(values, q):
    values = sorted(values)
    if not values:
        return 0.0
    k = (len(values) - 1) * q
    lo = int(k)
    hi = min(lo + 1, len(values) - 1)
    return values[lo] + (values[hi] - values[lo]) * (k - lo)

def load(path):
    rows = []
    for n, line in enumerate(Path(path).read_text().splitlines(), 1):
        if not line.strip():
            continue
        try:
            row = json.loads(line)
        except json.JSONDecodeError as exc:
            raise SystemExit(f'{path}:{n}: {exc}')
        missing = REQUIRED - set(row)
        if missing:
            raise SystemExit(f'{path}:{n}: missing {sorted(missing)}')
        rows.append(row)
    return rows

def by_case(rows, variant):
    grouped = defaultdict(list)
    for row in rows:
        if row['variant'] == variant:
            grouped[row['case_id']].append(row)
    return grouped

def case_stats(rows):
    return {
        'passed': all(bool(row['passed']) for row in rows),
        'wall': median(float(row['compile_ms']) + float(row['exec_ms']) for row in rows),
        'compile': median(float(row['compile_ms']) for row in rows),
        'tokens': median(float(row['prompt_tokens']) + float(row['completion_tokens']) for row in rows),
        'queue': median(float(row['queue_wait_ms']) for row in rows),
    }

def ratio(candidate, baseline):
    return candidate / baseline if baseline > 0 else float('inf')

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--baseline', required=True)
    parser.add_argument('--candidate', required=True)
    parser.add_argument('--report', default='budget-report.json')
    args = parser.parse_args()
    base = load(args.baseline)
    cand = load(args.candidate)
    for name, rows in (('baseline', base), ('candidate', cand)):
        hashes = {row['suite_hash'] for row in rows}
        if len(hashes) != 1:
            raise SystemExit(f'{name}: mixed suite_hash {hashes}')
    if {row['suite_hash'] for row in base} != {row['suite_hash'] for row in cand}:
        raise SystemExit('suite_hash mismatch: baseline and candidate are not comparable')
    base_cases = by_case(base, 'baseline')
    cand_cases = by_case(cand, 'candidate')
    paired = sorted(set(base_cases) & set(cand_cases))
    missing = sorted(set(base_cases) - set(cand_cases))
    extra = sorted(set(cand_cases) - set(base_cases))
    if missing:
        raise SystemExit(f'candidate dropped cases: {missing[:10]}')
    need = LIMITS['min_paired_cases']
    if len(paired) < need:
        raise SystemExit(f'need at least {need} paired cases, got {len(paired)}')
    ratios = {'wall': [], 'compile': [], 'tokens': []}
    candidate_queue = []
    failures = []
    for case_id in paired:
        b = case_stats(base_cases[case_id])
        c = case_stats(cand_cases[case_id])
        candidate_queue.append(c['queue'])
        if not b['passed']:
            failures.append(f'{case_id}: baseline did not pass')
            continue
        if not c['passed']:
            failures.append(f'{case_id}: candidate regressed correctness')
            continue
        for key in ratios:
            ratios[key].append(ratio(c[key], b[key]))
    if failures:
        report = {'verdict': 'FAIL', 'reason': 'correctness/coverage', 'failures': failures[:20]}
        Path(args.report).write_text(json.dumps(report, indent=2))
        print(json.dumps(report, indent=2))
        return 1
    checks = {
        'p95_wall_ratio': pct(ratios['wall'], 0.95),
        'p95_compile_ratio': pct(ratios['compile'], 0.95),
        'median_tokens_ratio': median(ratios['tokens']),
        'p90_case_tokens_ratio': pct(ratios['tokens'], 0.90),
        'queue_wait_p95_ms': pct(candidate_queue, 0.95),
    }
    verdict = 'PASS'
    reasons = []
    warnings = []
    for key, limit in LIMITS.items():
        if key == 'min_paired_cases':
            continue
        if key == 'queue_wait_p95_ms':
            if checks[key] > limit:
                warnings.append(f'{key}={checks[key]:.0f} > {limit}')
        elif checks[key] > limit:
            verdict = 'FAIL'
            reasons.append(f'{key}={checks[key]:.3f} > {limit}')
    report = {
        'verdict': verdict,
        'checks': {key: round(value, 3) for key, value in checks.items()},
        'limits': LIMITS,
        'paired_cases': len(paired),
        'extra_cases': extra,
        'reasons': reasons,
        'warnings': warnings,
    }
    Path(args.report).write_text(json.dumps(report, indent=2))
    print(json.dumps(report, indent=2))
    return 0 if verdict == 'PASS' else 1

if __name__ == '__main__':
    sys.exit(main())

A record is one JSON object per line. The minimum fields are shown as a Python template so the shape is explicit; the harness serializes this as JSONL.

record = {
    'run_id': '2026-11-07T10:00:00Z',
    'case_id': 'cpp-vector-reserve-01',
    'variant': 'baseline',
    'suite_hash': 'sha256:...',
    'server_class': 'free',
    'passed': True,
    'prompt_tokens': 412,
    'completion_tokens': 188,
    'compile_ms': 310,
    'exec_ms': 42,
    'queue_wait_ms': 1800,
}

The run command is intentionally boring:

python3 budget_gate.py --baseline baseline.jsonl --candidate candidate.jsonl --report budget-report.json

A failing report should be readable in one glance:

verdict=FAIL
p95_wall_ratio=1.31 (limit 1.20)
median_tokens_ratio=1.08 (limit 1.15)
queue_wait_p95_ms=41200 (warn at 30000)

How the gate decides

Check	Default gate	On breach
`p95_wall_ratio`	`> 1.20`	fail
`p95_compile_ratio`	`> 1.25`	fail
`median_tokens_ratio`	`> 1.15`	fail
`p90_case_tokens_ratio`	`> 1.50`	fail
`queue_wait_p95_ms`	`> 30000`	warn
dropped case	any	fail

The ratios are paired by case, not calculated from aggregate totals. That matters. A candidate can keep the same total token count by trading a cheap case for an expensive one. The paired p90 catches tail hogs; the median catches broad quota burn; the wall and compile gates catch review latency that a pass-rate table never shows.

The queue warning is separate for a reason. On a shared free server, a 40-second queue wait is a capacity signal, not a model-quality signal. It should not fail correctness, but it should stop a team from comparing a laptop baseline with a remote candidate and calling the difference a regression.

A five-step runbook

Freeze the suite hash and write it into every row. Do not compare across suite edits; that is a different benchmark with different accepted-patch economics.
Run the baseline with at least three repeats per case on the intended server_class. Record the prompt hash and model ID if the harness already has them; the budget gate only needs the fields above.
Run the candidate in the same measurement window. For MonkeyCode's free server option, record queue_wait_ms; shared capacity can add minutes that are not model execution time.
Emit JSONL and run the gate. Fix correctness failures before reading budget ratios. A candidate that fails two golden cases has no meaningful median token ratio.
Archive the report next to the prompt lock file. A later model swap can then be compared with a known budget instead of a memory of one.

Where MonkeyCode fits

MonkeyCode's open-source project is relevant to this workflow because the candidate side can run on its free model access and free server option. The operator-supplied offer at the time of writing includes a 10-million-token free allowance and a free server option. Quotas, capacity, and duration are not permanent; verify current terms before sizing a run.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The cost model changes, but the gate does not. Free tokens still run out, and free shared servers still queue. The useful move is to keep baseline and candidate on the same server class, record queue wait separately, and treat a budget regression as a failed eval even when every test is green. The free allowance is enough to try the workflow on a small suite; check the current terms before scaling it.

Limitations and who should not use this

The gate has sharp edges. Token ratios are only comparable when both variants use the same tokenizer; otherwise compare billable units or quota consumption, not raw token counts. Compile time depends on flags, cache state, and machine class, so pin those or the ratio measures the host. Shared free capacity adds variance even with repeats; run baseline and candidate in the same window when possible.

It also starts after correctness. If the candidate fails a golden case, the budget report is noise. If the suite has fewer than 20 paired cases, tail percentiles are unstable; use the gate as a directional check, not a hard merge block. Teams without per-case token and timing telemetry should add that telemetry before adopting the gate. A one-off patch review with no model choice does not need it. And no shared free server should be used to promise a hard real-time SLO; use the free option to find regressions, not to guarantee latency.

What to do next

Run the gate against the last two candidate models on a 20-case suite. A 1.2x p95 wall regression with a flat pass rate is a real finding. So is a 0.98 pass rate with a 2x token burn. Publish the budget report next to the eval score; the pair is more useful than another leaderboard screenshot.

Prompt Diff, Eval Drift: Lock-File the Prompt Half of Your AI C++ Eval Harness

Finley Li — Thu, 05 Nov 2026 15:21:17 +0000

A typical failure looks like this. Someone tightens one sentence in the system prompt — prefer becomes must — and the next sweep of the C++ golden set drops fourteen points. The model gets blamed, a migration gets proposed, and nobody notices that the harness never recorded which prompt produced the baseline.

The numbers in that sketch are illustrative, not a benchmark. The failure shape, though, is common: eval harnesses version their golden cases and their graders, and leave the prompt as an untracked string in a YAML file.

This post covers the missing half. It is a lock-file discipline for everything on the prompt side that can move a verdict, plus a paired replay that tells you whether a drop came from the prompt, the model, or the box.

Why the prompt is the unstable half

A C++ patch harness has two halves. The execution half — compiler, standard flag, sanitizers, test runner — is usually pinned, because someone got burned by a -O2-only failure once. The generation half is usually not pinned at all.

That asymmetry is backwards. Compiler versions change a few times a year. Prompt templates change on Tuesday afternoon, in a PR titled "clarify instructions", with no reviewer looking at eval impact.

Three things drift independently:

The prompt template text, including whitespace and example ordering.
The decoding parameters, which often live in a different file from the template.
The provider-reported model identifier string, which can change meaning under a stable name.

Only the third is outside your control. The first two are pure version-control hygiene, and they cause most of the confusion.

Step 1: Hash every input that can move a verdict

The artifact is a single JSON file committed next to the harness. It stores hashes, not content, so the diff stays readable and the prompts can still be edited normally.

{
  "prompt_template_sha256": "<sha256 of harness/prompt.tmpl>",
  "system_prompt_sha256": "<sha256 of harness/system.txt>",
  "golden_manifest_sha256": "<sha256 over sorted golden/*.json>",
  "grader_version": "2.3.1",
  "decode": { "temperature": 0.2, "top_p": 1.0, "max_tokens": 4096 },
  "model_id_reported": "<the string the provider returned, verbatim>",
  "toolchain": {
    "cc": "clang",
    "cc_banner": "<first line of clang --version>",
    "std": "c++20",
    "sanitizers": ["address", "undefined"]
  }
}

Two details matter more than they look. Record the model identifier exactly as reported, never a friendly alias you invented — you want to detect a silent re-point, not describe your intent. And hash the golden manifest as a tree, so adding, removing, or renaming a case all register as drift.

Step 2: Fail the build on drift, not on vibes

The check is short enough that there is no excuse for skipping it. Plain Python 3.11, standard library only.

#!/usr/bin/env python3
"""tools/lockcheck.py - fail CI when a verdict-moving input changed."""
import hashlib, json, pathlib, subprocess, sys

ROOT = pathlib.Path(__file__).resolve().parents[1]
HARNESS = ROOT / "harness"


def sha256_file(path: pathlib.Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def sha256_tree(paths) -> str:
    h = hashlib.sha256()
    for p in sorted(paths):
        h.update(str(p.relative_to(ROOT)).encode())
        h.update(p.read_bytes())
    return h.hexdigest()


lock = json.loads((HARNESS / "harness.lock.json").read_text())
current = {
    "prompt_template_sha256": sha256_file(HARNESS / "prompt.tmpl"),
    "system_prompt_sha256": sha256_file(HARNESS / "system.txt"),
    "golden_manifest_sha256": sha256_tree((ROOT / "golden").glob("*.json")),
}

drift = {k: (lock[k], v) for k, v in current.items() if lock[k] != v}
for key, (locked, now) in drift.items():
    print(f"DRIFT {key}\n  locked: {locked}\n  now:    {now}", file=sys.stderr)

banner = subprocess.run(
    [lock["toolchain"]["cc"], "--version"],
    capture_output=True, text=True, check=True,
).stdout.splitlines()[0].strip()
if banner != lock["toolchain"]["cc_banner"]:
    print(f"WARN toolchain changed: {banner}", file=sys.stderr)

sys.exit(1 if drift else 0)

Toolchain changes warn instead of failing, because a container rebuild should not block a prompt fix. Prompt and golden-case changes fail hard, because those two are the ones people edit casually.

Step 3: Replay both versions, paired

A before/after comparison across two nights is worthless. CI machines differ, the provider's load differs, and C++ tests are timing-sensitive at the margins. Run the same golden set, same order, same repeat count, on both prompt versions, back to back, in one job.

Store one file per case per repeat, runs/<run-id>/<case>/<k>.json, containing at minimum {"verdict": "pass" | "fail"}. Then build a flip table.

#!/usr/bin/env python3
"""tools/flip.py A_DIR B_DIR - paired verdict comparison."""
import json, pathlib, sys
from collections import defaultdict


def pass_rate(run_dir):
    rates = defaultdict(list)
    for f in sorted(pathlib.Path(run_dir).glob("*/*.json")):
        rec = json.loads(f.read_text())
        rates[f.parent.name].append(rec["verdict"] == "pass")
    return {k: sum(v) / len(v) for k, v in rates.items()}


a, b = pass_rate(sys.argv[1]), pass_rate(sys.argv[2])
rows = []
for case in sorted(set(a) | set(b)):
    pa, pb = a.get(case), b.get(case)
    if pa is None or pb is None:
        kind = "missing"
    elif pa == pb:
        kind = "stable"
    elif pa == 1.0 and pb < 1.0:
        kind = "regressed"
    elif pb == 1.0 and pa < 1.0:
        kind = "fixed"
    else:
        kind = "unstable"
    rows.append((case, pa, pb, kind))

for case, pa, pb, kind in rows:
    print(f"{kind:9s} {case:44s} A={pa:.2f} B={pb:.2f}")
summary = defaultdict(int)
for *_, kind in rows:
    summary[kind] += 1
print("\nsummary:", dict(summary))

Run it with at least five repeats per case. With fewer, 1.00 -> 0.50 is indistinguishable from noise and the table will lie to you in a confident font.

Step 4: Read the flip table before reading the diff

The four categories answer different questions, and conflating them is where teams lose days.

Category	Meaning	Next action
`regressed`	Was always passing, now not	Real signal — read the prompt diff
`fixed`	Was always failing, now passes	Verify it fixed the intended bug, not the grader
`unstable`	Passing inconsistently on both sides	Harness bug, not a prompt bug
`stable`	No change	Ignore, however tempting

unstable deserves emphasis. If a case cannot pass ten times out of ten under the locked prompt, it never belonged in a regression gate. Every unstable row you find is a case that has been injecting randomness into every comparison you have run.

Step 5: Where a free tier and a free server actually help

A paired replay multiplies work: two prompt versions, N golden cases, k repeats, plus the compile-and-sanitize step for each patch. That is the point where a cost gate quietly turns five repeats into two, and two repeats produce an unstable column you cannot trust.

MonkeyCode provides free model access and a free server option, both operator-supplied as of writing. In this workflow the free model access removes the billing reason to lower k, and the free server option is a reasonable place to park the nightly job: lockcheck.py on every commit, the paired replay once per night. Quotas, available models, and server specs change without notice, so check the project page for current numbers rather than trusting a figure quoted in any article, this one included.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

One caution. The product is a means to run the loop, not a substitute for it. A free tier that lets you run two thousand generations is worthless if the golden set has forty cases and the grader is a regex.

Step 6: Know what the lock does not cover

The lock file pins your inputs. It says nothing about three failure modes worth naming.

First, grader bugs. A grader that silently accepts patches with uninitialized reads will show a beautiful fixed column. Canary cases — deliberately broken patches the grader must reject — belong in the manifest, not in a comment.

Second, upstream model drift. When the provider changes what a model identifier means, your lock file records only that the string stayed the same. The flip table catches the effect; the lock file cannot predict it. Re-baseline deliberately, and record the baseline run id in the lock file so the next person knows which sweep the numbers came from.

Third, test-environment leakage. Address-space randomization, thread scheduling, and filesystem ordering can flip a C++ case without any prompt change at all. Randomize case order between the two arms of the replay — if the flip table changes shape, the harness is leaking state.

Who should skip this

Teams with fewer than roughly twenty golden cases should not build a lock-file pipeline yet. With a set that small, every case flip is a conversation, not a statistic, and the ceremony costs more than it returns.

Solo developers running one-off generations should skip it too. The discipline pays off when a failure gets misattributed to a model — which requires at least two people and a shared baseline.

And anyone whose real problem is a flaky C++ test suite should fix that first. A harness cannot stabilize tests it did not write; it can only record how unstable they are.

Start with the cheap half

Before touching prompts again, commit a lock file and a twenty-line check. The first time a passing sweep drops fourteen points, you will know within one command whether the prompt changed, the toolchain changed, or the model moved beneath a stable name — and you will spend the afternoon on the actual bug instead of on a migration proposal.

If you want to run the paired replay somewhere it does not compete with your CI minutes, MonkeyCode's free tier is a low-stakes place to start.

Kill-Matrix Pruning: Cutting the Golden Cases That Never Catch an AI C++ Patch

Finley Li — Tue, 27 Oct 2026 12:48:26 +0000

A team running an AI patch pipeline once showed me their nightly eval: 214 golden cases, a clean green dashboard, and a production regression that the suite had walked straight past. The postmortem took an hour. Then someone asked the obvious question, and nobody had an answer: how many of those 214 cases had ever failed for a reason worth knowing?

We instrumented the suite instead of arguing about it. Sixty-one cases had never failed on any patch the pipeline had ever scored. Thirty-three failed on nearly every patch, which made them noise rather than signal. The suite looked thorough and was mostly ceremony.

This article describes the instrument we built: a kill matrix. It is a small, reproducible way to find out which golden cases actually earn their runtime, and which ones you are paying for out of habit. A free-tier model and a free-tier runner make the measurement cheap enough to run weekly, but the method stands on its own.

Step 1: Define the mutants before you look at the cases

A kill matrix needs a fixed oracle side. That side is a set of known-wrong C++ patches, called mutants, seeded by hand or by a mutation tool. The rule: every mutant must be a bug a reviewer would reject in a real PR.

Start with twenty. Small, boring, semantics-bearing mutations work best:

// original
std::size_t clamp_index(std::size_t i, std::size_t n) {
    return i < n ? i : n - 1;
}

// mutant M07: boundary flipped, one-past-end reachable
std::size_t clamp_index(std::size_t i, std::size_t n) {
    return i <= n ? i : n - 1;
}

// mutant M12: unsigned underflow when n == 0
std::size_t clamp_index(std::size_t i, std::size_t n) {
    return i < n ? i : n - 1;  // unchanged, but n == 0 now allowed by caller
}

Record each mutant as a patch file plus a one-line description of the defect. Do not include anything you cannot defend in a code review. Style-only mutants produce a matrix full of false kills.

Step 2: Build the matrix with a fixed seed and a fixed compiler

The matrix is a two-dimensional table: mutants down the rows, golden cases across the columns. Each cell answers one question — did this case fail when it saw this mutant?

The runner below is deliberately dumb. It compiles the mutant, applies the case, and writes a boolean. No model, no grader, no rubric. The point of this pass is to measure the cases, not the model.

# build_kill_matrix.py
import json, pathlib, subprocess, sys

ROOT = pathlib.Path(__file__).parent
MUTANTS = sorted((ROOT / "mutants").glob("*.patch"))
CASES = sorted((ROOT / "cases").glob("case_*.py"))

def run_case(case: pathlib.Path) -> bool:
    """Return True when the case FAILS (i.e. the mutant is killed)."""
    proc = subprocess.run(
        [sys.executable, str(case)],
        cwd=ROOT / "build",
        capture_output=True,
        timeout=120,
    )
    return proc.returncode != 0

matrix = {}
for mutant in MUTANTS:
    subprocess.run(["git", "apply", str(mutant)], cwd=ROOT, check=True)
    subprocess.run(["cmake", "--build", "build", "-j"], cwd=ROOT, check=True)
    row = {}
    for case in CASES:
        row[case.stem] = run_case(case)
    matrix[mutant.stem] = row
    subprocess.run(["git", "checkout", "--", "."], cwd=ROOT, check=True)

(ROOT / "kill_matrix.json").write_text(json.dumps(matrix, indent=2))
print(f"{len(MUTANTS)} mutants x {len(CASES)} cases written")

Two hygiene rules keep the matrix honest. First, freeze the compiler and flags for the whole run; a matrix built across two toolchains is two matrices stapled together. Second, run each cell once and record timing, because a case that takes four minutes to kill one mutant is a different purchase than one that takes four seconds.

Step 3: Read the matrix in four buckets

The raw JSON is not the deliverable. The classification is. Sort each case by how many mutants it kills, then look for the four shapes that matter.

Shape	Signal in the matrix	Action
Discriminating	Kills at least one mutant no other case kills	Keep, and keep it fast
Redundant	Its kills are a strict subset of another case's kills	Fold into the stronger case
Never fires	Zero kills across the whole mutant set	Investigate or delete
Always fires	Kills every mutant, including the baseline	Broken case, or it is testing the build

The "never fires" bucket is where the silent regressions hide. A case with zero kills is either asserting something no realistic bug can violate, or it is silently skipping itself — a missing fixture, a disabled #ifdef, a test binary that swallowed a nonzero exit. Both need a human, not a deletion.

Step 4: Compress with greedy set cover, then validate on a holdout

Once you know which cases kill which mutants, the minimal suite is a set cover problem. Greedy selection gets you within a constant factor of optimal and takes about thirty lines.

# prune_suite.py
import json

matrix = json.load(open("kill_matrix.json"))
mutants = list(matrix)
cases = list(next(iter(matrix.values())))

killers = {c: {m for m in mutants if matrix[m][c]} for c in cases}
chosen, uncovered = [], set(mutants)

while uncovered:
    best = max(cases, key=lambda c: len(killers[c] & uncovered), default=None)
    if best is None or not (killers[best] & uncovered):
        break
    chosen.append(best)
    uncovered -= killers[best]

print(f"kept {len(chosen)}/{len(cases)} cases; uncovered mutants: {sorted(uncovered)}")
json.dump(chosen, open("suite.json", "w"), indent=2)

Split the mutant set 70/30 before you run this. Prune on the 70 percent, then check how much of the 30 percent the reduced suite still kills. A pruned suite that drops below roughly ninety percent holdout coverage was fit to the mutants, not to the code.

Step 5: Put the sweep on a free runner and measure the spend

This is where the harness meets infrastructure. The account's outreach brief points at two MonkeyCode options — free model access and a free server option — and both are relevant here for one reason: the kill matrix is a volume job. Twenty mutants across a hundred cases is two thousand executions, and the interesting question is not whether you can afford it once, but whether you can afford it every week.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Treat the free tier as an unverified budget until your own numbers exist. The brief cites a free token allowance; that figure is a marketing claim until a script counts what your sweep actually consumes. Log tokens per mutant, wall-clock per case, and failures per run, then compare against the allowance. If the ratio is comfortable, schedule the sweep weekly. If not, run it on the top twenty cases only and prune quarterly.

# one sweep, three numbers worth keeping
/usr/bin/time -f "wall=%e s" python build_kill_matrix.py 2>> sweep.log
grep -c '"true"' kill_matrix.json >> sweep.log
wc -l sweep.log

The same caution applies to the free server. Nothing in the brief says how long it stays up, how fast it builds, or whether it survives a -j build of your project. Measure those three things on day one, and keep a fallback runner configured so a capacity surprise costs you one sweep, not one quarter.

Step 6: Guard the pruning against its own success

Aggressive compression has a failure mode: the suite becomes fast, green, and blind. Three cheap guards prevent it.

Holdout mutants. Keep the thirty percent permanently out of selection, as in Step 4.
Canary cases. Add two cases that must fail on every run, one by design and one by an injected marker. If either passes, the harness is broken, not the code.
Quarterly re-seed. Models drift and so do bugs. Add ten fresh mutants every quarter drawn from real review rejections, and re-run the matrix.

Who should not do this

Skip this method if your mutant set would be invented rather than harvested from real rejections — a matrix of synthetic bugs measures your imagination, not your suite. Skip it if you have fewer than roughly fifteen cases, because set cover over a tiny universe just renames the problem. And skip it if the eval runs on a shared cluster where a two-thousand-execution sweep would starve other jobs; the measurement is not worth the queue.

What remains is a cheap habit. Once a quarter, you learn which of your golden cases still catch something, which never did, and what a full sweep actually costs on whatever free capacity you are leaning on. The dashboard stops being a number and starts being an argument.

Shuffle the Golden Cases: Catching Eval State Leaks That Order Hides

Finley Li — Tue, 20 Oct 2026 09:32:28 +0000

Shuffle the Golden Cases: Catching Eval State Leaks That Order Hides

A patch-eval suite can be green for months and still be lying. The lie usually has nothing to do with the patch. It has to do with the sequence.

Here is the failure shape. Case 12 compiles src/vec.cpp with -DEVAL_VARIANT=sse and drops build/vec.o into a scratch directory shared by the whole run. Case 17 wants the scalar variant, but its build script skips compilation when build/vec.o already exists. Case 17 links the SSE object, passes its test, and reports PASS. The patch under review never touched vector code.

Nothing in case 17's log mentions case 12. Nothing in the diff mentions either. The suite is order-dependent, and because the order was fixed in the manifest, the dependency was invisible.

1. Treat order as an input you must vary

Most eval harnesses treat the case list as configuration. It is really an input variable — as much as the patch, the compiler flags, or the model version.

If a verdict changes when that input changes, the suite has a leak. If it does not, order independence is a property you can assert, not a hope you maintain.

The gate is simple: run the suite in a canonical order twice, then in a handful of seeded shuffles. Compare verdict vectors, not logs. Flag any case whose verdict moves.

2. Make the manifest the only source of order

Cases live in a JSON manifest. Each entry owns its command, its timeout, and its expected verdict. No shell script reorders anything downstream.

[
  {"id": "c01_vec_basic",   "cmd": ["bash", "cases/c01.sh"], "expect": "PASS", "timeout": 300},
  {"id": "c12_vec_sse",     "cmd": ["bash", "cases/c12.sh"], "expect": "PASS", "timeout": 300},
  {"id": "c17_vec_scalar",  "cmd": ["bash", "cases/c17.sh"], "expect": "PASS", "timeout": 300},
  {"id": "c22_asan_owner",  "cmd": ["bash", "cases/c22.sh"], "expect": "FAIL", "timeout": 300}
]

Two details matter. First, expect is recorded even for cases that must fail — a negative case that starts passing is a regression too. Second, every case runs in the same scratch root per suite run. Isolating each case in a fresh container would hide exactly the bug this gate exists to find.

3. Run canonical twice, then seeded shuffles

The harness below is the whole gate, roughly sixty lines. It is plain Python on purpose: it needs to be readable enough that someone debugging a flip can trust it.

# shuffle_gate.py — order-dependence gate for a patch-eval suite
import json, random, shutil, subprocess, sys, time
from pathlib import Path

CASES = {c["id"]: c for c in json.loads(Path("cases.json").read_text())}
ROOT  = Path("scratch")

def run_suite(order, tag):
    work = ROOT / tag
    if work.exists():
        shutil.rmtree(work)          # reset once per run, never per case
    work.mkdir(parents=True)
    rows = []
    for cid in order:
        case = CASES[cid]
        t0 = time.monotonic()
        try:
            p = subprocess.run(case["cmd"], cwd=work, capture_output=True,
                               text=True, timeout=case.get("timeout", 300))
            got = "PASS" if p.returncode == 0 else "FAIL"
        except subprocess.TimeoutExpired:
            got = "TIMEOUT"
        rows.append({"id": cid, "verdict": got,
                     "secs": round(time.monotonic() - t0, 2)})
    return rows

def vector(rows):
    return {r["id"]: r["verdict"] for r in rows}

Canonical order runs twice with different tags so a TIMEOUT or a flaky model call shows up before shuffling begins. Shuffles use a fixed seed list committed to the repo, because a flip you cannot reproduce is a flip you cannot fix.

def main():
    base_order = list(CASES)
    a = run_suite(base_order, "canon_a")
    b = run_suite(base_order, "canon_b")
    report = {"canonical": {}, "shuffles": []}

    va, vb = vector(a), vector(b)
    for cid in base_order:
        if va[cid] != vb[cid]:
            report["canonical"][cid] = {"a": va[cid], "b": vb[cid]}

    for seed in (17, 41, 1234, 90210):
        order = base_order[:]
        random.Random(seed).shuffle(order)
        rows = run_suite(order, f"shuf_{seed}")
        v = vector(rows)
        flips, prev = [], None
        for r in rows:
            if v[r["id"]] != va[r["id"]]:
                flips.append({"case": r["id"], "base": va[r["id"]],
                              "now": v[r["id"]], "after": prev})
            prev = r["id"]          # heuristic attribution, see step 5
        report["shuffles"].append({"seed": seed, "order": order, "flips": flips})

    Path("leaks.json").write_text(json.dumps(report, indent=2))
    dirty = report["canonical"] or any(s["flips"] for s in report["shuffles"])
    sys.exit(1 if dirty else 0)

if __name__ == "__main__":
    main()

The exit code is the point. A gate that prints a warning gets ignored by week three.

4. Read the verdict vector, not the log tail

Logs are noisy. Four hundred lines of compiler output and one ASan report look the same whether a case leaked state or genuinely regressed.

The vector is small and diffable. On a clean suite, leaks.json carries empty objects and four empty flip lists. On a dirty one it names the case, the canonical verdict, the shuffled verdict, and the case executed immediately before it.

The after field is a lead, not a verdict. A leak can travel through the filesystem or the environment and surface many cases later — the SSE object in step 1 survives three unrelated cases before case 17 consumes it.

5. Confirm by isolation, then quarantine

Attribution only becomes evidence when the pair reproduces in isolation. Run the two cases alone, in the suspected order, with the same scratch root. If the flip returns, the pair is real.

Three artefacts come out of confirmation:

A minimal reproducer — two case IDs and a seed, added as a permanent assertion in tests/test_order.sh.
A quarantine entry in leaks.json with a note pointing at the shared resource: build/vec.o, port 8080, ASAN_OPTIONS, /tmp/eval_cache.
A fix on the offending side, not the victim. Case 12 should build out-of-tree; case 17 should not be silently patched to clean the world first.

Quarantine means the entry stays visible and the gate keeps failing until the fix lands. Deleting the case removes the signal and restores the false green.

6. Shuffling is not free, and that changes the budget

A four-shuffle matrix on a twenty-case suite is five full suite runs. On a shared build machine that cost lands on wall clock and disk, not just CPU.

This is where free model access changes the triage loop rather than the verdict. Confirmed flips are rare; most shuffled runs are clean and never need the expensive model. Re-grading triage runs with a free-tier model through MonkeyCode keeps the matrix affordable, and the free server option gives the harness a persistent box for the seed list, the scratch root, and leaks.json between runs. The strong model is then reserved for the few flips that survive isolation.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

One caveat about shared boxes: a free server is also a shared-resource magnifier. Ports, /tmp, and disk quotas are the most common leak carriers in exactly this environment, which makes the gate more valuable there, not less.

7. Decision table for the flip report

Observation	Likely cause	Action
Flip only under seeded shuffles, stable in both canonical runs	Filesystem, port, or env leak between cases	Confirm pair in isolation; fix the producing case
Flip between `canon_a` and `canon_b`, same order	Case nondeterminism or unseeded model output	Pin seed and flags; re-check the grader with a canary
Case passes alone, fails after any neighbour	Environment mutation by whoever ran first	Diff `env` and `cwd` before and after the neighbour
Whole suite flips in one run	Harness-level state: disk full, port held, stale scratch root	Reset the scratch root once per run, never per case
Flip appears only in `-O2` builds	Optimization-sensitive undefined behaviour	Route it to an opt-split gate, not this one

8. Limits, and who should skip this

The gate finds order coupling. It says nothing about whether a grader is correct, whether golden cases cover the patched line, or whether the model chose a sound fix.

Skip it when every case already runs in a fresh container or VM — isolation removes the leak class by construction, and the shuffle matrix is pure cost. Skip it for suites so slow that five runs exceed the CI budget; sample ordered pairs instead of full shuffles. Skip it below roughly ten cases, where a human reading the manifest beats a matrix.

Adopt it when cases share a workspace, a port, a cache, or an environment — which is to say, on most real C++ patch suites running on one machine.

A suite that only passes in one order is not a suite. It is a script with a fixed cast, and the patch you are grading is not the thing deciding the outcome. Commit the seed list alongside the cases, so the next person sees the same flips you did.

Flaky Is Not Failed: A Repeat-k Triage Loop for AI-Written C++ Patches

Finley Li — Mon, 19 Oct 2026 08:17:05 +0000

Six golden cases turn red in the same CI run. The team spends two days reverting a patch that was correct, and the single case that had genuinely regressed gets fixed last, almost by accident.

That scenario is a composite — the timeline is illustrative, the pattern is not. Any harness that grades model-written C++ against a fixed set of golden cases will eventually produce a run where the same case disagrees with itself. Two things follow. A two-state pass/fail result cannot describe that world, and a harness without a vocabulary for "I do not know" will convert infrastructure noise into engineering decisions.

The artifacts below are templates. They target Python 3.11 and a CMake/CTest tree, and they were not executed for this article, so treat paths and the runner call as starting points.

Step 1 — Make the result three-state

A golden case can fail for reasons that have nothing to do with the patch: a rate-limited model call, a build that timed out on a shared box, a runner that lost its cache. Grading those as fail is how noise becomes a revert. The runner below records infra_unknown as a distinct outcome and refuses to reason about it later.

# triage.py — repeat each golden case K times, then classify. Template.
import hashlib, os, re, subprocess, tempfile
from dataclasses import dataclass

K = int(os.environ.get("REPEAT_K", "5"))
INFRA_RE = re.compile(r"(429|503|connection reset|build timeout|no space left)", re.I)

@dataclass
class Attempt:
    case: str
    repeat: int
    status: str   # pass | fail | infra_unknown
    exit_code: int
    digest: str
    note: str = ""

def digest(blob: str) -> str:
    # Timing and pid lines track the scheduler, not the behaviour.
    cleaned = "\n".join(
        line for line in blob.splitlines()
        if not re.search(r"(total time|pid=|elapsed)", line, re.I)
    )
    return hashlib.sha256(cleaned.encode()).hexdigest()[:12]

Step 2 — Reuse the case, not the process

Every repeat must start from a clean state. The environment pinning in this runner is deliberate: LC_ALL=C and TZ=UTC remove two classic sources of "it passed on the other machine", and a per-repeat TMPDIR keeps stale artifacts from leaking across attempts.

def run_case(case: str, repeat: int) -> Attempt:
    with tempfile.TemporaryDirectory() as td:
        proc = subprocess.run(
            ["ctest", "--test-dir", "build", "-R", f"^{case}$",
             "--output-on-failure"],
            capture_output=True, text=True, timeout=900,
            env={**os.environ, "LC_ALL": "C", "TZ": "UTC",
                 "REPEAT": str(repeat), "TMPDIR": td},
        )
    blob = proc.stdout + proc.stderr
    if INFRA_RE.search(blob):
        return Attempt(case, repeat, "infra_unknown",
                       proc.returncode, "", blob[-400:])
    return Attempt(case, repeat,
                   "pass" if proc.returncode == 0 else "fail",
                   proc.returncode, digest(blob))

Step 3 — Classify into four verdicts

The interesting bucket is not flaky. It is stable_pass_varying: a case that passes every single repeat while printing two different digests. That case is a silent regression waiting for a slightly different machine, and it is invisible to any harness that only reads exit codes.

def classify(attempts: list[Attempt]) -> dict:
    graded = [a for a in attempts if a.status != "infra_unknown"]
    infra = len(attempts) - len(graded)
    if infra * 2 > len(attempts):
        return {"verdict": "inconclusive", "infra": infra}
    passes = sum(a.status == "pass" for a in graded)
    digests = len({a.digest for a in graded if a.status == "pass"})
    if passes == 0:
        verdict = "stable_fail"
    elif passes < len(graded):
        verdict = "flaky"
    elif digests > 1:
        verdict = "stable_pass_varying"
    else:
        verdict = "stable_pass"
    return {"verdict": verdict, "infra": infra,
            "passes": passes, "digests": digests}

Step 4 — Bind each verdict to a CI action

A verdict that does not change what CI does is a log line, not a gate.

verdict	CI action
`stable_pass`	green
`stable_pass_varying`	yellow: normalise the output before it becomes a red
`flaky`	red on `main`, quarantineable on a branch
`stable_fail`	red: the regression the harness exists to find
`inconclusive`	no grade at all; re-run the sweep later

That last row carries the most weight. Against a shared model endpoint, inconclusive will be common, and a harness that turns it into a failure teaches reviewers to ignore failures.

Step 5 — Quarantine with an expiry date

Unstable cases need somewhere to live that is not /dev/null and not a permanently ignored test.

# quarantine.yaml
- case: containers/unordered_dedup
  reason: iteration order leaks into stdout digest
  owner: finley
  opened: 2026-10-19
  expires: 2026-11-02

# gate.py — an expired quarantine is a build failure, not a reminder
import datetime as dt, sys, yaml

today = dt.date.today()
bad = [q for q in yaml.safe_load(open("quarantine.yaml"))
       if dt.date.fromisoformat(q["expires"]) < today]
if bad:
    sys.exit("expired quarantine: " + ", ".join(q["case"] for q in bad))

Step 6 — Treat the source of the nondeterminism

Repeat-k is triage, not treatment. Three C++ changes remove most of the churn seen in practice.

// 1. Order leaks into output: per-process hashing, different bytes each run.
std::unordered_set<std::string> seen;     // digest churns
std::set<std::string>           seen_ok;  // stable bytes, assertable

// 2. Wall clock: inject it, then assert on it.
struct Clock {
    virtual ~Clock() = default;
    virtual std::time_t now() const = 0;
};
struct FixedClock final : Clock {
    std::time_t t;
    std::time_t now() const override { return t; }
};

// 3. Worker count: 3 threads locally, 32 in CI, different interleavings.
const unsigned workers =
    std::min(4u, std::max(1u, std::thread::hardware_concurrency()));

Add -fsanitize=address,undefined to the triage build rather than the shipping build. Uninitialised reads and signed overflow produce exactly the intermittent failures that repeat-k flags, and a sanitizer report names the bug instead of leaving a pass-rate puzzle. setarch -R is also worth a try when a crash reproduces under one address layout only.

What k buys, and what it does not

k	model calls per case	catches a ~50% flake	catches a ~10% flake
2	2	often	rarely
5	5	~94%	~41%
10	10	~99.9%	~65%

The arithmetic is the binomial one: a 10% flake survives ten repeats with probability 0.9^10 ≈ 0.35. So repeat-k is a coarse filter, and it is honest to say so before anyone wires k=20 into every pull request. Rare nondeterminism is caught better by digest divergence on green runs and by a nightly sweep, not by raising k in the merge path.

Where free model access and a free server fit

A k=10 sweep over 40 golden cases is 400 model calls, and on a metered endpoint that arithmetic appears on an invoice. MonkeyCode's free model access and free server option make such a sweep affordable for a side project or a first pass on a feature branch, which is the cheapest moment to discover that six of your cases were never stable.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two constraints shape how the tool is used here. Free capacity is shared and rate-limited, so the INFRA_RE branch in Step 1 is not decoration — it is the point of the three-state design. The free tier also belongs in the triage loop, not the release gate, because a merge decision should never depend on capacity someone else controls. The offer itself (the operator states 10M free tokens plus a free server option) is worth confirming on the project's own page before planning around it.

To see the classifier fire before wiring it into CI, point it at the two cases you already suspect and spend the first sweep on a scratch branch.

Who should skip this

Projects whose evals are already deterministic: fixed seeds, ordered containers, injected clocks, no concurrency. Repeat-k would buy nothing there.
Teams that cannot store per-attempt records. Without digests and repeat indices, the classifier has no input and no audit trail.
Anyone planning k=20 on every pull request. The wall time is real, and nightly is the honest cadence.
Harnesses that grade prose rather than behaviour. Output digests are meaningful when the program prints bytes, not when a model writes a paragraph.

The habit worth keeping is small: a red run should produce a verdict with a name, and flaky and inconclusive are names. Without them every red run looks identical, and the team slowly learns to treat all of them as noise — including the one that was real.

Model Swap or Grader Bug? A Replay-First Eval Harness for AI C++ Patches

Finley Li — Sat, 17 Oct 2026 06:02:53 +0000

A patch queue goes red over a weekend. The pass rate falls, nobody touched the compiler, and two people are already rewriting prompts to win the metric back. The real change was one line in a grader that stopped trimming trailing whitespace from program output.

That is not a model problem. It is an attribution problem. An eval harness that only reports a pass rate cannot tell you which of its three inputs moved: the prompt, the fixture, or the grader. The workflow below fixes that by making grading replayable, so every failure gets sorted into a bucket before anyone edits a prompt.

Three inputs, three hashes

An eval case has a prompt, a fixture (the code under test), and a grader. Each one can change independently. If the harness stores only a verdict, all three changes collapse into the same red cell in a spreadsheet.

The fix is boring and effective: hash all three inputs at record time, and hash them again at grading time. A verdict that changes with identical hashes is a model-side event. A verdict that changes with identical model output is a grader-side event. Anything else is a case edit that has to be reviewed like code.

case_id            = split-range-001
prompt_hash        = 8a3f...   (sha256 of prompt file contents)
fixture_hash       = 1c90...   (sha256 of the .cpp file)
grader_hash        = e41d...   (sha256 of the grading script)
output_hash        = 77b2...   (sha256 of the recorded completion)

Golden cases small enough to read

A case that needs a paragraph of explanation to justify its expected output will not survive a reviewer's attention span. Keep each file under a screen. One prompt, one fixture, one expected observable.

{
  "id": "split-range-001",
  "prompt_file": "prompts/fix_split_range.md",
  "fixture": "fixtures/split_range.cpp",
  "compile_flags": ["-std=c++20", "-O2", "-Wall", "-Werror"],
  "stdin": "0 1 2 3 4\n",
  "expect": { "exit_code": 0, "stdout": "0 1 2 3\n", "stderr_empty": true }
}

Note what is not in that file: no prose rubric, no model name, no temperature. Those belong to the run configuration, because they are exactly the things that get swapped without anyone recording it.

Step 1: record once, replay forever

Recording stores the completion alongside its input hashes. Replay grades the stored completion offline, with no network call. The first run of a new case is the only one that spends anything.

# evals/harness.py — reference implementation, Python 3.11+
import hashlib, json, pathlib, subprocess

ROOT = pathlib.Path(__file__).parent

def digest(*parts: str) -> str:
    h = hashlib.sha256()
    for p in parts:
        h.update(p.encode())
        h.update(b"\x00")
    return h.hexdigest()[:16]

def input_hash(case: dict) -> str:
    prompt = (ROOT / case["prompt_file"]).read_text()
    fixture = (ROOT / case["fixture"]).read_text()
    grader = (ROOT / "graders" / "io_match.py").read_text()
    return digest(prompt, fixture, grader)

def grade(case: dict, completion: str) -> dict:
    src = (ROOT / case["fixture"]).read_text() + "\n" + completion
    build = ROOT / "build" / case["id"]
    build.mkdir(parents=True, exist_ok=True)
    (build / "main.cpp").write_text(src)
    cc = subprocess.run(
        ["g++", *case["compile_flags"], "main.cpp", "-o", "a.out"],
        cwd=build, capture_output=True, text=True,
    )
    if cc.returncode != 0:
        return {"compiles": False, "stderr": cc.stderr[-2000:]}
    run = subprocess.run(
        ["./a.out"], cwd=build, input=case.get("stdin", ""),
        capture_output=True, text=True, timeout=10,
    )
    exp = case["expect"]
    ok = (
        run.returncode == exp["exit_code"]
        and run.stdout == exp["stdout"]
        and (not exp.get("stderr_empty", True) or run.stderr == "")
    )
    return {"compiles": True, "ok": ok, "stdout": run.stdout}

The compile step is the expensive part, and it is also the part people skip when they grade on stdout alone. A patch that prints the right bytes but fails -Werror is not a pass.

Step 2: split replay from live

Two subcommands, one report format. The replay arm never touches the network; the live arm calls whatever endpoint the run configuration points at.

# no tokens spent; grades recorded completions
python evals/harness.py replay --report out/replay.json

# spends tokens; records new completions into cassettes/
python evals/harness.py live --endpoint "$EVAL_ENDPOINT" --report out/live.json

# attribute every difference
python evals/harness.py diff out/replay.json out/live.json

Run replay on every pull request. Run live on a schedule, or when a model or prompt change is proposed. That ratio matters: replay is deterministic, cheap, and suitable for a hard gate; live is a sampling instrument and should never block a merge on a single flip.

Step 3: where free capacity actually helps

MonkeyCode's free model access and free server option are the two pieces that make the live arm affordable for an individual maintainer. The value is not the free tier itself; it is that repeated grading runs become cheap enough to do often. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two honest constraints. First, quota and lifetime terms on a free tier change, so read the project's current page instead of trusting a blog post — including this one. Second, a free workspace is the wrong place to store anything you would not publish, so keep cassettes free of proprietary fixtures.

The harness itself is provider-shaped, not provider-bound. --endpoint is a shell variable for a reason: swap it and the diff logic keeps working.

Step 4: classify the drift, do not just count failures

This is the whole point. One function, three buckets, checked in order.

def classify(case, recorded, replayed, live) -> str:
    if input_hash(case) != recorded["input_hash"]:
        return "CASE_DRIFT"      # prompt, fixture, or grader was edited
    if replayed["ok"] != recorded["ok"]:
        return "GRADER_DRIFT"    # same inputs, same output, different verdict
    if live["ok"] != replayed["ok"]:
        return "MODEL_DRIFT"     # new completion, new verdict
    return "STABLE"

Order is not cosmetic. If the case hash moved, everything downstream is uninterpretable, so it gets reported first and the run is marked "not comparable" rather than "failed".

The resulting report reads like a triage list:

split-range-001   STABLE
parse-args-004    GRADER_DRIFT   verdict ok->fail, output identical
wide-int-002      MODEL_DRIFT    verdict fail->ok
sort-stable-007   CASE_DRIFT     fixture changed; run not comparable

GRADER_DRIFT on a passing case is the most valuable line in the file. It means a grader edit silently reclassified work that nobody asked to re-evaluate.

Step 5: gate on transitions, not totals

A CI gate that fails on "pass rate below 90%" will block merges for reasons unrelated to the merge. Gate on the classification instead.

- run: python evals/harness.py replay --report out/replay.json
- run: python evals/harness.py diff out/replay.json out/last-green.json --fail-on GRADER_DRIFT

MODEL_DRIFT on a live run should open an issue, not fail a build. CASE_DRIFT should require a reviewer to look at the fixture diff. Only grader-side movement deserves a hard stop, because it corrupts every future comparison until someone explains it.

One caveat on sensitivity: this harness assumes your graders can already fail when they should. Proving that is a separate problem with its own fixtures, and it is worth solving before you trust any bucket above.

When to replay and when to go live

Question	Replay	Live
Did a grader edit change verdicts?	yes	no
Did a new model change outputs?	no	yes
Is this cheap enough for every PR?	yes	no
Is this deterministic?	yes	no
Can it catch a fixture typo?	yes	yes

Limitations and who should skip this

Cassettes rot. A recorded completion is only valid for the fixture and flags it was recorded against, so a fixture edit invalidates it by design — which is good, but it means cassettes are not a long-term archive.

Cases that depend on wall-clock time, thread scheduling, or unseeded randomness will produce MODEL_DRIFT noise that is really nondeterminism in the test. Fix the fixture first or drop the case.

Finally, this harness measures individual case flips, not statistical significance. A team trying to decide whether a model is "better" by a few percentage points needs many samples and a proper interval; a single replay run will happily report a difference that is noise. And anyone using this as a security or correctness sign-off for shipped code should stop here — golden cases guard a pipeline, they do not certify a binary.

If the goal is to keep the live arm running without owning a runner, the replay path is the place to start: it costs nothing, it catches the failure mode that burns the most time, and it works before any endpoint is configured.

Rubric Graders Need Canaries: Testing the Test That Grades C++ Patches

Finley Li — Fri, 16 Oct 2026 05:00:35 +0000

Rubric Graders Need Canaries: Testing the Test That Grades C++ Patches

A pipeline starts scoring C++ patches with a model-backed rubric. Diffs that change printed precision, drop a noexcept guarantee, or leak a file descriptor now receive a 1-10 score before a human opens them. For a few weeks the dashboard looks reasonable, reviewers trust the number, and nobody asks whether the grader still separates good patches from bad ones. Then a prompt edit lands, the upstream model is swapped, and every patch scores 8. Nothing crashes. The grader simply stops discriminating, and the pipeline keeps reporting green.

That failure mode is independent of which model you call. A rubric grader is a program with an untested dependency, and it deserves the treatment the code under it already gets: golden cases, a canary set, a hard gate, and a ledger. The harness below is standard-library Python, a few JSON canary files, and one cron line. It is deliberately boring, because the interesting part is that the grader fails loudly instead of silently.

Layer 0 thinks, layer 1 judges

Split grading into two layers before writing any code. The deterministic layer answers questions with one correct answer: does the patched tree compile, does the golden case exit zero, does stdout match after a fixed locale and precision. The rubric layer answers questions with a range of defensible answers: is the naming consistent with the header it implements, does the commit message describe the actual change, does the new overload respect the documented contract.

Most silent regressions live on the boundary. A grader that is only deterministic cannot see prose contracts. A grader that is only a rubric cannot see a changed std::setprecision. Layer 0 gates the build, layer 1 explains the patch, and each layer needs its own canaries.

A canary here means a labeled example with a fixed expected answer that is committed to the repository. It is not a test of the patch. It is a test of the test.

Step 1 - Freeze golden cases for the deterministic layer

Start with one golden case per documented output contract. Keep the case small enough that a failure points at one behavior, and pin everything that could drift: locale, precision, and the toolchain described in your own build docs.

// golden/report_precision.cpp - contract: ratio printed with 3 decimals
#include <iostream>
#include <iomanip>

int main() {
    const double ratio = 2.0 / 3.0;
    std::cout << std::setprecision(3) << std::fixed << ratio << std::endl;
    return 0;
}

Run it through the same command the eval batch uses, so the grader and the patch pipeline cannot disagree about flags:

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --target report_precision
./build/report_precision > out.txt
diff -u golden/report_precision.expected out.txt

The diff is the grader. If it passes, the patch is not yet good, but it is at least not obviously broken.

Step 2 - Commit canaries in both directions

A canary set that only contains bad patches rewards a paranoid grader. A canary set that only contains good patches rewards a lenient one. Include both, and store each patch as its own file so the JSON stays readable and diffs stay reviewable.

eval/
  canaries/
    precision-3-to-2.json
    honest-refactor.json
    tie-case-review.json
    diffs/
      precision-3-to-2.patch
      honest-refactor.patch
      tie-case-review.patch

Each canary declares what it expects and why. The note field is the part humans actually maintain.

{
  "id": "precision-3-to-2",
  "kind": "rubric",
  "direction": "must-fail",
  "patch_path": "diffs/precision-3-to-2.patch",
  "expected": 3,
  "note": "Breaks the output contract in docs/report.md. Any score above 5 means the rubric is not reading the contract."
}

{
  "id": "honest-refactor",
  "kind": "rubric",
  "direction": "should-pass",
  "patch_path": "diffs/honest-refactor.patch",
  "expected": 8,
  "note": "Pure rename plus a comment fix, output byte-identical. Below 6 means the rubric punishes chore diffs."
}

{
  "id": "precision-3-to-2-build",
  "kind": "build",
  "patch_path": "diffs/precision-3-to-2.patch",
  "golden_cmd": "./build/report_precision | diff -u golden/report_precision.expected -",
  "note": "Must exit non-zero. If this passes, the deterministic layer is not running."
}

Keep the count honest. Twenty labeled canaries drawn from real review comments are worth more than two hundred generated ones, because generated canaries encode the same assumptions as the rubric prompt.

Step 3 - Measure agreement before trusting scores

Agreement is the only cheap health signal a rubric grader has. Exact match is strict; within-one match tolerates the ordinary noise of a 1-10 scale. Both numbers belong in the gate.

#!/usr/bin/env python3
# grade_gate.py - run canaries before trusting rubric scores.
import json, os, pathlib, subprocess, sys

CANARY_DIR = pathlib.Path(os.environ.get('CANARY_DIR', 'eval/canaries'))
LEDGER = pathlib.Path(os.environ.get('GRADER_LEDGER', 'eval/grader_ledger.jsonl'))
RUBRIC_ID = os.environ.get('GRADER_VERSION', 'unversioned')


def load_canaries():
    for path in sorted(CANARY_DIR.glob('*.json')):
        canary = json.loads(path.read_text())
        patch = (CANARY_DIR / canary['patch_path']).resolve()
        canary['diff_text'] = patch.read_text()
        yield canary


def call_model(prompt):
    # Transport layer. Plug in whichever client your runtime provides.
    # This function is untrusted by design; that is why canaries exist.
    raise NotImplementedError('wire your client here')


def rubric_score(diff_text, rubric):
    reply = call_model(rubric + '\n\nDIFF:\n' + diff_text)
    return json.loads(reply)

Now the comparison itself, plus the gate. The thresholds are policy defaults, not measured results: a canary set of forty concrete review-derived cases can usually carry a higher bar than one of twelve fuzzy ones. Tune them once, write the number in the repo, and treat any change to it as a reviewable commit.

def check_rubric_canaries(canaries, rubric, min_exact=0.80, min_within_one=0.95):
    exact = within_one = 0
    misses = []
    for canary in canaries:
        if canary.get('kind') != 'rubric':
            continue
        expected = canary['expected']
        got = rubric_score(canary['diff_text'], rubric)['score']
        if got == expected:
            exact += 1
        else:
            if abs(got - expected) == 1:
                within_one += 1
            misses.append({'id': canary['id'], 'expected': expected,
                           'got': got, 'delta': got - expected})
    total = sum(1 for c in canaries if c.get('kind') == 'rubric') or 1
    rates = {'exact': exact / total, 'within_one': (exact + within_one) / total}
    ok = rates['exact'] >= min_exact and rates['within_one'] >= min_within_one
    return ok, rates, misses

Build canaries run the golden command instead of a model, and check the opposite thing: they must fail.

def check_build_canaries(canaries, workdir):
    results = []
    for canary in canaries:
        if canary.get('kind') != 'build':
            continue
        subprocess.run(['git', 'apply', canary['patch_path']], cwd=workdir, check=True)
        proc = subprocess.run(canary['golden_cmd'], cwd=workdir, shell=True)
        subprocess.run(['git', 'restore', '.'], cwd=workdir, check=True)
        results.append({'id': canary['id'], 'exit': proc.returncode,
                        'ok': proc.returncode != 0})
    return results

Step 4 - Keep a grader ledger next to the token ledger

A score is only meaningful with the grader revision and the model identity attached. Write one JSON line per gate run, including the token count, so drift and cost are visible in the same file.

def log_run(rates, misses, model_id, tokens):
    record = {'grader': RUBRIC_ID, 'model': model_id, 'rates': rates,
              'misses': misses, 'tokens': tokens}
    with LEDGER.open('a') as fh:
        print(json.dumps(record), file=fh)

if __name__ == '__main__':
    rubric = pathlib.Path(sys.argv[1]).read_text()
    canaries = list(load_canaries())
    ok, rates, misses = check_rubric_canaries(canaries, rubric)
    build = check_build_canaries(canaries, '.')
    log_run(rates, misses, os.environ.get('MODEL_ID', 'unknown'),
            int(os.environ.get('TOKENS_USED', '0')))
    healthy = ok and all(r['ok'] for r in build)
    print(json.dumps({'healthy': healthy, 'rates': rates, 'misses': misses,
                      'build': build}, indent=2))
    sys.exit(0 if healthy else 1)

The exit code is the whole interface. A healthy gate returns 0 and the eval batch proceeds; an unhealthy gate returns 1 and the batch stops before it spends anything on scoring.

Step 5 - Run the gate where nobody has to remember it

A gate that depends on a human typing a command is a gate that will be skipped during the week it matters most. Two things make the scheduled version practical: a machine that stays on, and model access that does not require a per-run decision.

MonkeyCode, described by its maintainers as an open-source project, offers free model access and a free server option, with the free tier described as including 10M tokens. That combination is enough to host a weekly canary run plus the rubric calls it makes, without reserving a dedicated box. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

# cron: every Monday 06:17, on the free server option
17 6 * * 1 cd /srv/eval && python3 grade_gate.py rubrics/patch_v3.md >> logs/canary.log 2>&1

Read the ledger rather than the estimate. The claim worth trusting is not a predicted token count; it is the tokens field on last week's line, compared with the line before it.

Which layer owns which signal

Signal to grade	Layer 0 (deterministic)	Layer 1 (rubric)
Compiles, links, exits zero	Yes	No
stdout under fixed locale and precision	Yes	No
Wall-clock or allocation budget	Yes	No
Naming consistent with the header	Partly, via lint rules	Yes
Prose output contract in docs	No	Yes
Commit message matches the change	No	Yes
Whether the change is worth merging	No	Advisory only, never binding

If a row can be pushed into layer 0, push it. Every signal moved out of the rubric layer removes a source of drift, and the rubric layer keeps only the judgments a compiler cannot express.

Limitations and who should skip this

Canaries are a sample, and agreement is not correctness. If the labels are wrong, the gate certifies a wrong grader with a confident number. Re-label a handful of canaries every few months, especially the ones that never fail.

Rubric scores drift when the model or the prompt changes, and only a scheduled run of the ledger detects that. Single-run scores stay noisy even with a fixed prompt, so treat a one-point swing as a signal to check the ledger, not as a verdict on the patch.

The quota and server claims above are operator descriptions of the offer, not measurements taken for this article. Verify current terms before wiring a paid-critical pipeline to a free tier, and keep the gate able to run against a different client.

Skip this approach if you have fewer than roughly twenty real diffs to label, because reading the patches is then cheaper than grading them. Skip it for safety-critical, licensing, or compliance judgments, which need deterministic evidence and a named reviewer. Skip it if diff text cannot leave your network, unless every layer runs locally. And skip the rubric layer entirely if your grading budget exceeds the cost of the human review it is meant to save.

The harness is small on purpose. Golden cases, canaries in both directions, one agreement gate, one ledger. Start with a single canary that should fail and confirm the gate returns non-zero; if it returns zero, the grader was never watching.

Mutants Before Merges: A Two-Sided Sensitivity Gate for C++ Eval Harnesses

Finley Li — Thu, 15 Oct 2026 11:15:02 +0000

An illustrative trace. A 40-line C++ patch clears the compiler, passes 41 golden cases and merges. Two weeks later a downstream consumer reports wrong scaling on a load path none of the goldens touch. Nothing in the pipeline crashed. The harness was blind — every test passed because no assertion ever inspected the broken path.

That trace is a composite pattern, not a report on a named team. The defect class behind it is measurable all the same. This article builds a gate that asks whether a C++ eval harness is capable of failing, and that refuses to grade a patch when the answer is no.

Golden cases measure agreement, not sensitivity

Golden cases answer one question: does current output match recorded output? That is agreement. It says nothing about whether the harness would notice if the behavior underneath changed.

Two properties matter. Sensitivity is the probability that the harness fails when behavior breaks. Specificity is the probability that it passes when behavior is correct. A harness that fails on every run has perfect sensitivity and no usefulness; a harness that passes everything is the failure mode in the opening trace.

More golden cases raise agreement coverage. They raise sensitivity only if the new cases touch code that was previously unasserted, and stdout diffs frequently do not. A case that checks a final total can pass while an intermediate bound check was deleted.

The gate below measures both properties before a patch is graded, using mutants as calibrated faults.

Step 1 — Freeze the harness contract

A gate needs a fixed pass/fail definition, or the mutants measure noise. Pin the build type, the runner and the wall-clock ceiling in one place, and treat any change to them as a change to the gate itself.

cmake -S . -B build -DCMAKE_BUILD_TYPE=RelWithDebInfo
cmake --build build -j2
ctest --test-dir build --output-on-failure   # the bench command

Everything downstream calls exactly these commands. If this patch class needs sanitizers, keep a second profile and run the gate twice, one ledger per profile.

Step 2 — Run a flake pre-pass

Kill rates computed on a flaky harness are fiction. Before generating any mutant, run the bench three times on the untouched tree and require identical results.

for i in 1 2 3; do
  ctest --test-dir build --output-on-failure >/dev/null 2>&1 || echo "flake run $i"
done

Any deviation stops the gate. Ordering dependence, timing assumptions and shared temp files surface here far more cheaply than inside a mutant run.

Step 3 — Build a two-class mutant seed set

Mutants are small deliberate faults. They fall into two classes with opposite expected results.

Class	Expectation	Purpose	Examples
sensitivity	harness must FAIL	proves the harness can see a break	`<=` to `<`, sign flip, deleted early return, dropped `const`, rounded float literal
inert	harness must PASS	proves the harness is not failing on trivia	reformatted comment, reordered `#include`, renamed local, added dead branch

Hand-write ten sensitivity mutants and three inert ones. They cost nothing and they encode what the team actually cares about. Model-drafted mutants extend the set afterwards; the note further down explains why they are candidates rather than ground truth.

[
  {"id": "M01", "class": "sensitivity", "file": "src/scale.cpp",
   "find": "if (n <= cap)", "repl": "if (n < cap)"},
  {"id": "M02", "class": "sensitivity", "file": "src/scale.cpp",
   "find": "return a - b;", "repl": "return b - a;"},
  {"id": "M03", "class": "sensitivity", "file": "src/scale.cpp",
   "find": "  const auto lim = limit;", "repl": "  auto lim = limit;"},
  {"id": "I01", "class": "inert", "file": "src/scale.cpp",
   "find": "// clamp input", "repl": "// clamp input to cap"}
]

Each anchor must match exactly once. An ambiguous anchor is a skipped mutant, never a silent edit. That rule keeps the ledger honest when a patch moves code around.

Step 4 — Drive the gate

The driver below is an illustrative reference implementation, not code executed for this article. It restores each source file in a finally block, so a crash cannot leave a mutant in the tree.

#!/usr/bin/env python3
# mutgate.py - two-sided sensitivity gate for a C++ eval harness
import json, pathlib, subprocess, sys, time

ROOT = pathlib.Path(sys.argv[1] if len(sys.argv) > 1 else ".").resolve()
KILL_FLOOR = float(sys.argv[2]) if len(sys.argv) > 2 else 0.60
BUILD = "cmake --build build -j2"
BENCH = "ctest --test-dir build --output-on-failure"
TIMEOUT = 90

def run(cmd, timeout=TIMEOUT):
    t0 = time.time()
    try:
        p = subprocess.run(cmd, cwd=ROOT, shell=True, capture_output=True,
                           text=True, timeout=timeout)
        return p.returncode, (p.stdout + p.stderr)[-1200:], round(time.time() - t0, 1)
    except subprocess.TimeoutExpired:
        return 124, "wall clock exceeded", timeout

def observe():
    rc, log, secs = run(BUILD)
    if rc != 0:
        return "BUILD_ERROR", log, secs
    rc, log, secs2 = run(BENCH)
    return ("FAIL" if rc != 0 else "PASS"), log, secs + secs2

def main():
    mutants = json.loads((ROOT / "mutants.json").read_text())
    ledger = {"clean_control": None, "mutants": [], "verdict": None}

    state, log, secs = observe()
    ledger["clean_control"] = {"state": state, "secs": secs}
    if state != "PASS":
        ledger["verdict"] = "REJECT_HARNESS"
        (ROOT / "mutgate-ledger.json").write_text(json.dumps(ledger, indent=2))
        print("REJECT_HARNESS: clean tree does not pass")
        return

    for m in mutants:
        src = ROOT / m["file"]
        original = src.read_text()
        hits = original.count(m["find"])
        if hits != 1:
            ledger["mutants"].append({"id": m["id"], "class": m["class"],
                                      "state": "SKIPPED", "note": f"anchor x{hits}"})
            continue
        src.write_text(original.replace(m["find"], m["repl"], 1))
        try:
            state, log, secs = observe()
        finally:
            src.write_text(original)
        ledger["mutants"].append({"id": m["id"], "class": m["class"],
                                  "state": state, "secs": secs})

    sens = [x for x in ledger["mutants"]
            if x["class"] == "sensitivity" and x["state"] != "BUILD_ERROR"]
    killed = [x for x in sens if x["state"] == "FAIL"]
    inert_noise = [x for x in ledger["mutants"]
                   if x["class"] == "inert" and x["state"] == "FAIL"]
    build_errors = [x for x in ledger["mutants"] if x["state"] == "BUILD_ERROR"]
    rate = len(killed) / len(sens) if sens else 0.0

    if build_errors:
        ledger["verdict"] = "REVIEW_BUILD_ANCHORS"
    elif inert_noise:
        ledger["verdict"] = "REVIEW_GRADER_NOISE"
    elif rate >= KILL_FLOOR:
        ledger["verdict"] = "ACCEPT"
    else:
        ledger["verdict"] = "UNKNOWN_HARNESS_BLIND"

    (ROOT / "mutgate-ledger.json").write_text(json.dumps(ledger, indent=2))
    print(json.dumps({"kill_rate": round(rate, 2), "verdict": ledger["verdict"]}))

if __name__ == "__main__":
    main()

Run it as python3 mutgate.py . 0.60. The threshold is a policy choice, not a discovered constant. Start at 0.6, raise it for safety-critical patches, and record the value beside the ledger.

Step 5 — Read the verdict table, not just the number

Clean control	Inert mutants	Kill rate	Verdict
PASS	all PASS	at or above floor	ACCEPT - the harness earns the right to grade this patch
PASS	all PASS	below floor	UNKNOWN - survivors mark unasserted paths; write cases instead of merging on green
PASS	any FAIL	any	REVIEW - harness fails on harmless change; fix specificity first
PASS	-	-	REVIEW - an anchor or mutation caused a build error; kills are not counted
FAIL	-	-	REJECT - the harness itself is broken; no patch can be graded

UNKNOWN is the useful state. It converts a false green into an explicit work item, and it costs one line in the review checklist: survivors list, case inventory, owner.

Step 6 — Turn survivors into cases

Every sensitivity mutant that survives is a concrete assertion the harness does not have. Map it to the smallest case that would fail: an input where n == cap, a negative operand, a value that must not be rounded. Group survivors by the function they touch and open one case-writing task per group rather than one per mutant.

Keep the ledger as a per-patch artifact. Two runs of the same patch with different verdicts usually mean the mutant set changed, not the code.

Where disposable mutants and a free runner fit

A mutation gate needs a lot of throwaway work: dozens of candidate mutants per patch, plus a runner that rebuilds each time in a sandbox. Both halves tolerate imperfection, which changes the economics.

Model-drafted mutants are cheap candidates, not ground truth. A model produces edits that look plausible, which correlates with the bugs a reviewer would miss, but the distribution is biased toward what the model finds natural. It will miss the boundary case a maintainer has been burned by. The workflow that holds up: hand-write the sensitivity and inert seed set, ask a model for thirty more candidates per patch, then keep only those whose anchors resolve and whose class is unambiguous.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator states the current offering includes free model access and a free server option, with a free token allowance advertised at the time of writing as 10 million tokens. Allowances, hardware and terms change, and the project is the only authority on its own limits, so verify the current terms before depending on them.

The free server option fits the runner half because mutation runs are disposable by construction: a container that rebuilds a mutant, runs the bench and is discarded holds no state worth keeping. Keep the merge-blocking golden run in whatever CI the team already trusts, and spend the disposable capacity on the sensitivity suite.

Limitations and who should not use this

Kill rate is a heuristic, not a probability of catching real defects. Mutants and real bugs are different distributions, and a high kill rate can coexist with an unasserted path that no mutant happened to hit.

BUILD_ERROR mutants are excluded from both numerator and denominator on purpose. Counting a compile failure as a kill inflates the rate and hides the case where a refactor broke the anchors.

Mutant runs multiply CI time by the size of the set. With 40 sensitivity mutants and a 30-second bench, the gate costs roughly 20 minutes plus rebuilds. Budget for it, or run it nightly against a sample of recent patches.

Skip this approach if the harness is already a long-running fuzzing or sanitizer campaign, if the codebase needs formal guarantees rather than sampled faults, or if a two-person team has fewer than a dozen golden cases to begin with. In that state, writing cases teaches more than measuring them.

Closing

A harness that cannot fail cannot be trusted when it passes. Build the two-sided gate once, keep the mutant set small and hand-seeded, and read UNKNOWN as a scheduled task rather than a failure. If the runner half is what is blocking you, MonkeyCode's free model access and free server option are a reasonable place to try it on real patches.

Debug Totals Lie at Release: An Opt-Split Gate for C++ Patch Evals

Finley Li — Tue, 13 Oct 2026 00:25:07 +0000

Debug Totals Lie at Release: An Opt-Split Gate for C++ Patch Evals

The failure mode is ordinary, and it hides in a green log. A grader accepts a C++ patch because the debug binary prints 18432. A later release build of the same sources prints 18416. Nobody edited the prompt between those two runs. The optimizer met a signed addition that had already left the representable range.

That split is a silent regression. A single golden print, captured under one flag set, cannot see it.

This note describes an opt-split gate for AI C++ patches. One frozen input tape, one hand-written oracle, and the candidate built at -O0 and -O2. The gate fails when the oracle stays stable and the candidate does not. A prompt hash pin stops a later draft from changing the test without a recorded edit.

What a single print misses

Generated C++ often looks finished inside a driver that prints one number. The driver and the patch share a translation unit. Warnings are quiet, or they were never enabled. The build uses whatever optimization the local default happens to be, which is frequently closer to -O0 than to the flags a package will ship.

Signed overflow, a read of an uninitialized scalar, and a dependency on evaluation order can all survive that setup. A second flag set sometimes keeps the arithmetic inside the range the oracle demands. Sometimes it stops hiding the bug. The opt-split gate does not prove the absence of undefined behavior. It catches the cheaper mismatch: debug output and release output disagree, while a checked oracle prints the same bytes twice.

Files the gate owns

Four inputs define a run, and none of them is a dashboard screenshot. prompt.txt is the only prose the draft step may read. prompt.pin stores the SHA-256 of that file from the last accepted wording change. tape.txt lists signed integers, one token per line, and ends with END.

oracle.cpp is written by a person. candidate.cpp is the patch under test, and it must export the same signature the oracle exports. The contract stays narrow so the grade is about values, not about prose.

// contract.hpp
#pragma once
#include <cstdint>
#include <optional>

std::optional<std::int64_t> fold_checked(const std::int64_t* data, std::size_t n);

A returned integer is the exact sum. std::nullopt means the fold is rejected. The candidate is scored against the oracle decision, not against a golden string sitting in the prompt.

Step 1. Pin the prompt before a draft exists

A changed prompt is a changed test, even when every tape byte is identical. The pin check refuses to grade on drift. It does not try to be clever about whitespace.

python3 - << 'PY'
import hashlib, pathlib, sys
prompt = pathlib.Path("prompt.txt").read_bytes()
digest = hashlib.sha256(prompt).hexdigest()
pin = pathlib.Path("prompt.pin")
if not pin.exists():
    pin.write_text(digest + "\n", encoding="ascii")
    print("pinned", digest)
    sys.exit(0)
expected = pin.read_text(encoding="ascii").strip()
if digest != expected:
    print("prompt pin mismatch")
    print("expected", expected)
    print("actual  ", digest)
    sys.exit(2)
print("prompt pin ok", digest)
PY

Treat a mismatch as a harness failure, not as a patch failure. Update prompt.pin only in a commit that quotes the new wording. Nightly jobs must not rewrite the pin to make themselves green.

Step 2. Freeze the tape outside the driver

The driver reads tape.txt, calls fold_checked, and prints REJECT or a decimal integer. It contains no expected answers. A model that can see the golden line inside the driver will often echo it, and the echo will match at every optimization level.

// tape_driver.cpp
#include "contract.hpp"
#include <fstream>
#include <iostream>
#include <string>
#include <vector>

int main() {
    std::ifstream in("tape.txt");
    std::vector<std::int64_t> data;
    std::string tok;
    while (in >> tok) {
        if (tok == "END") {
            break;
        }
        data.push_back(std::stoll(tok));
    }
    std::optional<std::int64_t> got = fold_checked(data.data(), data.size());
    if (!got) {
        std::cout << "REJECT\n";
        return 0;
    }
    std::cout << *got << "\n";
    return 0;
}

Start with three tapes, and keep them in separate files. One sum fits in 64 bits. One sum crosses the top of the signed 64-bit range. One file contains only END, the empty range.

The samples below are unexecuted fixtures until a local compiler has produced the four binaries. The method does not depend on a hosted runner.

10
20
30
END

9223372036854775807
1
END

Step 3. Build both sides at two levels

Hold the standard, the warning set, and the include path fixed. Change only the optimization flag. Record the compiler identity before the first object file exists, so a later green row can be tied to a toolchain.

mkdir -p out
g++ --version | head -n 1 | tee out/compiler.txt
CXXFLAGS="-std=c++20 -Wall -Wextra -Werror -I."
g++ $CXXFLAGS -O0 -o oracle_o0 oracle.cpp tape_driver.cpp
g++ $CXXFLAGS -O2 -o oracle_o2 oracle.cpp tape_driver.cpp
g++ $CXXFLAGS -O0 -o cand_o0 candidate.cpp tape_driver.cpp
g++ $CXXFLAGS -O2 -o cand_o2 candidate.cpp tape_driver.cpp

A candidate that fails to compile receives a compile grade, and the matrix stops. Partial binaries are not compared. An oracle that fails to compile means the gate is misconfigured. That result is not a patch defect.

The oracle itself should be boring. This implementation is a proposal for the fixture, not a claim that a model emitted it, and not a measured result.

#include "contract.hpp"
#include <cstdint>
#include <limits>
#include <optional>

std::optional<std::int64_t> fold_checked(const std::int64_t* data, std::size_t n) {
    __int128 acc = 0;
    for (std::size_t i = 0; i < n; ++i) {
        acc += data[i];
        if (acc > std::numeric_limits<std::int64_t>::max()) {
            return std::nullopt;
        }
        if (acc < std::numeric_limits<std::int64_t>::min()) {
            return std::nullopt;
        }
    }
    return static_cast<std::int64_t>(acc);
}

Run each binary from a directory that contains only the chosen tape.txt. Capture stdout bytes. Do not trim, and do not pass the text through a shell that rewrites newlines.

./oracle_o0 > out/oracle_o0.txt
./oracle_o2 > out/oracle_o2.txt
./cand_o0 > out/cand_o0.txt
./cand_o2 > out/cand_o2.txt
cmp -s out/oracle_o0.txt out/oracle_o2.txt

Step 4. Score a four-cell matrix

Compare the files with cmp. A missing newline is a mismatch. A numeric parser that forgives spaces will hide the drift this gate exists to catch.

Oracle `-O0` vs `-O2`	Candidate vs oracle at `-O0`	Candidate vs oracle at `-O2`	Grade
differ	any	any	oracle unstable; do not score the patch
match	differ	any	functional miss
match	match	differ	opt-split fail
match	match	match	pass for this tape

The third row is the silent regression. The patch met the contract in the debug build and broke it, or changed it, at -O2. Store the four stdout files beside the prompt hash. A later job that flips only cand_o2 is a regression even when the debug line still matches the earlier example.

Repeat the matrix for every tape. The patch passes the gate only when every tape passes. One opt-split fail fails the run. Do not average rows into a soft score. A mean will bury a release-only break under three comfortable debug matches.

Step 5. Attach a draft host without baking in its limits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The matrix does not need a hosted model. A local editor can drop candidate.cpp into the tree, and the four g++ lines still apply. MonkeyCode enters only when a team wants a draft step or a remote build host. It is an open-source project. Operator materials describe free model access and a free server option. Both are availability claims, so confirm them in the current project docs on the day of the run.

Keep the draft blind. It may read prompt.txt and contract.hpp. It may not read oracle.cpp, the tapes, or earlier stdout. Showing the oracle to the model turns the grade into a transcription exercise, and transcription will pass at both optimization levels for the wrong reason.

Do not hard-code a token allowance, a machine size, or a rental duration into prompt.pin or the grader. Those numbers move, and a stale ceiling will outlive the plan that published it. If the account cannot start a build that day, fail the job closed and leave the previous pin untouched.

The free server, when it is actually available, is a fit for the four compile-and-run lines. The pass/fail rule should stay in a script the reviewer can diff on a laptop. A clean split is local pin check, remote compile, local grade. The host returns stdout files and the compiler version line, and the reviewer's copy of the matrix script assigns the grade.

Teams that already keep prompt.pin can send the frozen prompt to the current free model access and run this opt-split matrix on the free server, after reading that day's limits in the project docs. The grade script should still run unchanged if that host is offline.

Limitations, and who should skip the gate

Agreement across -O0 and -O2 is a screen, not a proof. Undefined behavior can survive both flag sets, and it can disappear under both. Sanitizer builds belong in a separate gate. This matrix also ignores throughput. A patch may pass every tape and still be too expensive to ship, so grade cost with a different fixture.

Skip the approach when the specification allows divergence under fast-math flags, or when the contract is approximate and a tolerance is the real oracle. Skip it when the oracle and the candidate are supposed to reject different edges. That mismatch is a spec argument, not a regression.

Skip it for patches that cannot be built as one extra translation unit, such as a plugin ABI or a generated lexer. The sample driver uses std::stoll and will throw on a corrupt tape. That is a harness bug, and the run should fail before any patch grade is written.

The __int128 accumulator is a GCC and Clang extension, not portable C++. On a toolchain without it, replace the oracle with a checked add written in standard C++. Pin that choice next to the compiler version line. Otherwise two machines will disagree about the oracle and the matrix will blame the patch.

What the log must hold

Each run record needs the prompt hash, the compiler version string, the four flag lines, and the stdout hashes. A green debug line without those fields is how a single total survives review. The release build was never graded, so nobody had a file to open.

The opt-split gate makes that file mandatory. Same tape, two optimization levels, and a human oracle that does not travel inside the prompt. When the release total moves and the oracle does not, the patch does not ship.