DEV Community: Vladyslav Dmitriiev

The AI writes the tests. It doesn't get to grade them.

Vladyslav Dmitriiev — Wed, 22 Jul 2026 15:11:37 +0000

Originally published at dmitriiev.dev

Here is the rule the whole thing is built on: every stage is checked by something that did not produce its output. A parser, a different model, a real exit code, a person. No agent gets to declare its own work done.

That is not an AI idea. It is one of the oldest in engineering under a newer name: the thing that makes something should not be the thing that certifies it. Compilers, CI, code review, mutation testing all run on it. Producer is not validator.

Most of the 2026 wave of AI in testing forgets it. The pitch is to let an agent write and run your tests. A lot of what ships drops a model into the hot path of CI: the same kind of system that wrote an assertion also decides, on every build, whether that assertion is reasonable. That is a closed loop. A model is good at inventing a plausible test and much worse at catching its own wrong assumptions, and letting it grade itself just lets it be confidently wrong twice in the same direction. Writing tests is the cheap part an LLM is genuinely good at. Proving they are worth running is the work, and it takes someone other than the writer.

So the writing here is almost incidental, and every artifact a model produces gets handed to something else to check. None of this is a whiteboard design; it comes out of maintaining real Playwright suites, not out of a diagram. The whole thing is public, demo app included, and runs end to end in a few minutes: ai-qa-pipeline.

The input is prose, and prose is checked before any model runs

You describe a feature in plain UI language, no selectors and no code:

# Feature: Checkout

### Rejects an invalid card number
- Add a product to the cart
- Go to checkout and enter card number 1234
- Submit
Expect: an error message says the card number is invalid

Before a single model is called, a parser reads that file and can reject it for free. No feature heading, no scenarios, or a scenario without exactly one Expect: line, and it stops right there. That last one is a real constraint, not a formality: if you cannot name one outcome for a scenario, the contract makes you split it. One scenario, one assertion. It has an ergonomic cost, a checkout you think of as one journey with four things to verify becomes four scenarios repeating the same steps, and the trade is deliberate: one clean primary assertion per test is what makes the mutation check downstream mean anything.

Every generation step has an independent check bolted to it

Each box hands off to a different kind of judge:

feature.md
   │
   ▼
Feature Reviewer  ── is this testable, or would a writer have to guess?
   │ pass
   ▼
Writer  ◄────────┐
   │ draft       │ reject with a problem list
   ▼             │
Judge (a different model) ── approve, or send it back
   │ approve
   ▼
real Playwright run  ── green by exit code, not by a model's opinion
   │ red
   ▼
Debugger ── may fix mechanics, must not touch meaning
   │ green
   ▼
Judge again ── did the fix quietly weaken anything?
   │
   ▼
tests/generated/  (scratch, untrusted)

The Feature Reviewer judges testability, not taste. It rejects a scenario only when a step is not a concrete user action, or the expected outcome is not observable in the UI, or the scenarios contradict each other. It is told in as many words not to reject for style or for edge cases it would have added. Scope is the feature owner's call, not the agent's. The line it draws is concrete: Submit checkout, expect checkout to work gets rejected, because "works" is nothing the UI can show, while Submit with card number 1234, expect an error that the card is invalid passes, because there is exactly one observable outcome to assert. A rejection stops the pipeline and prints what was wrong, and the fix is editing the feature file, not arguing with the model.

The writer works from ground truth, not vibes. It gets three things and nothing else: the feature file, the app's actual source, and one hand-written frozen spec as the house style. Every data-testid and every piece of UI text it uses has to exist in that source. The app source is the anti-hallucination anchor: selectors get copied from what is really there instead of imagined from what a shop probably looks like. That anchor is not free, though. Someone has to point the writer at the right source, and on a real app curating that slice is manual work, which the last section is honest about.

The judge runs on a different model on purpose. Writer on Sonnet, judge on Opus by default. Be honest about how independent that actually is: both are Anthropic models trained on overlapping data, so this is a weak form of producer-is-not-validator, not a clean-room second opinion. The value is narrower and still real. The judge is a separate, more capable instance with no stake in the draft it is reading, so it does not defend the writer's habits the way the writer does. The point is the independence, not Opus; the judge is a config flag, and pointing it at a different vendor entirely buys you more of the same thing. It is not magic either way. Both read the same static source, so a selector that only breaks in the rendered DOM can slip past both, and that is what the real Playwright run downstream is for. What a second model reliably catches is the writer rationalizing its own draft: a tautological assertion, a missing await, an Expect: outcome asserted more weakly than the feature asked for. The write-and-judge loop is capped at three drafts. Three rejections kill the run with the judge's last problem list attached, and in practice that almost always means the feature file was ambiguous.

The debugger is where agentic pipelines usually cheat

If the real run comes back red, a Debug Agent gets the spec, the Playwright output, and the app source. Its contract is asymmetric, and the asymmetry is the whole point. It may fix locators, waits, navigation, ordering, test data. It may not change what any test asserts: expected text, counts, URLs, the choice of primary assertion. And if the spec is red because the assertion actually contradicts how the app behaves, that is not a bug to paper over. The agent returns NOFIX: <why> and the pipeline stops with that reason, because a spec that can only go green by asserting less should fail loudly.

The debugger is told not to touch assertions, and the pipeline does not trust the telling. Any spec that went through debugging goes back to the judge after it turns green. The judge never sees a diff, and it does not need one: it re-reads the primary assertion against the fixed reference it used the first time, the feature file's Expect: line. An assertion quietly loosened to force a pass, exact text swapped for a bare visibility check, no longer matches what the feature demanded, and gets rejected on a green run. Two honest limits. The re-read is a model judgment, not a proof, so it is only as strong as the Expect: line is specific: a vague expectation still buys a vague assertion. And it guards the assertion, not the whole mechanical contract, so a debugger that swallows a failure in a retry wrapper or swaps a UI step for an API shortcut is not something the assertion check alone will catch. What it does shut is the most common way agentic fixers cheat: optimizing for make-it-pass by asserting less.

Every path out of this system is bounded and named. Three drafts, two repairs, one re-review. When it cannot produce a spec it can stand behind, you get a PipelineError carrying the evidence, the judge's problems or the Playwright tail or the NOFIX reason, not a weakened green. The happy path is three model calls; the worst case is ten and then it stops. There is no unbounded agent loop anywhere in it.

Generated is not the same as trusted

Nothing a model wrote is trusted on its word. All of it lands in tests/generated/, a gitignored scratch directory that your normal test run, your typecheck, and your linter never look at. A human reads the diff, and, more to the point, watches it run: npm run review opens the scratch suite in Playwright's UI mode, where you step through each test and see what every locator actually matched and whether the primary assertion lines up with what is on screen. If it holds, npm run freeze moves the file into the baseline. That move is the single step in the whole system with no automation behind it, and it stays that way.

The model never touches CI

Generation is a dev-time activity. Once a spec is frozen, CI never calls a model again. It runs plain deterministic Playwright and a mutation gate. No agent in the build, no token bill per push, no model deciding whether your release is green. Keeping the model in that hot path, the way self-healing CI and in-pipeline test generation do, buys you a class of red build you cannot reproduce, because the thing that failed was a sampling temperature, not your app.

Why trust a test a model wrote

Not on its word. The frozen baseline goes through the same gate as the first two posts, playwright-mutation-gate (green tests that can't fail, and tests that check the browser instead of the server). It inverts each spec's primary assertion and re-runs the test. A test that stays green with its key assertion flipped was never testing anything, and the gate fails the run.

On the bundled demo shop the pipeline generated assertions for every feature, and the gate killed all fifteen. No hollow greens in the generated baseline.

15 killed, 0 survived, 0 cannot-mutate, 0 errors
Gate passed: every marked assertion can fail.

The repo also keeps one deliberately weak spec outside the trusted baseline: a receipt greeting drawn from a name the browser cached at checkout rather than from the order the server confirmed. Inversion clears it, because the greeting really is on screen. The gate's --behavior pass corrupts the value in the server response and catches it staying green anyway. That is the exact wrong-source-of-truth trap the second post is about, kept runnable next to the pipeline that would otherwise let it through.

Running it on something real

The demo shop is small and server-rendered, so its whole source fits in one prompt. A real app needs a curated slice of the frontend, the templates and components for the flows under test, pointed at through src/pipeline.ts, plus a hand-written reference spec for the house style and stable data-testid attributes on the elements your flows touch. The agents treat that source as ground truth, which is what keeps them from inventing selectors. There is no black-box mode that works from a rendered DOM instead, and there will not be one until it can keep that guarantee.

What it does not solve

A few things this deliberately leaves on the table. A pipeline whose whole job is to earn trust cannot pretend they are not there.

Curating that frontend slice is real work. On a large app it is the main cost of adoption, more than anything in the pipeline itself, and feed it the wrong components and the writer is back to guessing.

Generated specs match the house style of the one reference spec you hand the writer, but they do not wire themselves into your existing Page Objects or shared helpers. On a mature suite with its own abstractions, that reference spec has to carry them, or you refactor after freezing. The pipeline writes tests that look like your best hand-written one; it does not know your framework.

Frozen specs are ordinary Playwright tests once they land, and they rot like any others. A renamed data-testid or a redesigned flow breaks them, and someone maintains them by hand or regenerates. This solves the blank-page-and-trust problem, not the lifecycle after it.

Test data, auth, and seeded state are on you. The feature files describe user actions; getting the app into the state to run them is fixtures you write, the same as any E2E suite.

And the deepest check is stack-shaped. The mutation gate's --behavior pass, the one that proves an assertion is bound to the server rather than to a cached copy, only sees values that cross the wire as text. It bites on server-rendered pages and verbatim APIs and goes quiet on strings a client composes in the browser (the second post has the detail). The pipeline still generates and runs tests against a React app; you just get less of that one guarantee exactly where the values are assembled client-side.

Clone it, run the pipeline against the demo shop, and watch a spec earn its way into the baseline: ai-qa-pipeline.

I build pipelines like this for teams. If your suite is green and you are not sure what that buys you, that is the conversation I have at automation.dmitriiev.dev.

Your green tests are lying

Vladyslav Dmitriiev — Thu, 16 Jul 2026 15:03:58 +0000

CI goes green, the PR merges, everyone moves on. The whole ritual rests on one assumption nobody checks: that the test would have gone red if the feature were broken. For a surprising number of tests, that assumption is false. They pass because they cannot fail.

I maintain a production Playwright suite, and I wanted an actual answer instead of a feeling. So I took every test's main assertion, inverted it, and re-ran the suite. If the original assertion is doing real work, the inverted version must fail. If the test stays green with its own assertion flipped, the assertion is hollow and the green checkmark has been decorative the whole time.

Before the results, here's what hollow looks like in practice. None of these are exotic. All of them pass in CI every day.

The floating assertion. Playwright's web-first assertions are async. Drop the await and the expectation becomes a promise nobody looks at:

test('shows confirmation toast', async ({ page }) => {
  await page.getByRole('button', { name: 'Save' }).click();
  expect(page.locator('.toast')).toBeVisible(); // no await
});

The test finishes before the assertion resolves. Delete the toast from the app and this still passes. Recent Playwright versions flag some of these with no-floating-promises-style lint rules, but only if the lint rule is on, and in most suites I've seen it isn't.

The negative assertion on a selector that never matched. Asserting absence is a footgun, because absence is also what you get when your selector is wrong:

await expect(page.locator('#eror-banner')).not.toBeVisible();

The typo means this element has never existed on any page, so the assertion is trivially true forever. The error banner can be up in the user's face and the test agrees everything is fine.

The guarded assertion. Someone got tired of a flaky check and wrapped it:

if (await banner.isVisible()) {
  await expect(banner).toContainText('Payment failed');
}

Now the assertion runs only when it would pass. When the banner doesn't render at all, which is the actual bug, the whole block is skipped and the test is green.

The test that isn't running. test.skip(true, 'flaky, revisit later') from eight months ago. The test file exists, the test count in the report includes it, and it has executed zero times since the commit.

Notice what these four have in common: only the first is a lint problem. A dropped await is visible in the syntax, and no-floating-promises will catch it if someone turned it on. The other three are perfectly well-formed code. No static tool can know that #eror-banner has never matched anything, that a guard condition never fires, or that a skip from last winter was supposed to be temporary. The only way to expose them is to run the test and demand that it fail.

Why this is getting worse right now

Hollow tests aren't new, but AI test generation industrialized them. An LLM will happily produce a spec that compiles, follows your fixtures, uses idiomatic locators, and asserts something trivially true, or asserts against a field that doesn't exist behind a guard that never fires. It looks like coverage. The industry has started calling it coverage theater, and the numbers back the name: in Mabl's 2026 survey (n=996), teams report spending around a fifth of the week manually auditing AI-written tests, while 35% of production bugs are still found by customers first.

Manual audit doesn't scale against a generator. If the machine writes tests faster than you can read them, the review has to be a machine too.

Inverting assertions as a gate

The check I described is mutation testing, but aimed at a different layer than usual. Stryker and friends mutate your application code and ask whether the tests notice. That's the right tool for unit suites. For E2E it's usually impractical: you'd be rebuilding and redeploying the app per mutant. Flipping it around is cheap: mutate the assertion and ask whether the test notices. One inversion per test, no app rebuild, and the runtime cost is a re-run of each test.

I packaged the workflow as playwright-mutation-gate. You mark the one assertion each test exists for:

await expect(page.getByRole('heading', { name: 'Order confirmed' })).toBeVisible(); // @primary-assert

The gate copies the spec, inverts the marked assertion (toBeVisible() becomes not.toBeVisible() and vice versa), and runs each mutation through npx playwright test. Every marked assertion gets a verdict. Killed means the mutated test failed: the assertion is alive. Survived means it stayed green with its own assertion flipped: hollow, and the gate exits 1. Cannot-mutate means the line resisted inversion, and the gate reports why instead of silently passing. Tests with no marker also fail the gate by default, because an unmarked test is a test whose main claim nobody has stated.

The cost is easy to reason about: one Playwright run per marked assertion, so the gate takes roughly as long as your suite times one. My 64 mutations ran against a live staging environment as a background job: noticeable, and clearly not something to bolt onto every commit. Treat it like any expensive suite: feed it only the spec files a PR touched (the CLI takes file paths, so git diff slots in naturally), shard it through --playwright-args, or run the full sweep nightly. A hollow assertion doesn't get less hollow between merges; catching it a day later is fine.

The "one primary assertion per test" constraint is doing double duty here. It's what makes the mutation tractable, and it's a discipline worth having anyway: if you can't point at the single line a test exists to check, the test is probably asserting incidentals.

What the suite confessed

Running the gate across the full suite: 16 spec files, 64 marked assertions. Verdict: 63 killed, 0 survived, 1 cannot-mutate.

The 63 are the good news, and genuinely useful news: I can now say the suite's greens are load-bearing, instead of assuming it. A flaky test even got a fair trial along the way. The gate counts a mutation as killed if the inverted assertion failed at least once, so retries can't launder a hollow assertion into a survivor.

The one cannot-mutate was the interesting find. It pointed at a spec covering location-dependent search results, whose entire describe block sat under test.skip(true, ...). The skip was legitimate when it was written: the staging proxy rewrites the forwarded-IP header the feature reads, so the behavior can't be exercised in that environment. The comment promised a revisit. The revisit never came, and for months nothing in the CI output distinguished that file from a healthy one.

The gate flagged it through simple mechanics. A skipped test never executes, so there is no way to observe its inverted assertion fail. Rather than guess, the gate refuses to issue a verdict and says why. That refusal is the point. A verdict of "I could not check this" is information; a green checkmark over a test that never ran is misinformation.

This was a suite in good shape: 63 of 64 assertions just proved they can go red, about as clean a bill of health as E2E coverage gets. And it still contained an entire feature with zero automated coverage, invisible in every green run for months. The blind spot didn't survive because anyone was careless; it survived because no human rereads skipped specs, and nothing in the toolchain is responsible for them. If a well-kept suite hides one of these, I don't want to guess what an average one hides.

What this does not catch

The gate answers one question: can this assertion fail at all. It does not judge whether the assertion checks the right thing. Assert on a page heading that appears on every page of your app, and the inversion will dutifully fail, the mutation is killed, and the test still proves nothing about your feature. Weak assertions are a different disease from hollow ones. Catching them takes code review, or Stryker-style mutation of the application itself where you can afford it.

A kill is also slightly generous by construction. Playwright actions carry implicit assertions (click() fails on its own if the button never appears), so a mutated test can die before it ever reaches the flipped assertion, or die of ordinary flakiness on that run. A kill proves the test is capable of failing, not that the marked line is what failed. The asymmetry works in your favor, though: survived has no such excuse. A test that stays green with its own primary assertion inverted is hollow, full stop, and that's the verdict the gate is built around.

There are mechanical limits too: one marked assertion per test, and multi-line expect chains report as cannot-mutate rather than getting clever. The README's Limitations section is honest about all of this, because a gate you can't trust to report its own blind spots would be recreating the exact problem it exists to solve.

Try it on your suite

The gate is MIT-licensed, on GitHub and npm. Mark your primary assertions, run

npx playwright-mutation-gate tests/*.spec.ts

and read the table. My suite was in good shape and still had a dead spec hiding in the green. On a suite that has never been through a check like this, expect the first run to surface a percent or two of assertions that can't fail, and, if your codebase is anything like mine, at least one spec that quietly stopped running. The only way to know which checkmarks you can trust is to make every test prove, once, that it knows how to go red.