DEV Community: Shipwright

Catch prompt regressions in CI before your users do

Shipwright — Mon, 09 Nov 2026 15:03:02 +0000

Change a prompt to fix one case and you can silently break five others. Nothing throws, no test goes red, the app still returns confident-looking text — and the regression ships to your biggest customer. If you edit prompts on vibes, you will ship one of these eventually. The fix is to treat prompts like the code they are: version them, and run an eval suite against the real production path before every change.

An eval suite points at your actual prompt

The trap with prompt tests is grading a copy of the prompt that drifts from what production runs. So a suite here calls the real helper — the same function your route calls — and grades its output. The shape is deliberately small:

export interface EvalSuite {
  name: string;
  // The production code path under test — call your real prompt/helper here.
  run: (input: string) => Promise<string>;
  cases: EvalCase[];
}

Because the suite runs the production code path, a prompt edit is exercised the moment you run the suite — no separate fixture to keep in sync.

Two kinds of grader: cheap-and-exact, then fuzzy

Most of what you want to assert is format and policy: does the output contain the disclaimer, does it avoid a banned phrase, does it match a shape. Those are deterministic string checks — instant and free. Reach for the model only for things a string cannot capture ("politely declines without inventing a policy"), and write the rubric as a checkable statement, not a vibe:

export const llmJudge =
  (rubric: string, model: ModelId = "claude-opus-4-8"): Grader =>
  async (output) => {
    const { data } = await generateObject({
      model,
      schema: verdictSchema, // { pass: boolean, reasoning: string }
      system:
        "You are grading the output of an AI product feature against a rubric. " +
        "Judge strictly: pass only if the output clearly satisfies the rubric.",
      messages: [{ role: "user", content: rubric + "\n\n" + output }],
    });
    return { pass: data.pass, detail: data.reasoning };
  };

The judge returns a typed verdict via structured output, so a grader is just a function returning pass plus a reason — deterministic and LLM graders compose in the same list.

Make CI fail, not just print

An eval that logs a warning gets ignored. The runner returns a single boolean — true only if every case passed — so have your suite script exit nonzero on false and a red eval blocks the merge exactly like a failing unit test:

const ok = await runSuite(suite);
process.exit(ok ? 0 : 1);

Pair that with a versioned prompt registry: each prompt carries a version — log it with every eval run, so when a suite goes red you can tell whether the cause was a prompt edit or a model upgrade — the same deterministic registry that makes prompt caching actually work.

Things that bite

Evals cost real tokens. They make live API calls, and the LLM judge is itself a model call. Keep deterministic graders first and gate them so the cheap checks fail fast before you pay for a judge.
A vague rubric grades vaguely. "Is it good" gives you a coin flip. "Declines the refund and offers to escalate, without inventing an exception" is checkable.
Judge strictly. A lenient judge passes everything and tells you nothing; the system prompt here forces a strict pass bar on purpose.
Version the prompt, or you are debugging blind. Without a version stamped on the run, a regression could be the prompt, the model, or the weather.

Shipwright — the Next.js and Claude starter kit this blog documents — ships this eval harness — composable graders, an LLM judge, and a runner that returns a pass/fail you can gate CI on — alongside the versioned prompt registry and per-request cost metering, so quality regressions can surface in a pull request instead of a support ticket. You can try the prompts it grades in the live demo.

How to keep an AI free tier from bankrupting you: two ceilings and a cheap model

Shipwright — Sun, 08 Nov 2026 15:03:15 +0000

The fastest way to lose money on an AI product is a generous free tier pointed at an expensive model. Free users have no reason to be frugal and every reason to test your limits, so if your free plan can reach your best model with no ceiling, your worst-case cost per free user is unbounded. A free tier is supposed to be top-of-funnel, not a faucet into your API account. Here is how I make one safe with two independent ceilings and one cheap model.

Ceiling one: a monthly request quota

Every AI route checks the user's usage against their plan limit before doing any model work, and returns HTTP 402 when they are over. That check is the whole enforcement point:

export async function checkQuota(userId: string): Promise<QuotaCheck> {
  const plan = await getUserPlan(userId);
  const { requests } = await getMonthlyUsage(userId);
  return {
    ok: requests < plan.monthlyRequestLimit,
    plan,
    used: requests,
    limit: plan.monthlyRequestLimit,
  };
}

The free plan's limit is 25 requests per calendar month. The route calls this first; when ok is false it answers 402 instead of calling the model. The count comes from the same per-request usage table that drives billing, so the number the quota reads is the number you actually spent — I wrote up that schema in tracking LLM costs per user.

Ceiling two: a per-minute rate limit

A monthly quota bounds the total, but not the burst. Without a second ceiling, a user could spend the entire month's allowance in ten seconds and hammer your provider with concurrent requests. A separate per-minute rate limiter caps that burst. They defend against different attacks — one bounds the month, the other bounds the second — so you need both, not either.

Gate the model, not just the volume

Volume caps alone are not enough, because 25 requests on your most expensive model is a completely different bill than 25 on your cheapest. So the plan also declares which models it may use, and requests for anything else are clamped to the plan's default:

free: {
  monthlyRequestLimit: 25,
  allowedModels: ["claude-haiku-4-5"],
  defaultModel: "claude-haiku-4-5",
},

A free user who crafts a request asking for Opus does not get Opus — the plan clamps it to Haiku. Model access is a billing decision, enforced in one place, not an if-statement scattered across routes.

The math that makes it safe

Haiku 4.5 bills at 1 dollar per million input tokens and 5 per million output. A chatty free request of roughly 1,000 input and 500 output tokens costs about $0.0035. Twenty-five of them is about nine cents per free user per month. Break-even sits well under 1% conversion — a free tier at that cost is a marketing channel, not a liability.

Now run the same 25 requests against Opus at 25 dollars per million output, with longer answers, and the per-user cost jumps 20 to 50 times — and without the quota, it has no ceiling at all. That gap is the entire argument for gating both volume and model.

Things that bite

Fail-open versus fail-closed is a real decision. This kit fails open when no database is configured, so the public demo stays usable without setup. In production, a metering outage that silently disables the quota is a spend leak — decide on purpose which way it fails.
Anonymous users still need an identity. The public demo attributes usage by IP address, so unauthenticated traffic is capped too. Skip that and "free" quietly means "unlimited to anyone not logged in."
A rate limit is not a quota. One bounds the burst, the other the month. Shipping only one leaves the other door open.
Cap on requests, keep the tokens. Users understand "25 requests"; bill and reason about margin on the token and cost columns underneath.

Shipwright — the Next.js and Claude starter kit this blog documents — ships this whole safety layer wired together: per-plan quotas, model gating, the per-minute limiter, and the metering that feeds them. You can watch the free tier's cost accrue in real time in the live demo.

Tracking LLM costs per user: the schema that billing can actually trust

Shipwright — Sat, 07 Nov 2026 15:03:21 +0000

Your model provider tells you what the whole account spent. It does not tell you that one free-tier user generated a third of it. Until cost is attributed per user, every pricing decision — plan limits, free-tier caps, whether that enterprise deal is even profitable — is a guess. Here is the schema I use, and the two decisions in it that took real debugging to learn.

One row per model request

Every request writes exactly one row, at the moment usage is known — from the stream's finish hook, or right after a non-streaming call returns:

export async function recordUsage(args: RecordUsageArgs): Promise<void> {
  await db()
    .insert(usageEvents)
    .values({
      userId: args.userId,
      model: args.model,
      route: args.route, // which feature spent the tokens, e.g. "chat"
      inputTokens: args.usage.inputTokens,
      outputTokens: args.usage.outputTokens,
      cacheReadTokens: args.usage.cacheReadTokens,
      cacheCreationTokens: args.usage.cacheCreationTokens,
      costMicroUsd: Math.round(costUsd(args.model, args.usage) * 1_000_000),
    });
}

The column choices matter more than they look:

route — a month in, you will want to know which feature burns the money, not just which user. Adding the column later means backfilling history you no longer have.
The two cache columns — cache reads bill at roughly a tenth of the input rate and writes at 1.25x, so a request with heavy cache reads costs a fraction of its naive token count. Without these columns your numbers drift from the real invoice, and you cannot see whether prompt caching is actually saving you anything.
userId as plain text, not a foreign key — deliberate. The public demo attributes anonymous traffic as ip:<addr>; a hard FK to the users table would reject those rows. Once every AI route requires auth, tighten it to a real reference.

Integer micro-USD. Never floats.

The cost column is a bigint holding millionths of a dollar: a request that costs $0.004173 is stored as 4173. One request priced as a float looks fine. A month of requests summed as floats accumulates binary rounding error, and now your dashboard, your quota check, and your Stripe invoice each show a slightly different number — three sources of truth is zero sources of truth.

Integers sum exactly. Convert to dollars once, at render time, and let every consumer read the same column.

The rollup that quotas, dashboards, and billing all share

Per-user monthly usage is one aggregate over that table:

const [row] = await db()
  .select({
    requests: sql`count(*)`,
    inputTokens: sql`coalesce(sum(input_tokens), 0)`,
    outputTokens: sql`coalesce(sum(output_tokens), 0)`,
    costMicroUsd: sql`coalesce(sum(cost_micro_usd), 0)`,
  })
  .from(usageEvents)
  .where(and(eq(usageEvents.userId, userId), gte(usageEvents.createdAt, monthStart)));

This single number is load-bearing three ways: the quota middleware compares it against the plan limit and returns 402 when exceeded, the dashboard renders it to the user, and usage-based billing reports it to Stripe. Because all three read the same rows, they cannot disagree — which is the entire point of the design.

An index on (user_id, created_at) keeps the rollup fast; it is the only query pattern the table needs to serve at request time.

Things that bite

Decide your no-database behavior on purpose. This kit fails open — recordUsage warns and returns when no DATABASE_URL is set, so the demo runs with zero config. Fine for a starter, wrong for production billing: there, failing to record should be loud.
Month boundaries are UTC. The rollup starts at the first of the month, UTC. Pick a convention and keep it consistent with your billing period, or users get quota resets that disagree with their invoice.
Meter requests, not just tokens. Plans here cap monthly request volume, because "requests" is a number users can reason about. The token data stays for margin analysis either way.
IP attribution is a stopgap. Shared offices and mobile carriers pool many humans behind one address. Good enough to cap an anonymous demo, not good enough to bill.

You can watch this pipeline run in the live demo: every message prints its model, token counts, cache activity, and exact cost — the same numbers that land in the table. Shipwright, the Next.js + Claude starter kit this blog documents, ships the whole loop — schema, metering, quotas, and the Stripe wiring — already connected.

Claude prompt caching: the break-even math and the mistake that turns it off

Shipwright — Fri, 06 Nov 2026 15:04:11 +0000

Prompt caching is the biggest cost lever most Claude apps leave on the table — and the only one that can also make your bill worse. The pricing has a shape people miss: cache reads are cheap, but cache writes carry a surcharge. If you cache things that never get re-read, you are paying extra for the privilege.

Here is the actual math, using the numbers my metering layer bills against.

The four line items on every request

Every Claude request can produce up to four token counts, each at a different rate. This is the pricing function from my model registry — the same one the dashboard, quotas, and billing all read:

/**
 * Cost of a single request in USD.
 * Cache reads bill at ~0.1x the input rate; cache writes at 1.25x (5-minute TTL).
 */
export function costUsd(model: ModelId, usage: TokenUsage): number {
  const m = MODELS[model];
  return (
    (usage.inputTokens * m.inputPerMTok +
      usage.outputTokens * m.outputPerMTok +
      usage.cacheReadTokens * m.inputPerMTok * 0.1 +
      usage.cacheCreationTokens * m.inputPerMTok * 1.25) /
    1_000_000
  );
}

Two things to internalize:

Reads are a 90% discount. A cached token bills at roughly a tenth of the normal input rate.
Writes are a 25% surcharge. The first time a prefix is cached, those tokens cost 1.25x normal input. Caching is an investment, and investments can lose.

Break-even: one re-read pays for the write

Take a 2,000-token system prompt on Claude Opus 4.8 (5 dollars per million input tokens):

Uncached, that prefix costs 1.0 cent per request.
Writing it to the cache costs 1.25 cents — once.
Each subsequent read costs 0.1 cents.

So for N requests inside the cache window: uncached spend is 1.0N cents, cached spend is 1.25 + 0.1(N-1). At N=1 you lost a quarter of a cent. At N=2 you are already 33% ahead. By N=10 the prefix costs 2.15 cents instead of 10.

The general rule falls out of the multipliers: the write premium is 0.25x, each re-read saves 0.9x — so a single re-read within the TTL pays for the write three times over. In a chat product, where every turn resends the same system prompt and history prefix, that condition is nearly always met. Caching a one-off batch job that never repeats a prefix, though, is a pure 25% surcharge.

The mistake: one dynamic value at the top

Cache matching is a byte-exact prefix match. A single dynamic value early in your system prompt — a timestamp, a request ID, a "today is Wednesday" — changes the bytes, misses the cache, and triggers a fresh 1.25x write on every single request. That is the worst configuration available: you pay the write surcharge every time and never collect a read.

This is why my prompt registry keeps system prompts deterministic, with a comment that has survived every refactor:

/**
 * Keep system prompts deterministic (no timestamps, no per-request IDs):
 * prompt caching is a byte-exact prefix match, and a single dynamic value at
 * the top of the prompt disables caching for everything after it.
 */

Anything per-request — the user name, the date, retrieved context — belongs after the stable prefix, or in the user message, never at the top of the system prompt.

If you cannot see the hit rate, assume zero

My usage table records cache reads and writes as their own columns on every request, because the failure mode here is silent: nothing breaks when caching stops working, the bill just quietly grows. If cache reads are near zero in production while writes keep happening, a dynamic byte crept into your prefix and nobody noticed.

You can watch this live in the demo chat — the under-the-hood panel prints each request with cache reads at 0.1x and writes at 1.25x as separate line items, next to the exact cost. The same discipline as aborting streams when the client disconnects: measure the money path first, then the savings become visible instead of theoretical.

Things that bite

The TTL is five minutes. Caching pays inside active sessions and bursty traffic; a user who returns after lunch starts with a fresh write. Do not model savings off your total request count — model them off requests that arrive within minutes of each other.
Writes without reads are a surcharge, not a win. Audit any endpoint that caches but rarely repeats a prefix.
Very short prompts may not cache at all. Models enforce a minimum cacheable prefix length; below it, your cache_control marker is silently ignored. Check the current provider docs for the threshold rather than assuming.
The ratios hold across models, the absolutes do not. On Haiku the same 2,000-token prefix costs 0.2 cents uncached — caching still helps, but the cents at stake are 5x smaller than on Opus. Spend optimization effort where the tokens are expensive.

Shipwright — the Next.js starter kit this blog documents — ships this whole loop wired together: deterministic versioned prompts, cache-aware cost attribution per request in integer micro-USD, and a live panel that shows the math on every message.

Your Next.js AI streaming route keeps billing you after the user leaves

Shipwright — Thu, 05 Nov 2026 22:54:19 +0000

An AI chat endpoint feels finished the moment tokens start streaming into the browser. Then your first real bill arrives, and the token count is higher than what your users actually read. The usual reason: responses that keep generating after the user has already closed the tab.

Where the money leaks

In the Next.js App Router, a streaming route opens a connection to the model and pipes tokens back to the browser. If the user navigates away or closes the tab mid-response, the browser drops its connection — but the upstream request to the model keeps generating to completion. You pay for every token it produces, and nobody ever sees them. On long answers with an impatient audience, that waste adds up fast.

The fix: stop generating when the browser disconnects

When you build the SSE stream yourself with a ReadableStream, its cancel() callback fires the moment the client connection drops. That's your hook to abort the upstream model request instead of letting it run to completion:

export function POST(req: Request) {
  const upstream = anthropic.messages.stream({ model, max_tokens, messages });

  const body = new ReadableStream({
    async start(controller) {
      for await (const event of upstream) controller.enqueue(encode(event));
      controller.close();
    },
    cancel() {
      // The browser went away — stop generating (and paying).
      upstream.abort();
    },
  });

  return new Response(body, {
    headers: {
      "content-type": "text/event-stream",
      "cache-control": "no-cache, no-transform",
    },
  });
}

That one method turns "streaming works in the demo" into "streaming doesn't leak money in production."

Three things that bite you next

"no-transform" matters. Some proxies buffer SSE, so your "stream" arrives as a single blob at the very end. The Cache-Control no-transform hint tells them to leave it alone.
Raise the route's max duration. Long generations outlive the platform's default timeout and get cut off mid-answer; bump maxDuration on the route.
Meter what you spend. Cancelling saves money you can't see unless you measure it. Record input, output, and cache tokens per request, priced per model, so the savings show up in your numbers instead of your imagination. I wrote up the exact schema I use for that here.

Why it belongs in the foundation

None of this is hard. But it's the kind of thing you only discover you needed after it has already cost you — which is exactly why it should live in the foundation of an AI app, not the backlog.

I package this pattern — along with per-request cost metering, typed tool use, versioned prompts with an eval harness, and plan-gated quotas — into a Next.js 16 + Claude starter kit called Shipwright. The live demo shows the exact cost of every request as you chat.