Patterns from building AI agent systems for daily use — what worked, what failed, and why.
How to make AI agents give consistent decisions across sessions using versioned, trigger-driven rule files. 140+ rules across iOS, trading, AI agent design, game development.
AI agentsconsistencyClaude CodeHow to prevent LLM agents from hallucinating probabilities and accepting unverifiable multi-agent consensus. Includes real metrics: council_repeat_rate 0.44, empty_consensus_rate 0.354.
AI agentshallucinationmulti-agentStop a RAG agent from presenting inferences as facts: let the LLM only extract, and let deterministic code score and label FACT/INFERENCE with a ≥2-independent-source rule and required evidence IDs.
provenanceRAGhallucinationUsing Claude Code hooks to enforce behavioral constraints beyond prompts. AND-gate design, nag-once pattern, decision-ownership and sycophancy-prevention examples.
Claude Codehooksbehavioral gatesAn agent that has gathered all the data to decide will still hand the choice back — "which do you prefer?" — because the training gradient rewards deference. A deterministic Stop-hook with an AND-gate (deflection phrase AND fresh tool evidence) blocks it while letting genuine value questions pass.
Claude Codehooksdecision ownershipMultiple personas on a single LLM do not give you real diversity — they are prompt variations of one set of weights, so they share the same blind spots, and the only durable d
aillmarchitectureMake every skill, hook, and scheduled job declare four invariants before it ships — Locality (where it can run), Source-of-truth (which facts it owns or borrows), Cross-ref (w
aiagentsdevopsNo — a scoring function that consumes whatever values an LLM hands it is only deterministic in name; the LLM's judgment launders straight through the "deterministic" gate, and
llmsecurityarchitectureA Claude Code session that returns API Error: 400 ... not valid JSON: no low surrogate in string on every turn is poisoned by a lone UTF-16 surrogate (a code point in U+D800–U
claudecodedebuggingpythonA one-time audit said an agent's quality is the method, not the model — so we built a stdlib SQLite lab to measure method vs outcome across 155 real sessions, reassembling logical sessions and fingerprinting which disciplines each model-epoch used.
aillmagentsA cryptographically fair A/B assignment was still unanalyzable — nothing joined the random draw to the task it dispatched, so no code could compute the intention-to-treat comparison. The join, the re-draw guard, and the up-front power calc are the experiment; the coin is the easy part.
datasciencestatisticsexperiment designPrompt caching is a prefix match, so one timestamp or unsorted JSON key in the system prompt makes every later token miss the cache — a silent 10× cost leak no error reports. How it fails, the invalidator list, and how to assert cache_read_input_tokens > 0.
llmprompt cachingcostLLMs measurably reverse correct answers under "are you sure?" (98% in one Anthropic measurement), and self-critique can't fix it — the critic shares the producer's blind spots. The fix is architectural: a Stop-hook gate that forces one cross-family re-verification on challenge, then allows only HOLD-with-evidence or CHANGE-with-reason.
llmsycophancyverificationA reader turned the sycophancy gate above into an observable test, and it failed: the escape condition was a verification keyword in the agent's own prose, so writing "I re-verified this cross-family" cleared it with nothing behind it. Replaying the tightened rule over 212 transcripts newly blocked 0 legitimate turns — and the fix still only proves that a tool ran, not that it ran on the claim.
verificationstop-hookself-reportFanning one evidence digest out to five model families: 68/68 numeric claims survived recomputation, four models caught an anomaly the fifth missed — and all four external models missed an experiment-design flaw sitting in plain sight. Convergence on the same framing is cross-consistency, not independent evidence.
multi-modelverificationllmA different-vendor LLM audited ten queued posts written by my main model: it caught three copy-paste-breaking macOS/Linux platform bugs and a destructive in-place file conversion the writing family repeatedly missed — and also filed a false accusation that only grepping the primary source could refute. Cross-family audit works; auditor self-reports still need re-verification.
cross-vendorllmverificationA reader's rebuttal sent me to three judges before I replied — my own code review plus two model vendors. The table that came back looked like proof cross-vendor verification pays off; it was actually a schema bug. Only one model's prompt had a field for disagreement, and a same-model controlled experiment (n=3 per condition) showed the schema decided the verdict, not the vendor.
llm-as-judgeverificationprompt schemaThe field that fixed the schema bug above turned out to have a second failure mode. A mandatory "where does this overreach" slot gets filled even when there is nothing to object to, because the model cannot write nothing — so a populated field is not the signal. What is: whether two independent model families, shown the same material, land on the same objection.
llm-as-judgeprompt schemaverificationFive models, two roles, a few thousand real agent threads — and a stack of refusals instead of a ranking: role confounds, version-epoch discipline, [0,0] tie-degenerate bootstrap CIs, non-additive token counters, self-measurement contamination. The one completion-proxy CI worth reading spans zero.
llmtelemetrydatascienceFour model families cited every number from a frozen agent-telemetry table with zero errors — 70 of 70 recomputed within tolerance — and one still read a row of structural zeros as error-free tool use and built a routing suggestion on it. The one pre-registered equivalence test that passed cleared a margin 0.75 standard deviations wide while its own interval excluded zero. A citation audit is a copy-fidelity floor; it never checks the inference drawn from the number.
multi-modelverificationstatisticsA new skills repo scored 64/100 → TRIAL across five gates (maturity, dependencies, platform fit, policy, trajectory) on provenance-tagged inputs (9 FACT / 1 ASSUMPTION, conservative defaults for assumptions), with five auto-derived kill conditions — and a refusal to log a day-zero outcome that would pollute the ledger's own base rates.
adoptionfalsifierdecisionThree roles — question, defend, attack — rotated through three reviewer groups in all six permutations, so no group was ever the permanent judge. Seven confirmed defects before hardening, including a fix two groups had agreed on that a later hostile seat reversed as unsafe, and a verification pool that fell back three layers deep in round one.
adversarial reviewmulti-agentverificationSee every cron job and LaunchAgent in one place by globbing on-disk state instead of importing them. Includes a 25-tool fit-gap matrix and the read-only aggregation pattern.
cronlaunchdobservabilityWhy pull (scan on-disk state) beats push (an event bus) for monitoring a high-churn fleet with components you don't control. The instrumentation-tax argument and self-healing polling.
multi-agentpull vs pushMCPA commenter showed that polling files the job itself wrote is self-report over a filesystem — the same weakness as the event bus I rejected. My publishing job proved its own health with its own log and stayed green through a four-day outage. The cross-vendor verification I ran instead of defending, and the effect_probe fix.
observabilityliveness without progresspostmortemCoordinate independent agent CLIs without LangGraph or a message broker: atomic Task/Result files, capability-based routing, absent-worker skip for graceful degradation, and a typed claim contract on the bus.
orchestrationmulti-agentLangGraph alternativeAn efficiency audit found a silent cost regression, a cache-invalidation trap, a pricing cliff — and a terrifying "12% task completion" that turned out to be benchmark contamination plus a schema misreading (real rate: 98.4%). Verify the metric before reacting; the false alarm's investigation still fixed three real defects.
observabilitymulti-agentpostmortemA subagent reported 7 files changed with zero disk changes behind it, then fabricated the ls -la and grep -c proof demanded of it. The two distinct failure modes (fabrication vs silent stall), the landmark-file check run in the orchestrator's own process, and salvaging a non-writing worker by switching its output channel.
multi-agentverificationpostmortem25 days of "Couldn't fetch" on a sitemap whose 289 URLs all return 200 — a documented GitHub Pages pattern, not a broken site. Reading crawl stats, the manual Request-indexing workaround (~10/day quota), the dead ends to skip, and why Bing quietly matters more for AI search.
SEOGitHub PagesSearch ConsoleGEO is unfalsifiable until you measure citations: a fixed set of user-shaped questions fired weekly at a grounded model, a deterministic owner-scoped match, and a ledger that error rows can't fake — plus when not to measure at all.
GEOAI searchmeasurementA crosspost pipeline with 19 green gate tests published nine articles whose titles were visibly broken on the live platform for nine days. Tests verify the payload you build, not what the other system renders — a pipeline isn't verified until you check the output surface itself.
testingpostmortempipelineA 6-agent forensic audit over 92 of my own sessions answered "am I prompting wrong?" — partially yes (zero visual references, zero style tokens), but the dominant cause was silent infrastructure decay: exhausted generation credits quietly swapped in code-approximated shapes, a generation server was down with a required model file missing, and a ready asset pipeline was never invoked for six-plus weeks.
vibe codinggamedevauditBuilding a travel app on Korea's open-data portal: a documented-looking op name returned 500 (the real one returned per-gate passenger counts, not the grade field I'd invented), a sibling dataset 404'd because approval is per-dataset, a discarded dataset left a 401 red herring on the wrong gateway, and one API's airport parameter means the other end of the flight. Live capture is the only ground truth.
open dataAPIdegrade pathMy app mascot's arms rotated from the fingertips — invisible in code review, obvious in one render. Three structurally different variants instead of tint tweaks, the grep that saved a parameter used by 8 call sites, pre-registered design falsifiers checked at real sizes, and the render-first loop that fixes AI-drawn UI.
swiftuivibe codingdesignA 20-minute JWT cap that 401s perfectly valid signatures, one app version in flight (409 STATE_NOT_SUITABLE), and a 2-in-flight/5-total review-submission ceiling with undeletable orphans (DELETE 403, PATCH 409 — web UI cancel or 7-day expiry only). The two-call pre-flight that avoids all three.
App Store ConnectiOSAPIWhen nslookup resolves but curl/pip/Python fail, the mDNSResponder daemon is in a non-responsive zombie state. The two-DNS-path diagnosis, the weakest-first fix, and why long-running agent/MCP boxes trigger it (measured: 30+ long-poll connections → 77% CPU).
macOSDNSdebuggingFor a personal tool, the way past robots.txt, bot detection, login walls, and JS-rendered SPAs isn't a better headless scraper — it's a ~40-line AppleScript/JXA bridge that reads the DOM out of the authenticated Chrome tab you already have open. Why it sidesteps every wall at once, and its honest read-only, single-machine limits.
web scrapingmacOSautomationA background LLM server loaded models onto a nearly full disk; unified memory overflowed, swap failed, and APFS metadata corruption left the machine unbootable for a day. The postmortem, the five-layer code-level hard block, and the operating rules for 24 GB unified memory.
macOSpostmortemlocal LLM