DEV Community: cilly

Treating Firestore as a public cache

cilly — Thu, 23 Jul 2026 00:03:01 +0000

Using Firestore as "the app's primary database" is easy at first. Writes and reads complete in one place, and onSnapshot gives you realtime push for free. But as a service grows, keeping Firestore as the SoT (source of truth) exposes some fatal constraints.

Where SoT-Firestore hurts

Weak transaction boundaries, missing complex queries, the cost structure, the difficulty of migrating away. The harshest one: you cannot narrow related updates with a WHERE. For the use case "update a set of documents matching a condition, consistently, in one go," Firestore is structurally weak.

Redefining it as a public cache

In one project, I made Cloud SQL the SoT and treated Firestore as a "read-optimized projection." Writes go through the backend (Next.js / Cloud Run), and the SoT (Cloud SQL via Data Connect) and Firestore are both updated within the same write path. Separately, a Cloud Run reconciliation job runs and checks and repairs consistency against the SoT as authoritative. This reconciliation job is enqueued at the start of the request, before the DB updates. Because it's queued first, even if the write dies halfway, the consistency check always runs afterward and repairs the state. From the client's perspective, Firestore is a rebuildable cache — in the worst case it can be rebuilt from the SoT.

// Writes go through the backend. The backend updates both the SoT
// (Data Connect → Cloud SQL) and Firestore. Clients never write Firestore directly.
await backend.updateEntity({ id, title });

// Changes are pushed back in realtime via Firestore's onSnapshot
unsubscribe = onSnapshot(entityRef, (snap) => render(snap.data()));

Benefits and costs

The benefits are clear. The SoT side (SQL) brings transactional consistency and complex queries; the read side (Firestore) brings freedom in data modeling and realtime push; and consistent syncing with external systems (OpenSearch / BigQuery, etc.) coexists naturally. Since the backend updates Firestore at write time, the frontend gets immediate reflection through its onSnapshot stream. No optimistic-update tricks like "hold the value I just wrote in the UI for a while."

The cost is not latency — it moves to double writes and consistency. The backend writes two places, SoT and Firestore, so if one fails they can diverge (this is not a single distributed transaction). What closes that gap is the Cloud Run reconciliation, verifying and repairing Firestore with the SoT as truth. You can only shrug Firestore off as "a cache that can break and be rebuilt" because that consistency check stands behind it. The operational cost concentrates in this double-write path and the reconciliation route.

Which to choose

The choice between "Firestore as SoT" and "Firestore as cache" comes down to the expected lifespan of the service. A short-lived prototype: the former. Something you intend to grow for years: the latter. And if you're in between — the cost of switching over later is far higher than the cost of treating it as a cache from day one.

Originally published at cilly-yllic.github.io.

Provisioning GCP / Firebase environments from a single settings.yml

cilly — Thu, 23 Jul 2026 00:02:58 +0000

Every time you add a service on GCP and Firebase, the same manual work repeats. Create the project, attach billing, enable APIs, set up the Terraform service account and Workload Identity, configure Firestore / Auth / Storage / App Hosting. Multiply all of that by the number of environments (dev / stg / prd). No matter how carefully you write the runbook, as long as human hands are involved, environments drift.

So I built a platform where dropping a single settings.yml per service provisions everything automatically. The configuration is the source of truth; the infrastructure is nothing more than its projection.

What gets built

Let's start with the "generated" side — the overall picture.

A shared bootstrap project lends permissions, keyless, to per-environment service projects.

The structure has three layers.

Identity & Trust — Terraform Cloud and GitHub Actions operate GCP through Workload Identity Federation (OIDC). No service account key files are ever issued.
Shared bootstrap project — the "platform for the platform": the WIF pool / provider, the Terraform service account, the Cloud Run router described below, and Secret Manager. Created once per organization.
Per-environment service projects — independent GCP projects per environment, like my-service-dev-001. Inside each, Firebase resources (Auth / Firestore / Storage / App Hosting / Data Connect / Functions …) line up according to feature flags.

Holding no keys is what makes this work. Each environment's Terraform SA is impersonated from the bootstrap SA, and that permission only fires through an OIDC token exchange. There is no long-lived credential anywhere that could leak.

How it gets generated

Now the "generating" side. From merging settings.yml to applying against GCP, everything is chained without polling.

One settings file flows in a straight line: GitHub Actions → Terraform Cloud → GCP.

The flow has two stages.

# terraform/settings.yml in the service repo (excerpt)
service: my-service
environments:
  dev-001:
    labels: [tier:dev]
    firebase_platform:
      firebase: true
      firestore: true
      app_hosting:
        - { backend_id: web, location: asia-northeast1 }
      notifications:
        - url: https://hooks.slack.com/services/...  # notify apply results

Action A (project-bootstrap) reads settings.yml, filters target environments by status / labels, and creates GCP projects / SAs / WIF in one batched run.
The run's completion notification (HMAC-signed) is received by a Cloud Run router, which verifies it and fires GitHub's repository_dispatch.
That triggers Action B (firebase-platform), which stands up a workspace per environment and applies the Firebase resources.

Instead of keeping a polling orchestrator running, I chained webhooks: Terraform Cloud notification → Cloud Run → GitHub. Only TFC knows the state, so the result notification to Slack is also sent by TFC — the one party that knows whether the apply succeeded.

Design decisions that pay off

Feature flags decide everything down to APIs and IAM. Write firestore: true in settings.yml and the corresponding API enablement, resource creation, and role grants to the CI SA all cascade from it. Leave it unset and nothing is created (zero side effects). Users never need to memorize which GCP APIs a feature requires.

Configuration ownership stays with the service team. Which features to use is the application's concern, so settings.yml lives in the service repo and the platform side only reads it. The platform never becomes a central registry hoarding every service's configuration.

Deletion has a separate safety net. Removing an environment from the settings normally destroys it, but if its name is listed in retained_envs, it is only removed from state and the GCP resources stay. A config mistake cannot take production down with it.

What I gave up

There are costs. The webhook chain adds one more operational component — the Cloud Run router. You have to think about HMAC secret rotation and a retry path for dropped notifications. Since it spans Terraform Cloud, GitHub Actions, and GCP, failure triage is harder than in a single repository.

Even so, the value of collapsing the cost of adding an environment to "add a few lines to settings.yml and run the Action" is significant. Manual drift disappears structurally, and anyone gets the same environment. When you push infrastructure all the way into "something you write and generate," the operational concern shifts from individual environments to the generating machinery itself.

Where it's published

The platform is open source — a monorepo of Terraform modules and GitHub Actions, free for anyone to use.

Repository — github.com/cilly-yllic/terraform-google-platform
Terraform Registry — registry.terraform.io/modules/cilly-yllic/platform/google

Originally published at cilly-yllic.github.io.

Enforce with config, not attention — scope and permission guardrails for coding agents

cilly — Wed, 22 Jul 2026 12:17:24 +0000

When you put a coding agent (like Claude Code) to work, where it really pays off is work that spans multiple repositories. Writing into one repository while referencing another's implementation. Fixing a shared library while verifying behavior in the repository that consumes it. Updating a generated-artifacts repository while reading the config repository. The agent's speed shines in this kind of "straddling" work far more than in single-feature fixes.

But cross-repository work widens the blast radius just as much. Accidentally editing a repository you only meant to reference. Pushing straight to another repository's main. Touching files unrelated to the goal. The more repositories are involved, the blurrier the boundary of "which ones may I touch right now, and how far am I allowed to go" becomes — and the agent re-reads that boundary from scratch every time, so there is structurally nothing equivalent to human attentiveness. That's why "asking nicely in the prompt to be careful" can't fundamentally prevent accidents.

So I built a mechanism that enforces guardrails through config files instead of attention. The idea is simple: before starting work, declare "the repositories that may be straddled" and "the permission for each," and make everything else technically impossible. The span of the cross-repository work itself gets frozen as configuration up front.

A session = a declaration of scope + permissions

The parent declares scope and permissions and spawns the session. Each straddled repository is handled per its permission — a "read-only repository" structurally cannot be modified.

I call the smallest unit a "session". One session = one unit of work, and at launch two things are fixed:

Scope … which repositories / directories may be touched
Permissions … per target, how far is allowed

Permissions have three levels:

read … reference only
write-local … edit + local commits. No push
write-push … push allowed (including PR creation)

The point is that permissions are held per target. In cross-repository work, repositories of differing trust naturally mix within one session. "Reference the config repo read-only, write-push to the artifacts repo." "Fix the shared library write-local but don't push; use the consumer repo read-only for verification." A session handles combinations like these as one unit of work — so it carries a single declaration (manifest) listing target-level pairs, and scope, permissions, and identity checks all concentrate there.

In fact, this very article was written in such a cross-repository session: referencing the repository the mechanism comes from as read, while writing the text into the article repository as write-push. The guarantee that the read side can never be modified is enforced by config, so there's no fear of breaking the source being referenced.

Build scope as "what isn't opened doesn't exist"

Scope is an allowlist. The list of directories the agent can access enumerates only the allowed targets. Repositories not listed cannot even be referenced. I deliberately avoided symlink tricks so that scope is complete in the config file alone.

The direction matters: you add what can be touched, never subtract from everything. Because the default is closed, any target you forget to configure fails safe (= invisible).

Hard part 1: keep the agent away from its own permission definitions

This is the single most effective design decision. From inside a session, the files defining scope and permissions themselves cannot be modified.

The logic goes like this. When you give an agent read permission, the config file describing "what counts as read" is also just a file. If the agent could edit it, it could notice "removing this deny lets me write" — room to loosen its own guardrails. An entity holding permissions must not be able to rewrite its own permission definition. So permission changes can only happen one level up (the parent that spawns sessions).

There's an implementation trap here, though. What you want to protect is "the permission and scope definition files themselves." If you conclude "just ban Edit / Write entirely," you fail: the agent can no longer write even its own work log. What's needed is not a per-tool ban but a per-path ban. Deny exactly the definition files by name, and keep the session's working area writable.

// What we protect is the session's own permission/scope definitions. Deny exactly those by name.
"deny": [
  "Write(.claude/**)",   // permission definitions (settings)
  "Write(/CLAUDE.md)"    // scope declaration
  // Working areas like session logs stay writable
]

Conversely, read-permission repositories get banned wholesale — since they're read-only, excluding the entire tree from Write / Edit is the straightforward, safe move. The problem appeared where "ban wholesale" collided with "keep my own logs writable."

I actually messed this up once. The session's own working directory was nested under a read-target repository. Banning the read target's whole tree took the session's logs down with it — the agent could no longer write its own log.

What fixed it was not narrowing the deny but physically separating the working area from the scoped targets. Move the session's working directory (logs, scratch notes) outside the protected tree entirely. Then read targets can be banned wholesale with confidence, and the session writes its logs freely. Not shaving down the protected area to make things fit — splitting the ground so that what you protect and where you write never overlap.

Hard part 2: double-lock push

Push is the scariest. Pushing to a read repository; pushing straight to main. Both are hard to undo.

Naively, narrowing the allowlist of push commands stops it. That works well against the agent's careless accidents. But as hardness goes, it's insufficient: an allowlist only checks "is this command shaped like git push", so it can't judge variants that switch the working directory like git -C <other-repo> push, nor which repository or branch the push is aimed at. Matching on command shape cannot protect the destination's permission.

So I made it two layers:

Narrow "what can push" with the allowlist … only sessions holding write-push get a push command at all. Sessions with only read or write-local are never given one.
A pre-execution hook … inspects the command string, derives the push destination's repository path, looks up its permission, and decides.

The hard layer is the second. It looks at the command string about to run, works out the push destination from git -C <path> or the working directory, and checks it against the manifest (path → permission). Pushes to read / write-local paths are all blocked; even on write-push paths, direct pushes to main / master are blocked. If the destination can't be determined, it fails safe and rejects, prompting "specify the target explicitly with git -C <absolute path>". The hook physically enforces the per-destination permission that command-shape allowlists can't reach.

Make sessions resumable

Agent sessions die. If the context vanishes with them, you're back to square one every time, so work logs are kept in two layers:

Snapshot (always current, overwritten) … the one page that says "read this and you can resume": current task, target repositories and working branches, touched files, next move, unresolved decisions
Task log (chronological, append-only) … the history of events and decision rationale

On resume, a launch hook automatically injects the snapshot into context. The agent picks up "from where it left off" without being told anything. If guardrails are the mechanism that prevents accidents, this is the mechanism that prevents losing work.

Where this stands, and what's next

Honestly: this is still personal tooling. It's something I built to put guardrails on my own agent, on my own machine — not a production system.

But the shape itself — "a session = a declaration of scope + permissions" — has no reason to stay personal. I'm considering publishing it as a repository, installable globally, so anyone's machine can cut sessions the same way. As more people use coding agents daily, the need to fix "what the agent may and may not touch" as an external declaration will only grow, and I'd like this to become a shared foundation for it.

When you try to control an agent cleverly, you're tempted to write clever prompts. But what actually stops accidents is usually not cleverness — it's the config that makes them impossible in the first place.

Originally published at cilly-yllic.github.io.

Generating every Firebase layer’s types from a single contract.yml

cilly — Wed, 22 Jul 2026 12:17:21 +0000

Sharing types between frontend and backend is an old problem. A while back it was common to keep the two in separate repositories, so you had no choice but to hand-write the same type definitions in each — that is, duplicate them.

The next stage was private packages. Extracting shared types into a package removes the duplication, but now you have one more repository, and during development you bounce between the frontend, backend, and package repos and editors. Worse, every time you fix a type, nothing reaches the frontend or backend until the package goes through commit → push → version bump → publish. That cycle sits in the middle of every verification loop. Private packages also need separate authentication for CI and deploy pipelines to install them — registry tokens and permissions turn into a surprisingly annoying mechanism. Embedding a shared repo as a git submodule instead just changes the flavor of the chore: every consumer has to advance the referenced commit, and checkout skew happens easily.

What solved this was the monorepo. Shared libraries can be referenced directly within one repository, and frontend and backend can be developed in parallel. If the problem were just "frontend and backend use the same types," the story would end here.

But once you adopt an architecture where Cloud SQL (Data Connect) is the source of truth and Firestore is a read projection (cache) — see Treating Firestore as a public cache (Japanese) — the axis of type sharing changes. It's no longer "one type referenced from both sides"; the same model gets written over and over in different representations, one per layer. A monorepo lets you reference the same type from anywhere, but it does nothing to keep definitions that have split into different representations consistent with each other.

The same model, written six times

Add a single Product model and these are the files you hand-write:

The Data Connect GraphQL schema (type Product @table(...))
The shared TypeScript type (interface Product)
The Zod schema (ProductSchema)
The Firestore projection schema (relations resolved to ids, timestamp becomes Date)
The API request / response types and their validation Zod
The NestJS class-validator DTOs

Each is a different representation of "the same thing," but nothing verifies them against each other. Every added field means walking through all of these files, and if you forget one, the breakage shows up much later — at runtime. This is not a problem you prevent by being careful; the multi-maintenance structure itself is the cause.

Write the contract, generate everything else

So I built a tool that unifies model definitions into a YAML contract file and generates every representation from it: firebase-contract.

One direction: contract → compiler (IR) → generated artifacts. Drift between artifacts and contract is caught by --check in CI.

# contract.yml (excerpt)
models:
  Product:
    key: [catalog, productNo]
    fields:
      catalog: { type: Catalog, relation: true }
      productNo: int
      title: { type: string, nonempty: true, maxLength: 200 }
      status: ProductStatus
      metadata: { type: ProductMetadata, optional: true }
      createdAt: timestamp

generators:
  - { generator: typescript, out: '#contracts', split: true }
  - { generator: zod, out: '#contracts', split: true }
  - { generator: data-connect-graphql, out: src, split: true }
  - { generator: firestore, out: '#contracts', split: true }

A single fbc generate emits the TS types, Zod schemas, GraphQL schema, and Firestore projection all at once. Constraints like title's nonempty: true, maxLength: 200 flow identically into Zod — and, if you declare the API generators (api-validation / api-dto), into request validation and class-validator DTOs as well. "Only the validation is stale" stops happening.

Contracts can be split with imports, so in a multi-app monorepo the yml files can follow the repository layout. In the project where I introduced this, the root contract splits into 9 yml files across 2 apps, generating over 30 files.

Firestore is a projection, not "another schema"

This is where the tool earns its keep. When Firestore is a read projection, its schema stands in an odd position relative to Data Connect: not the same, but not unrelated either. Relations become resolved string ids, timestamp becomes Date, denormalized fields get added. Hand-written, you end up transcribing this "regular transformation plus a few additions" wholesale.

In the contract, a projection is declared as a derivation from a Data Connect model.

firestore:
  Product:
    from: Product
    collection: shops/{ws}/.../products/{productNo}
    omit: [catalog, log]
    fields:
      linkedCatalogTitle: { type: string, optional: true }

The projection rules — relation → id, timestamp → z.date() — are applied uniformly by the generator; humans only write pick / omit and the extra fields. Project-wide fields shared across projections can be declared once as fragments: and inserted into each projection with extends:. The generated Zod schema reuses fields whose chains are identical to the Data Connect side via .pick(), so only the parts where the representation actually changes appear in the file. The diff reads as design intent.

Data Connect's Any boundary

Data Connect stores embedded objects and JSON as an Any scalar, erasing the logical type. With hand-written schemas, what metadata "really is" lives in comments and memory. Generating from the contract, the GraphQL side keeps the logical type as a comment (metadata: Any # logical: ProductMetadata), and typed adapters that convert between the Any row and the logical type are generated alongside. This is possible because the contract knows where the type-erasing boundaries are.

Design decisions that pay off

Generators never see YAML. Their only input is an IR (intermediate representation), normalized after parsing and import resolution. Validation is an independent set of rule functions over the IR. Adding a generator requires no changes to existing code, and extensions like OpenAPI output are just a registry entry.

I invested in byte-for-byte reproduction of the hand-written files. The adoption target was a product already in production; if generated output differed from the existing hand-written files by even one byte, the verification would drown in diffs. The style options that absorb formatting variance and the raw escape hatch exist for this incremental migration — "reproduce the existing file exactly, then switch it over." Migration became a per-file loop of "lean on generation, confirm the diff is zero" instead of a big-bang rewrite.

Regeneration is idempotent. As long as the content doesn't change, files stay byte-for-byte identical, generation timestamp included (generatedAt carries over from the first generation; only updatedAt moves when content changes). CI's --check can mechanically detect contract-code drift precisely because of this idempotency.

The costs

The DSL is itself added complexity. Field options and operations have a ceiling of expressiveness, and every time you hit it you choose between growing the generator and escaping through raw. Debugging generated code is one level more indirect.

Normally "the team has to learn a new YAML DSL" would count as a cost too, but AI has offset it. Load a coding agent with the DSL rules and the product spec, and an instruction like "add an inventory-count field to Product" gets turned into the contract yml edit by the AI. A declarative contract concentrates model definitions in one place with small diffs, which makes it a friendly target for AI as well. If the output is wrong, it shows up in the fbc generate diff, so verification is mechanical. What humans need to master is no longer the DSL grammar but "reading the contract and confirming the intent."

What remains is the expressiveness ceiling and the indirection in debugging — but even minus that, collapsing "add one field" into "add one line to the contract and run fbc generate" is worth a lot. Multi-maintenance drift moved from "something to be careful about" to "something CI detects." When type consistency shifts from human attention to machinery, review attention also shifts — from transcription-checking individual type definitions to the design of the contract itself.

Why not let AI write the type definitions directly?

If AI can write the contract, a further thought suggests itself: hand the LLM the rules and the product spec, let it edit each layer's type definitions directly, and you need neither contract nor generator. If humans never touch the type definitions, doesn't the multi-maintenance chore disappear entirely?

I think this gets the division of labor backwards. LLM output is probabilistic, so direct edits to six representations carry the risk that something is misaligned every single time. More fundamentally, without a contract the mechanical definition of "aligned" ceases to exist. To verify that six representations express the same model, you need their common origin — something equivalent to the contract. --check can detect drift because a deterministic generator can state uniquely what the correct output is; AI's direct edits have no such verifiability.

Review cost changes too. If AI edits six files directly, humans read six files of diffs with suspicion, every time. With the contract approach, humans read only the yml diff, and the rest is derived deterministically. Turning fuzzy specs into a contract is the AI's job; expanding the contract into each representation is the generator's job. It's precisely because the probabilistic layer and the deterministic layer are separated that you can let AI write with confidence.

Where it's published

Published as an npm package. Through the RC series I tightened "the contract is authoritative" (unknown keys and out-of-vocabulary values become errors), verified every feature drift-free in a real project, and released v0.1.0 as the first stable version. The DSL's expressiveness and generator details keep improving with feedback from operation.

npm — npmjs.com/package/firebase-contract
README — github.com/cilly-yllic/my-packages

Originally published at cilly-yllic.github.io.