DEV Community: MetalBear

Four Layers of Validation in Kubernetes with Claude Code

Jake Page — Thu, 21 May 2026 13:37:08 +0000

Earlier this year, Moltbook, a social network for AI agents, launched, trended, and became a cautionary tale within the same week. Security researchers at Wiz found a Supabase API key sitting in its client-side JavaScript, which was the database’s only access control, with no Row Level Security to narrow what that key could reach. The result: 1.5 million API tokens, 35,000 email addresses, and thousands of private messages exposed to anyone with a browser console.

Moltbook was a greenfield project with no review process. The same class of mistake is far more serious when AI-generated code lands inside applications that already have pre-existing users, real data, and an existing trust surface. That’s increasingly the reality: as organizations adopt AI coding agents, more and more AI-generated code is landing directly in production services that already hold credentials and personal details of your users. A recent survey found that 95% of developers don’t fully trust AI-generated code, while only 48% consistently review it before committing, yet it’s shipping regardless.

Static review tools catch only some classes of issues: common CVEs, dependency hygiene, style violations, deterministic anti-patterns. What they can’t see are things like the actual name of your Kubernetes Secret in this cluster, whether your auth middleware is wired into the right route in this service, whether the request a real user will send makes it through the new code path without breaking something downstream.

Closing that gap takes four independent layers: AI agent skills that shape what gets generated, commands that audit what was generated, integration tests that hit staging endpoints, routing traffic to your local code, and preview environments that let a human review the change against staging dependencies before merging.

Layer 1: Skills (passive, shaping what gets generated)

Most AI coding assistants let you write down rules that shape generation. The simplest mechanism is a config file that the assistant loads into context on every prompt: .cursorrules in Cursor, CLAUDE.md in Claude Code, .github/copilot-instructions.md in GitHub Copilot. Drop NEVER/ALWAYS rules in there and the AI follows them. The downside is that those files load on every prompt, even when you’re working on something unrelated, and every rule you add costs tokens whether or not it’s relevant.

Claude Code goes a step further with skills: structured rule sets that ship as plugins (a directory with a SKILL.md and supporting reference files). Each skill has a description, and the model pulls a skill into context only when your prompt matches what that skill is meant to cover. If a skill never gets matched against the prompt, it never gets loaded, and you don’t pay for the tokens it would consume. We’ve already shipped six skills for AI agents working with mirrord, and this post adds a seventh focused on validation: k8s-validation, an open-source set of NEVER/ALWAYS rules for code that runs inside a Kubernetes cluster.

The skill covers two halves: the cluster-level concerns (Secrets, RBAC, pod hardening, NetworkPolicies, supply chain, file handling) and the application-level concerns that determine whether the workload behaves correctly inside the cluster (HTTP and parameter handling, auth, output sanitisation, API contracts, env-var configuration, test coverage).

Before and after

Without the skill, the model has no way of knowing about things like Kubernetes Secrets already mounted as environment variables in your pod, auth decorators other handlers in your service already use, or PII-sanitization utilities your team has already built. So it does the obvious thing: hardcodes the API key, skips the auth check, and returns the LLM output directly. With the skill loaded, the following prompt (“add a /summarise endpoint that calls OpenAI’s API”) produces something like this:

import os
from openai import OpenAI
from utils.sanitize import filter_pii

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

@app.route('/summarise', methods=['POST'])
@require_auth
def summarise():
    text = request.json.get('text', '')
    response = client.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": f"Summarise: {text}"}],
    )
    summary = response.choices[0].message.content
    return jsonify({"summary": filter_pii(summary)})

filter_pii here is a stand-in for any utility that strips personally identifiable information (names, SSNs, emails, etc.) out of free text. Different teams build this differently; the Microsoft Presidio library is one of the more common open-source starting points.

Three differences from what the model would produce without the skill: the key comes from an environment variable backed by a Kubernetes Secret, the endpoint sits behind @require_auth, and the LLM output runs through filter_pii before going back to the user.

What skills can’t do

Skills shape generation, but they don’t verify anything.

In the example above, the AI correctly followed the skill: it read the API key from the environment, applied @require_auth, and called filter_pii from utils.sanitize. But the skill has no way to verify that filter_pii actually works. If the utility in your codebase only strips email addresses and misses phone numbers, the skill can’t know that. A user document containing a phone number sails straight through the filter and into the response, and the code looks correct at every layer the skill can see.

Skills set a floor by preventing the obvious structural mistakes. They’re instructions to a model, not checks against reality.

Layer 2: Commands (active, checking what was generated)

Where skills shape what the model generates, commands check what already exists. They’re explicitly invoked by a developer, an agent, or a CI step, and they run a defined set of checks against the code in front of them.

The same install from Layer 1 also ships a slash command: /k8s-validation:audit. It scans your codebase for the same NEVER/ALWAYS rules the skill enforces during generation, traces data flow through handlers and queries, and classifies each finding by severity. Skills don’t always load (a vague prompt, or a quick edit to a file the model didn’t classify as Kubernetes-adjacent, and the rules never enter context). The audit is the backstop: it runs on the code regardless of how the code got written.

> /k8s-validation:audit content-api/

CRITICAL 2 | HIGH 4 | MEDIUM 1 | INFO 0

[CRITICAL] src/routes/summarise.py: Hardcoded OpenAI API key → Use os.environ
[CRITICAL] src/routes/download.py: User filename in path without sanitization → Use secure_filename()
[HIGH]     src/routes/summarise.py: No authentication middleware → Add @require_auth
[HIGH]     k8s/deployment.yaml: No SecurityContext defined → Add runAsNonRoot, drop ALL
[HIGH]     src/routes/summarise.py: Reads OPENAI_API_KEY but no manifest defines it → Add to deployment.yaml env block
[HIGH]     src/routes/summarise.py: New endpoint with no integration test → Add test in tests/integration/

Note that the output mixes security findings (hardcoded key, missing SecurityContext) with correctness findings, meaning “does the code do what was asked, given the rest of the system” (the Kubernetes deployment manifest doesn’t define the env var the code reads; the new endpoint shipped without an integration test). Both halves matter for AI-generated code.

Because the audit is a command you run rather than a rule the model loads, the same invocation works in three places: a developer runs it before opening a PR, an agent runs it as part of its own loop after generating code, and CI runs it as a merge gate. You can wire it into one, two, or all three.

## Validation Workflow
After generating or modifying any Kubernetes-related code, run `/k8s-validation:audit`
on the changed files. If any CRITICAL findings exist, fix them before proceeding.

What commands can’t do

The audit is still static analysis. It can find “you hardcoded a secret” or “you’re missing a SecurityContext,” but it can’t tell you whether your filter_pii regex actually catches the PII your users will send, or whether the environment variable you’re reading will resolve to a value in your staging cluster. Commands check the shape of the code, not the behavior.

Layer 3: Integration tests (runtime, proving it works)

Your team probably already has integration tests that hit your API endpoints, check response shapes, and verify that authentication rejects bad credentials. These tests encode what “correct behavior” actually means for your application.

The bottleneck is running them. Locally, you mock your database, your auth service, your message queue, and hope the mocks match reality. In CI, each cycle takes 5 to 10 minutes. For a human pushing a few times a day, it’s already frustrating enough. For an AI agent trying to fix a failing test, it’s a feedback loop far too slow to learn from: the agent burns tokens on every iteration, and the integration bugs only surface after the change has been written, pushed, and built.

mirrord changes this equation by letting your local process stand in for a deployed pod. Your local code gets the environment variables from the target pod, the same cluster-level files it has access to, and the same view of internal services. In steal mode, traffic destined for the targeted pod is intercepted and routed to your local process instead of whatever’s deployed in staging. Your existing integration tests, pointed at staging endpoints as usual, now run against your local code in seconds, not minutes.

The same pattern scales horizontally. Because mirrord can split a single pod’s incoming traffic between many local processes using header-based filters, multiple agents (or developers) can iterate against the same staging cluster simultaneously, each one routing its own slice of the traffic to its own local code. One staging environment, many concurrent agents, real downstream services for all of them.

What this catches that the other layers can’t

Consider a prompt like “have /summarise fetch the document from our content-store service first.” The agent writes a handler that calls http://content-store/documents/{id} and reads response.json()["title"].

The catch: content-store moved to v2 months ago and now returns {"document": {"name": ..., "text": ...}}. The flat title/body shape only exists in the AI’s training data. Skills generated structurally clean code (good). The audit confirmed the call was made and the response was consumed (also good). Neither layer knows what shape content-store actually returns today.

The setup to fix this is two processes. You or your AI agent starts a mirrord session, your e2e tests run as normal against the staging content-api endpoint:

# Terminal 1, run your local content-api in place of the deployed pod
mirrord exec --target deploy/content-api --steal -- python -m content_api

# Terminal 2, run the existing integration suite against staging as usual
pytest tests/integration/test_summarise.py

When the test hits staging’s content-api endpoint, mirrord steals the request and reroutes it to your local process. The local handler calls http://content-store/documents/..., and that outbound call also routes through mirrord, hitting the real content-store in staging. The real service returns {"document": {"name": ..., "text": ...}}. The local code does response.json()["title"] and crashes with KeyError.

You fix the code to read the new shape, rerun the test, it passes. The bug surfaces in your local code, against real downstream services, in seconds, instead of after a deploy cycle. The same pattern works for any other dependency the code touches: environment variables from the pod, files from mounted volumes, database queries against the real Postgres. mirrord runs your code, the cluster supplies its real environment.

Layer 4: Human review in a real environment

When the agent opens the PR, a human should still get to see the change running in a real environment, not just read the diff. mirrord’s Preview environments make that easy: a GitHub Action spins up an isolated pod in your staging cluster running the PR’s code, connected to all the real downstream services, and scoped to that PR via an environment key.

Reviewers can click through the actual feature instead of inferring behavior from the code or having to run the whole application locally. Most of the failures the previous three layers don’t catch, UX regressions, surprising interactions, “looks right but feels wrong”, show up the first time a human uses the thing.

Putting it together: agents handle layers 1–3, humans handle layer 4

Layers 1 through 3 can be run by the agent itself. Instead of generating code, opening a PR, and hoping CI catches the issues, the agent generates code shaped by skills, runs /k8s-validation:audit to check for structural issues, runs the integration tests via mirrord against real infrastructure, and fixes any failures before committing. The agent doesn’t even have to write the test from scratch: Layer 1’s validation skill includes a rule that any new HTTP handler must come with an integration test, so the test gets generated alongside the endpoint. Layer 3 just runs it against real infrastructure.

Layer 4 is the handoff. Once the agent has passed its own checks, it opens a PR and a human gets a live preview environment to click through rather than a diff to infer from. The failures that surface at that stage, UX regressions, surprising interactions, things that look right but feel wrong are exactly the ones that didn’t show up in the previous three layers.

We’ve documented the concrete setup (per-service mirrord configs, helper scripts, and an AGENTS.md that tells the agent which script to run) in our mirrord with AI agents guide. The broader argument for why this matters, including the token cost of agents stuck in a feedback-less loop, is in How to prevent token burn using mirrord with e2e tests.

Adopting the layers

The four layers are independent. Pick the ones that close your biggest gap and add the others when the next failure teaches you what’s still missing.

Install the validation skill with /plugin install k8s-validation@metalbear-co/k8s-validation-plugin in Claude Code, or as a local rules directory for Cursor and GitHub Copilot (see the repo README). The /k8s-validation:audit command ships in the same install. For the runtime layer, install mirrord and wrap your local service with mirrord exec --target deploy/<service> --steal -- <run-command>. For preview environments on each PR, the feature is included in mirrord for Teams.

Each layer closes a different gap. Stop at any point and you’ve made things better than they were.

The skills are open source. If your AI assistant generates something the skills don’t catch, open a PR.

New Features We Find Exciting in the Kubernetes 1.34 Release

Arsh Sharma — Thu, 04 Sep 2025 10:54:31 +0000

The Kubernetes 1.34 release, “Of Wind & Will (O’ WaW)”, sets sail with a collection of features that may not grab headlines, but instead improve the day-to-day experience of running and managing clusters. Think of it as steady winds filling the sails, incremental but powerful improvements that help cluster administrators navigate smoother waters. And since combing through the full release notes can feel like charting a course through rough seas, we’ve pulled together some of the most useful highlights from this release to keep you on course.

Features Moving to Stable: Anchored and Ready for the Voyage

Dynamic Resource Allocation with Structured Parameters

Today where most companies are running (or trying to) run AI workloads on Kubernetes, we feel this feature is going to be the most important one from this release.

Before this change, Kubernetes didn’t really “see” specialized hardware devices like GPUs. Instead, the device drivers running on each node were responsible for keeping track of hardware availability and assigning it to pods. Kubernetes itself had no awareness of what devices existed, how many were available, or where they were located.

This lack of visibility introduced a number of challenges. For example, if two pods requested a GPU on the same node, the driver had to resolve the conflict on its own, without the scheduler knowing what was going on. Sharing GPU resources across pods was equally messy, since Kubernetes couldn’t reason about device capacity or availability and only knew that a pod needed “a GPU,” not whether the hardware was free or oversubscribed.

With Kubernetes v1.34, this changes. The new Dynamic Resource Allocation (DRA) framework, along with by structured parameters, lets Kubernetes manage specialized hardware devices directly. Structured parameters let hardware drivers describe the exact capabilities and constraints of a device in a standardized format that Kubernetes can understand. For example, instead of just saying “this node has 1 GPU,” the driver can specify details like GPU memory, supported features, or whether the device can be shared across multiple pods.

With this feature, instead of drivers silently handling everything, they now publish available devices as Kubernetes objects called ResourceSlice. These slices describe what hardware exists on each node and expose details that Kubernetes can use to make better scheduling decisions.

In other words, Kubernetes is no longer “blind” to hardware like GPUs, it can see what’s available, allocate it fairly, and even support sharing more reliably. Here’s a simple example:

apiVersion: resource.k8s.io/v1
kind: ResourceSlice
metadata:
  name: resourceslice
spec:
  nodeName: worker-1
  pool:
    name: gpu-pool
    generation: 1
    resourceSliceCount: 1
  driver: dra.example.com
  sharedCounters:
    - name: gpu-memory
      counters:
        memory:
          value: 8Gi
  devices:
    - name: gpu-1
      consumesCounters:
        - counterSet: gpu-memory
          counters:
            memory:
              value: 8Gi

This example shows a single GPU on node worker-1, grouped into a pool called gpu-pool. The GPU has 8 GB of memory, published through sharedCounters. Because this information is now visible to Kubernetes, the scheduler knows exactly how much memory the device provides and can place pods accordingly, instead of relying on the driver to manage it behind the scenes.

Allowing Only Anonymous Authentication for Configured Endpoints

With Kubernetes v1.34, you can now safely expose specific API server endpoints like /healthz, /readyz, and /livez without accidentally granting broader anonymous access.

By default, Kubernetes treats unauthenticated requests as anonymous, assigning them the identity system:anonymous and group system:unauthenticated. While this design supports useful cases such as health checks, it also comes with risks. If a RoleBinding or ClusterRoleBinding is accidentally applied to system:anonymous, even unauthenticated code running in a pod could gain access to the API server.

The new update fixes scenarios like these by giving administrators fine-grained control over which paths are available to anonymous users. Instead of having to disable anonymous access entirely, which can break important functionality, you can now define a safe list of allowed endpoints using the new AuthenticationConfiguration object.

Here’s an example:

apiVersion: apiserver.config.k8s.io/v1beta1
kind: AuthenticationConfiguration
anonymous:
  enabled: true
  conditions:
  - path: /livez
  - path: /readyz
  - path: /healthz

In this configuration, anonymous requests are permitted only to the health check endpoints, while all other anonymous requests are denied, even if misconfigured RoleBindings exist.

Relaxed DNS Search String Validation

The dnsConfig.searches field in a Pod manifest controls how short hostnames are resolved. If your pod looks up myserver, Kubernetes will automatically append each domain listed under dnsConfig.searches and try them in order. For example, myserver.first-domain, then myserver.second-domain, and so on until it finds a match. This mechanism is especially important when migrating legacy applications into Kubernetes since many older services rely on custom DNS search domains, sometimes with underscores (_) or leading dots (.).

Until now, Kubernetes didn’t allow these in dnsConfig.searches, which meant some services simply couldn’t be deployed without changing their DNS assumptions which is a task that can be both risky and time-consuming.

With Kubernetes v1.34, these restrictions are lifted. You can now include underscores and dots in DNS search domains, which makes it much easier to bring legacy workloads into Kubernetes without rewriting configurations or service names.

Here’s what that looks like in practice:


apiVersion: v1           
kind: Pod          
metadata:
  name: app-1
spec:
  containers:
    - name: app-1
      image: nginx
  dnsPolicy: "None"
  dnsConfig:
    nameservers:
      - 10.244.0.69
    searches:
      - abc_d.staging.com  # Now valid
---
apiVersion: v1           
kind: Pod          
metadata:
  name: app-2
spec:
  containers:
    - name: app-2
      image: nginx
  dnsPolicy: "None"
  dnsConfig:
    nameservers:
      - 10.244.0.69
    searches:
      - .   # Now valid
      - default.svc.cluster.local
      - svc.cluster.local
      - cluster.local

With this update:

app-1 can now resolve fully qualified names like test.abc_d.staging.com without issue.
app-2 can use the special "." search entry to resolve hostnames exactly as written **before Kubernetes appends cluster defaults like svc.cluster.local.

So to sum up, if you’re migrating older workloads or working with services that depend on unconventional DNS naming conventions, Kubernetes no longer forces you to rework your DNS setup.

Features Moving to Beta: Stronger Winds Filling the Sails

PreferSameNode Traffic Distribution

Routing traffic efficiently in Kubernetes can be challenging, especially when services span multiple nodes or zones. By default, Kubernetes load-balances traffic across all healthy endpoints of a service, even if some are farther away. This can create unnecessary cross-node or cross-zone hops, which add latency and waste bandwidth. For many workloads, especially latency-sensitive applications like real-time inference, gaming backends, or high-throughput APIs, those extra hops can noticeably hurt performance.

Kubernetes v1.34 introduces new traffic distribution policies to address this:

PreferSameNode: Services can now prefer pods on the same node when the traffic originates from that node. This reduces latency and avoids network overhead from cross-node traffic.
PreferSameZone: The existing PreferClose policy has been renamed to PreferSameZone to make its behavior clearer. This helps workloads prioritize pods in the same zone for lower latency and reduced cross-zone costs.

Here’s an example of both in action:

apiVersion: v1
kind: Service
metadata:
  name: http-echo-service-1
spec:
  selector:
    app: http-echo-pod-1
  ports:
    - protocol: TCP
      port: 80
      targetPort: 5678
  trafficDistribution: PreferSameNode
---
apiVersion: v1
kind: Service
metadata:
  name: http-echo-service-2
spec:
  selector:
    app: http-echo-pod-2
  ports:
    - protocol: TCP
      port: 80
      targetPort: 5678
  trafficDistribution: PreferSameZone

With this configuration:

http-echo-service-1 routes traffic to a pod on the same node when possible. If none are available, it will fall back to another node.
http-echo-service-2 routes traffic to a pod in the same zone first, and only if necessary, sends it to a different zone.

VolumeSource: OCI Artifact and/or Image

In some deployments, multiple pods need access to the same configuration or data files. Traditionally, this might mean baking files into a ConfigMap or copying them into each pod, which can get messy as files grow larger or more complex.

With Kubernetes v1.34, you now have a cleaner option: image volumes. This feature lets Kubernetes mount an OCI image as a read-only volume that can be shared across multiple pods. Instead of duplicating files or manually syncing them, you package the files once into an image and then mount them wherever they’re needed.

Here’s an example:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  containers:
  - name: nginx-container
    image: nginx
    ports:
    - containerPort: 80
    env:
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    volumeMounts:
    - name: oci-volume
      mountPath: /data
      subPath: hello.txt
    - name: oci-volume
      mountPath: /data/
      subPathExpr: $(POD_NAME).txt
  volumes:
  - name: oci-volume
    image:
      reference: <your-oci-image>

In this example, the oci-volume references an OCI image that contains files. The pod mounts specific files from that image, hello.txt, into its container. Since the volume is read-only, multiple pods can safely share the same source without conflicts.

For developers using mirrord, this also means that you can now run your local code against a pod that already mounts an image volume in the cluster. Your local process will gain the same access to the shared files, so you can test how your code interacts with the data without deploying or modifying the cluster setup.

Features Moving to Alpha: New Currents on the Horizon

Allows Setting any FQDN as the Pod's Hostname

Prior to this feature, setting a pod’s fully qualified domain name (FQDN) wasn’t straightforward. You had to combine multiple fields: hostname, subdomain, and setHostnameAsFQDN: true, to build an FQDN. The result always followed Kubernetes’ enforced pattern:

<hostname>.<subdomain>.<namespace>.svc.<cluster-domain>

If any part of that was misconfigured, the final FQDN wouldn’t resolve as expected. This could cause serious issues for systems that rely on strict hostname matching, such as Kerberos, where authentication breaks if hostnames don’t line up exactly.

This pain point is now addressed through a new field: hostnameOverride. This field allows you to explicitly set the pod’s internal kernel hostname to whatever value you need, bypassing Kubernetes’ enforced DNS naming convention.

Here’s an example:

apiVersion: v1           
kind: Pod          
metadata:
  name: app
spec:
  hostnameOverride: app.nginx.example.domain
  containers:
    - name: app
      image: nginx

In this configuration, the pod’s internal hostname is set directly to app.nginx.example.domain. Unlike before, you don’t need to stitch together multiple fields or follow Kubernetes’ strict naming rules.

Takeaways From Of Wind & Will Kubernetes Release

The Of Wind & Will release is not about grand, sweeping changes, but it’s about steady progress. The kind of refinements that make Kubernetes a sturdier ship for the long journey ahead. One of the standout improvements is the graduation of Dynamic Resource Allocation (DRA), a huge step forward for teams running AI workloads. With DRA, Kubernetes finally gains proper visibility into GPUs and other specialized hardware, making scheduling and sharing these resources far more reliable. For anyone working on machine learning or high-performance computing, this is a game-changer.

Security also gets tighter with more fine-grained anonymous access controls, ensuring only the right endpoints are exposed without weakening cluster safety. And for those migrating legacy systems, features like hostnameOverride and expanded DNS search support smooth out long-standing pain points, making Kubernetes more welcoming to applications that weren’t originally built for it.
Together, these updates may feel like adjustments to the rigging rather than rebuilding the ship, but that’s what makes them powerful. They reduce friction, close gaps, and make Kubernetes more adaptable to the real-world seas we sail in. With 1.34, the community hands cluster administrators and developers not just new features, but stronger winds and steadier will to keep moving forward.