Five editions.

When you look at the final day, the packed hall, the conversations, the talks, the panel discussion, the activities, the networking, the volunteers running around, and the smiles at the end of the day, it can look like everything came together naturally.

It didn't.

An event like AWS Community Day Vadodara is the result of weeks and months of planning, conversations, follow-ups, spreadsheets, calls, last-minute changes, problem-solving, and a lot of people quietly doing things behind the scenes.

This is the story of how we planned this year's edition.

It Starts With a Simple Question: Should We Do It Again?

Every edition starts with the same question.

Can we make the next one better than the previous one?

By the time you reach the fifth edition, expectations are different. The community knows what AWS Community Day Vadodara feels like. Speakers have expectations. Sponsors want to understand the value of participating. Volunteers know the scale of the work involved. And attendees expect an experience that is worth giving up their Saturday for.

So planning doesn't really begin with picking a date or finding a venue. It begins with defining what we want the event to be.

For this edition, the goal was simple: build a day where people could learn from practitioners, meet others from the community, have meaningful conversations, and walk away with something useful.

Building the Organizing Team

The first major piece of the puzzle is the team behind the event.

Community events are never built by one person. Over the years, we've built a core team, and each person on it has taken on a role that lets us pull off an event at this scale: sponsorship, speaker management, activities, volunteer coordination, and everything in between. This year, that core team came together for our first planning call about three months before the event, and from there each of us just ran with our piece of it.

A large part of the planning involves breaking a huge event into hundreds of small tasks and making sure someone owns each one. The organizing team needs to constantly ask:

What needs to happen? Who owns it? What is the deadline? What depends on this? What happens if something goes wrong? And, inevitably: what did we forget?

Finding the Venue

We've hosted AWS Community Day Vadodara at Sun City Club and Resort for the last few years now, and it's become home for this event. But "same venue" doesn't mean "same plan." Every year we try to innovate around it, sometimes more tracks, sometimes fewer, more activities one year, a different food stall lineup the next.

There's also a constant we plan around every single year: rain. September in Vadodara is unpredictable, and a good chunk of the venue and layout planning is really about building in enough flexibility that an uncertain sky doesn't derail the day.

The venue determines how attendees move through the event and how the entire experience feels. We wanted enough space for people to comfortably attend sessions, interact with sponsors, network during breaks, and take part in the activities happening throughout the day.

Sponsorship: Making the Event Possible

One of the most important, and often least visible, parts of planning a community event is sponsorship.

Events cost money. Venue, food, branding, printing, AV, photography, logistics, attendee materials, badges, merchandise, and countless smaller expenses all add up.

Nirmal put in an enormous amount of effort coordinating sponsorships this year, with strong support from Jay, Dhaval, and Manan. Between conversations, proposals, follow-ups, requirements, branding, booth arrangements, deliverables, and coordination that ran right up until, and sometimes during, the event, this is a genuinely hard job, and Nirmal and team carried it well.

This year, we were fortunate to have the support of: phonon, automating outcomes, Rezoomex, Jeavio, SUDO Consultants, AVDEVS, iOS Development, APPMATTIC, CloudKida Pvt Ltd, Nishchay Labs, and KonfHub.

Every sponsor played a role in making this edition possible. Without sponsors supporting the initiative, an event at this scale simply wouldn't exist.

Building the Speaker Lineup

Then comes one of the most exciting parts: speakers.

This year we ran the entire speaker selection process through Sessionize, and it was a genuinely smooth experience end to end. A big thank you to Nirmal and Chirag for helping me with speaker selection, sifting through submissions and helping shape a lineup that felt right for this edition.

This year's speakers included: Dhaval Nagar, Praful Bagai, Darko Mesaroš, Srushith Repakula, Aric Pandya, Rucha Chormunge, Ancelia Patrao, Vishnu Rachapudi, Ashish Gajjar, Krutarth Rindani, Satyanarayan Sahoo, Kushal Nagrani, Ishan Jain, Sankalp, Sandeep Paranjpe, Pritesh Kiri, Dhruvesh Patel, Israr Mahedi, and M Lokhandwala.

Once speakers were confirmed, the harder part started for me: designing the agenda. Fitting topics, formats, and timings together so the day flows well is never trivial, and I have to thank Nilesh for the thinking that led us to placing Darko in the main track, a call that made the agenda click into place.

The Panel Discussion

The panel this year was Manan's idea, and it turned into one of the highlights of the day.

The AI Shift: Business, Talent & Future

Panel discussion featuring Vadodara's tech founders and CXOs.

AI is changing how businesses operate, how teams work, and what it actually means to be ready for the workforce ahead, but most of that conversation happens in hype cycles rather than in the room with people actually building through it.

This panel brought together Vadodara's tech founders and CXOs to cut through that noise. The conversation covered where AI is genuinely driving business value versus where it's just adding noise, what leading teams and careers looks like in an AI-first era, how students and early professionals should actually be preparing for this shift, and what it takes to build a business today that's still built to scale tomorrow. It was a grounded, city-local take on questions every builder in the room is already asking.

It was brilliantly executed by Chirag Darji as moderator, who kept the conversation sharp and made sure every voice on that stage got room to speak.

Activities

A community day shouldn't only be about sitting in a chair and listening, so this year we built out a set of activities:

• Ask Us Anything: AI and Career Growth

• Show Me Your Architecture

• Show Me Your Project

These came from my own experience attending events around the world and noticing how much value sits in these smaller, more interactive formats. I'm grateful to everyone who backed the idea early on and to the team that actually executed it:

Ravi Soni led AI in the Industry session, Nilesh Vaghela led career growth, Vrajesh Jayswal led the project track, and Udit Parikh led architecture. And Jay Patel was behind putting this whole plan into execution with all the logistics it needed to actually run on the day.

The Volunteer Army

If speakers are the people attendees see on stage, volunteers are the people who make sure the stage, and everything around it, works.

Registration. Badge distribution. Speaker assistance. Room coordination. Timekeeping. Microphones. Audience questions. Photography. Speaker movement. Food coordination. Sponsor support. Troubleshooting. And the hundred tiny things that nobody notices when they're done correctly.

To our volunteers: thank you for supporting the community, it was truly an amazing experience working with each and every one of you.

What stood out to me most was how everyone just stepped up, took ownership, and delivered without needing to be asked twice. That's rare, and it's exactly what made this edition special.

Always here if you need anything from my side.

Registration and Attendee Experience

Another major part of planning was the attendee journey.

From the moment someone registers, they begin forming an expectation of the event. What information do they receive? How easy is check-in? Where do they go? How do they know which session is happening where? Where do they get food? Where can they network? Where are the sponsors? Who can they ask for help?

The attendee experience isn't one thing. It is the sum of hundreds of small interactions throughout the day.

Communication

Communication becomes its own project as the event gets closer, and this year our social media team made sure that whatever we wanted to reach the audience, reached them.

Srujal Shah led this effort brilliantly, with amazing support from Malhar and help from Maitry. Whether it was speaker announcements, schedule updates, or last-minute changes, the social media team made sure nothing got lost along the way.

A lot of the graphics this year were appreciated by everyone too, and that's thanks to Shivang and Sara, who put in a huge amount of effort into small details across social media creatives, event day signage, banners, lanyards, and our five-year journey wall. Everything came together beautifully.

The website was also excellent this year, all thanks to Parth for putting it together, and as always, Shivang for the UI/UX design work behind it.

The Final Week

The final week always feels different. The long-term planning suddenly becomes very real.

Names are finalized. Badges are prepared. Schedules are checked again. Signage is reviewed. Speaker communication is reconfirmed. Volunteer responsibilities are assigned. Sponsor requirements are verified. Equipment is checked. Registration lists are reviewed.

That final week was largely carried by Jay Patel and Nirmal Pathak, who put in an enormous amount of effort to make sure everything was actually ready when the day arrived.

And then someone inevitably asks: "Are we sure everything is ready?"

The honest answer is never completely. There is always something. But eventually you reach the point where planning has to stop and execution has to begin.

Event Day

Then came 12th September.

The hall started filling up. Volunteers moved into their positions. Speakers arrived. Sponsors set up. Attendees started networking. Sessions began.

Behind the scenes, the AV and production setup was managed by Parth, with support from Dharmin, Nisarg, and Shivang. These folks did an amazing job keeping everything running smoothly without ever being visible from the audience's side of things.

A few people also went well above and beyond on the day itself: Ayush with videography, and Neel, Jenish, Preet, and Srujal helping out across a lot of behind-the-scenes work and support wherever it was needed.

A big thank you to Dhaval Pancholi as well, for an amazing opening note and for coaching and helping our MC team through the day. And to the MC folks themselves, Nidhi, Paree, and Drishti, who did a great job keeping the stage moving.

And suddenly, all those spreadsheets, messages, meetings, calls, and checklists turned into a real community event.

Five Editions In

Five editions have taught us something important.

Community isn't created by an event. An event simply gives the community a place to come together. The real community exists in the people who show up, contribute, share knowledge, introduce themselves to strangers, help someone solve a problem, volunteer their time, speak on stage, sponsor an initiative, organize behind the scenes, or simply attend and participate.

That's what makes AWS Community Day Vadodara special. It isn't the venue. It isn't the stage. It isn't the branding. It's the people.

Thank You

To our sponsors, phonon, automating outcomes, Rezoomex, Jeavio, SUDO Consultants, AVDEVS, iOS Development, APPMATTIC, CloudKida Pvt Ltd, Nishchay Labs, and KonfHub, thank you for believing in the community and helping make this year's edition possible.

To our speakers, thank you for sharing your time, experience, and knowledge with us.

To Chirag Darji, thank you for moderating our panel with such energy and for making sure every perspective on that stage came through.

To our volunteers and core contributors, thank you for everything you did behind the scenes. Much of your work may never be visible to attendees, but the event would not happen without you.

To our organizing team, thank you for the countless hours, discussions, decisions, and compromises that went into bringing everything together.

And finally, to everyone who attended, whether this was your first AWS Community Day Vadodara or your fifth, thank you for showing up. You didn't have to come. You chose to. And that's what makes all of this worth doing.

A special note, from Nirmal Pathak and me: this was Jay's event through and through. From the smallest details to the biggest plans, everything was carefully curated and executed by him and the team of volunteers behind him. The swags, the audio, every bit of coordination with the venue, all of it has Jay at the core. Thank you, Jay, for everything you pour into this community.

And a note for Nirmal Pathak, from me: it is difficult to imagine something and then put in the continuous effort it takes to actually get there. Thank you for that effort, Nirmal Pathak, as we fondly call him, the supreme leader of AWS User Group Vadodara.

What's Next?

The fifth edition is now a memory. But the work doesn't really end when the event does.

We want to know what worked, what didn't, which sessions people loved, which activities they want more of, and what the community wants to see next.

That's why feedback matters. Every response helps us understand how we can make the next edition better.

Because if there is one thing we've learned after five editions, it's this: the next AWS Community Day Vadodara starts the moment the current one ends.

And we'll see you there.

Thank you, Vadodara. ❤️

Five editions down. More to come.

AWS Community Day Vadodara is a community-driven event made possible by the collective efforts of organizers, volunteers, speakers, sponsors, partners, and attendees.

PS: I was unwell over the last couple of days leading up to and around the event, so I couldn't give this edition my full self the way I normally would. Thank you to everyone who stepped up and supported through it, it meant a lot.

Welcome to Everything about EKS & AI Infrastructure #75.

There’s a thread running under this edition, and it’s not about models — it’s about the layer underneath them finally catching up. DRA reaching GA and HAMi hitting CNCF Incubating in the same window means Kubernetes finally has a real, open answer for fractional GPU scheduling instead of vendor-specific hacks. The EKS control plane stories tell the same story from the reliability side: the etcd-journal rework that made the control plane compute-bound instead of disk-bound, version rollbacks that finally give clusters an undo button Kubernetes never had, and AWS DevOps Agent tracing a 66,000-requests-per-minute controller back to an invisible APF seat exhaustion in seven minutes flat.

Governance is the other half. Agentgateway’s CEL-based tool masking and LLMKube’s fail-closed ModelRouter are solving the same problem from opposite directions — one keeps an agent from touching a tool it shouldn’t, the other keeps regulated data from ever reaching a cloud model by construction, not by convention. Neither of these is glamorous work. Both are the difference between a platform you can actually trust with production traffic and one you’re hoping doesn’t get audited.

The scheduling primitive matures. The control plane stops surprising you. The agent stops being trusted by default. None of that shows up in a keynote. All of it is the actual job.

1. Performance Engineering in Modern AI Systems 🌩️

🌩️Dynamic Resource Allocation reaches GA in Kubernetes 1.35

DRA replaces the device-plugin era’s opaque, all-or-nothing GPU allocation with structured claims and device classes — the difference between “this pod gets a GPU” and “this pod gets exactly the slice of GPU it asked for, expressed as an API object instead of a vendor-specific hack.” For EKS clusters running mixed training and inference workloads on the same fleet, that’s the primitive fractional scheduling has been missing since Kubernetes started treating accelerators as indivisible bricks. It’s also the layer HAMi (below) builds directly on top of, so the two ship in the same window for a reason.

If you’re still running device-plugin-era GPU scheduling, DRA going stable is the trigger to start planning the migration. Worth watching whether AWS ships DRA support into EKS-managed add-ons or leaves it to cluster operators to wire up themselves — that’s the detail that decides how fast this actually lands in production versus staying a v1.35 changelog line.

🌩️NVIDIA’s TwoTower gets a 30B model to 2.4x faster generation

Nemotron-Labs-TwoTower splits generation across a frozen autoregressive tower that causally reads context and a trainable diffusion tower that denoises token blocks in parallel via cross-attention to that context — adapting an existing model to diffusion decoding rather than training one from scratch. Built on the open Nemotron-3-Nano-30B hybrid Mamba-Transformer MoE, it needed only ~2.1T training tokens against the backbone’s original 25T, which is what makes this a realistic thing to try rather than a full retrain.

The number that matters: 98.7% of the original model’s quality retained at 2.42x wall-clock throughput, tested against a real 30B backbone. Weights and code are released, so this is testable against your own serving stack this week — the same diffusion-decoding idea showed up in DiffusionGemma and DSpark, but this is the first version proven by adapting an off-the-shelf AR model instead of training one purpose-built for it.

🌩️Gary A. Stafford: Running LiteLLM as a Proxy in Front of Multiple Model Providers

The distinction most LiteLLM writeups blur: the library and the proxy are different things. litellm.completion() runs in-process straight to the upstream provider — nothing shows up in an admin UI, no virtual key gets checked. The proxy is the actual gateway: one HTTP surface in front of Bedrock, Anthropic, OpenAI, and in-cluster vLLM/TGI, with provider credentials living only on the gateway host while every caller authenticates with a scoped, revocable virtual key instead of a shared provider secret — and AWS’s own reference architecture deploys that identical proxy config on ECS Fargate or EKS behind an ALB.

Two gotchas worth carrying forward: reasoning models like Qwen3 and Gemma4 stream chain-of-thought into a separate reasoning_content channel that eats into max_tokens before user-facing text appears, so token budgets need headroom you wouldn’t expect from a non-reasoning model; and as of June 2026, Bedrock Mantle’s Anthropic Responses-API route breaks through LiteLLM because the gateway downconverts to chat-completions, which Mantle’s Anthropic models reject — route Claude through native bedrock/anthropic.* instead. Worth reading alongside the token-monitoring piece from edition #74, which flagged LiteLLM CVE-2026-42271 — pin to ≥1.83.7 if you’re standing this up net-new.

🌩️Scaling video generation across a GB200 NVL72 rack with TensorRT-LLM

Video diffusion models are a different scaling problem than LLM serving: no KV cache to reuse, one request generating one video, and the model re-running full attention over the entire frame sequence on every one of 30-50 denoising steps. NVIDIA’s VisualGen runtime handles this by combining four separate ways of splitting the work across GPUs — halving the guidance computation, splitting the sequence itself, splitting attention across a 2D grid of GPUs instead of a single line of them, and splitting the final decode step — all as config settings, with no changes to the model code required.

The headline number: one specific combination of those settings takes the core generation loop from 371 seconds on a single GPU down to 7 seconds across a 72-GPU rack — about 53x faster — and the identical settings work unchanged on a second, much larger model, which is the real proof this generalizes rather than being hand-tuned to one case. The sharp detail is why splitting attention across a 2D grid beats the older single-line approach at this scale: the single-line approach’s GPU-to-GPU communication cost stays roughly fixed no matter how many GPUs you throw at it, so it stops paying off past a certain point — at 72 GPUs it’s already 83% slower than the 2D-grid version. There’s a second bottleneck worth knowing too: once the generation loop is sped up 53x, the final decode step back into pixels becomes a quarter of total request time if left unsplit on one GPU — splitting that step as well shaves it down further and adds another 20% speedup end to end.

1. Starred Content ⭐

⭐Mark Vinciguerra, Daniel Schoonover & Tomonori Shimomura: SageMaker HyperPod EKS in a fully private VPC

A closed-network HyperPod cluster has no internet gateway, no NAT gateway, and no public subnets — every AWS API call a node makes (assuming a role, pulling an image, writing a log) has to traverse a VPC endpoint instead, and the failure mode when one is missing is silent: a pod just hangs in ContainerCreating with no error pointing at the actual cause. The post treats the eleven interface endpoints plus the S3 gateway endpoint as a dependency chain rather than a checklist: the VPC CNI plugin needs the ec2 endpoint before a pod can even get an IP, Pod Identity needs eks-auth and sts before that identity resolves to credentials, and only then can ecr.api/ecr.dkr pull the image — so a missing endpoint early in that chain surfaces as a confusing timeout several steps downstream, which is exactly why 9 times out of 10 a stuck ContainerCreating traces back to the missing ec2 endpoint specifically.

The three-phase structure is the part worth copying regardless of your compliance requirements: mirror exactly three images (NVIDIA’s device plugin, the MPI operator, the Kubeflow training operator — everything else pulls from AWS-owned ECR accounts directly) on a build host with internet access, run a single terraform apply against the empty private VPC, then verify connectivity with a two-layer check (raw TCP/443 plus an actual boto3 call, since the endpoint can exist while an IAM or endpoint policy still blocks the call silently) before flipping the EKS API to private and closing the air gap. Real numbers worth budgeting around: a single ml.p5.48xlarge consumes up to 81 IPs across its ENIs, which is why subnets need to be /18 or larger, and the eleven interface endpoints run about $290/month in endpoint-hours alone across 4 AZs before any data-processing charges. Worth reading alongside CommBank’s HyperPod-on-Kubernetes piece from edition #74 — that one covers the enterprise-constraints side (IRSA, MIG partitioning, NCCL debugging) on a no-outbound-internet cluster; this one is the missing piece showing exactly how you build the closed network those constraints run inside.

⭐Shyam Kulkarni, Kuntal Shah, and Vasanth Raju: Diagnosing EKS control plane performance issues with AWS DevOps Agent

API server overload is one of the harder EKS failure modes to catch because API Priority and Fairness makes it partially invisible to the workload causing it: when a seat is exhausted, the API server returns a 429, client-go retries transparently, and the request eventually succeeds — so the offending controller’s own logs show nothing wrong even as the EKS audit logs fill with hundreds of throttled requests. The post’s repro scenario makes the mechanism concrete: a controller mixing LIST/GET (fast, high-volume), WATCH (holds connections open, disproportionately expensive on seats), and MUTATE calls at 50 replicas × 80-100 req/s produces 1,600-2,000 req/s total, pushing basic kubectl latency from ~100ms to 1.5s+ while nodes and pods themselves stay healthy — which is exactly the trap, since standard troubleshooting instinct is to check node/pod health first.

The root cause AWS DevOps Agent surfaced by correlating CloudWatch metrics, EKS audit logs, CloudTrail, and cluster state: one controller deployment generating ~66,000 API requests per minute, consuming 99%+ of API server capacity — identified in about 7 minutes end to end. The detail worth keeping regardless of tooling: the audit logs are the only place this shows up cleanly, because they capture the request that got throttled, not just the retried response the application eventually saw. Any team running high-replica controllers or aggressive polling tools (CI/CD pipelines, monitoring agents) on EKS should know to check APF seat metrics and audit logs directly rather than trusting application-level error rates during a control-plane slowdown.

⭐Miro Nikolov & Barak Epstein: Multi-node KV cache offloading with GKE and Managed Lustre

Long-context and agentic workloads push KV caches past what local CPU RAM and host SSD cache tiers can hold, and the usual fix — pooling node-local NVMe into a distributed layer — pushes the complexity onto the cluster itself, which now has to handle its own cross-node data distribution and replication. This approach sidesteps that entirely: it points llm-d‘s open-source OffloadingConnector at Google Cloud Managed Lustre instead of local disk, turning a shared parallel filesystem into one cluster-wide KV cache tier that any GPU node can read from, with no host-capacity ceiling and no custom replication logic to maintain.

The numbers, on a six-node A3 Mega cluster serving Llama-3.3-70B with a 50,000-token prompt: over 50% TCO savings and nearly 60% fewer GPU-hours, driven by a 95% cache hit rate on offloaded prefilled KV caches — meaning nearly all repeat-prefix requests skip recomputation entirely. Layering CPU RAM offload in front of the Lustre tier as a hybrid (rather than Lustre alone) adds another ~40% TTFT improvement and ~30% lower end-to-end latency. The part worth knowing before you copy this: the offloading connector writes cache blocks to the filesystem but never deletes them, so this only works in production with a separate garbage-collection service running alongside it — their PVC Evictor sizes at roughly 1 replica per 72TB of Lustre capacity and shards the cache namespace across replicas to avoid redundant metadata scans.

This is GKE-native as published, but nothing in the mechanism is GCP-specific — llm-d‘s connector is the portable part, and AWS’s equivalent path is FSx for Lustre with its CSI driver in front of an EKS GPU node pool, running the identical OffloadingConnector config. Worth prototyping if you’re running long-context serving on EKS and hitting local-SSD cache limits; the manifest structure in the post (StorageClass, PVC, the vLLM kv-transfer-config block) maps over almost directly, swap Lustre CSI for FSx for Lustre CSI.

⭐Guidance for Accelerator-Optimized Agentic Bidding on AWS

Programmatic ad bidding runs on a hard timeout budget, which is exactly why GPU inference inside the bid path has been rare — you can’t add a model call if it risks missing the auction window. This guidance implements the IAB’s Agentic RTB Framework (ARTF) directly on EKS: an OpenRTB bid request fans out in parallel to four containers — a DLRM bid-shading model, a Wide & Deep segment activator, an NCF deal matcher, and a rule-based metrics enricher — each serving inference off NVIDIA Triton on GPU nodes, with the orchestrator merging their proposed bidstream “mutations” into one response before the auction continues. Bedrock AgentCore optionally exposes the whole pipeline as an MCP tool (extend_rtb), so an external agent can call into the bidding decision directly rather than only reading its output.

The infra is fully worked out, not a sketch: IRSA for Triton’s S3 model access, HPA and Cluster Autoscaler on both the GPU (g5.xlarge/A10G, or G7e for more throughput) and CPU node groups, a scheduled GPU shutdown to control cost, and a documented estimate of ~\(592/month for the demo config versus ~\)1,080/month running the GPU node 24/7. The catch that matters before you take this anywhere near production: the DLRM/Wide&Deep/NCF models are exported with seeded random weights specifically to exercise the GPU inference path — they don’t make meaningful predictions until you train them on your own bidstream data, and the repo is explicit that there’s no portable pretrained checkpoint to drop in since the embedding tables are keyed to your feature vocabulary. Treat this as a reference architecture for wiring GPU inference into an ARTF-compliant bidding pipeline on EKS, not a bidding model you can deploy as-is.

1. Announcements 📢

📢Amazon EKS adds Kubernetes version rollbacks

Open-source Kubernetes has never supported control plane rollback — once you upgrade, you’re committed, which is exactly why teams running hundreds of clusters build elaborate bake periods, staggered rollout groups, and months-long upgrade cycles instead of just upgrading and checking. The community’s KEP-4330 emulated-versions effort addresses this by keeping a cluster in a transitional compatibility state, but EKS’s version rollback does something stronger: it returns the cluster to the actual previous version that ran in production, one minor version at a time, with cluster insights automatically checking node version compatibility and add-on dependencies before the rollback proceeds (or --force to skip the checks if you’ve already assessed the risk yourself).

The mechanics: control plane rollback took about 20 minutes in AWS’s own walkthrough, comparable to a standard upgrade, and the cluster stays functional throughout. EKS Auto Mode clusters get node rollback too, but since node rollback respects pod disruption budgets by default, that step can take considerably longer depending on how tight your PDBs are — a new cancel API lets you halt a node rollback mid-flight if it’s taking too long, so you can loosen disruption budgets and restart rather than being stuck waiting. It’s available now at no additional cost across all commercial regions, for both self-managed and Auto Mode clusters, on any version in EKS standard or extended support — worth updating your upgrade runbooks around immediately, since the actual risk calculus on “should we upgrade this quarter” just changed for any team that’s been delaying purely on rollback-safety grounds.

📢HAMi advances to CNCF Incubating

HAMi gives Kubernetes fractional GPU allocation, hard memory and compute isolation, and device-aware scheduling across NVIDIA, AMD, and Ascend hardware — the multi-vendor accelerator layer that’s been missing since GPUs got treated as indivisible bricks rather than a resource you can slice. Incubating status means it’s cleared CNCF’s due-diligence bar on real production adopters, a healthy contributor base, and a security posture that stands up to review.

Read this alongside DRA reaching GA in the same window: HAMi sits directly on top of that scheduling primitive, so together they’re the closest thing Kubernetes has right now to a real, open, production-vetted accelerator-scheduling stack. For anyone running heterogeneous GPU fleets on EKS, this is the project to get hands-on with before building your own fractional-GPU glue code from scratch.

Community & Career 🤝

🤝Shubham Katara: Controlling MCP tools with agentgateway on Kubernetes

The MCP registry closing in on 10,000 tool-server records turned the open question from “can an agent connect to this tool” into “should it, and who decides” — a governance gap with no answer until now. Katara’s guide stands up agentgateway, the open-source, Rust-based, Linux Foundation proxy, on a kind cluster, then routes LLM and MCP traffic through it so agents hold zero real credentials directly; the gateway injects secrets at call time instead.

The mechanism worth copying is the RBAC layer: policies written as CEL expressions make forbidden tools simply invisible to the agent — allow get_me, hide search_repositories — without touching a line of agent code. For any EKS team running agent fleets past a handful of tools, this is the pattern: discover through the registry, enforce through the gateway, keep credentials and tool-access policy in the platform layer instead of scattered across a dozen agent configs.

🤝Plasticity.Cloud & Codriverlabs ship KubeMicroVM v1.0.0

KubeMicroVM brings AWS Lambda MicroVMs into the Kubernetes resource model as first-class CRDs — MicroVM, MicroVMImage, MicroVMReplicaSet — instead of the usual approach of standing up Firecracker or Kata isolation as bespoke cluster plumbing. The auth design is the most interesting part: pods get MicroVM tokens through a sidecar that does a two-step Kubernetes TokenReview + SubjectAccessReview before ever calling AWS, so app containers never hold AWS credentials directly, and tokens live in emptyDir medium: Memory rather than on disk. Built on Quarkus 3, JOSDK, and GraalVM native image on Java 25, with a namespace-labeling model (lambda.aws.amazon.com/manage-microvms=true) that keeps the operator scoped to opted-in teams by default.

It’s early — 14 stars, one fork, v1.0.0 tagged this week — and there’s no published benchmark behind the cold-start claims in the launch announcement, so treat “sub-second” as a number to verify yourself rather than a given. Worth a look if you’re weighing MicroVM-based isolation for agent code execution on EKS specifically and want Kubernetes-native CRDs instead of hand-rolled Firecracker orchestration; also worth knowing before you build on it that it ships under Elastic License 2.0, not a fully permissive OSI license, so check that against your use case before betting production infra on it.

🤝Frank Fan et al.: APEX Skills — agentic platform engineering for EKS

Checking whether a cluster is safe to upgrade today means running a deprecated-API scanner like kubent or eksup, cross-checking addon compatibility by hand against the EKS release notes, then separately eyeballing node health and workload disruption risk — three or four tools, stitched together manually, before you write a single line of the actual upgrade plan. APEX packages that whole sequence as agent skills instead: Recon reads the live cluster state, Upgrade Check runs the deprecated-API and addon-compatibility scoring against it (0-100 across 8 categories), and the bundled MCP server lets the same agent session act on what it finds — check upgrade insights, inspect a flagged pod, remediate — without switching tools or re-explaining cluster context each time.

The catch is exactly what determines whether that’s actually useful or just a faster way to get a wrong answer: an agent-generated 0-100 upgrade score is only as trustworthy as the checks behind it, and none of those individual checks are new capability — kubent and eksup already do deprecated-API and compatibility scanning well on their own. What APEX adds is the orchestration layer: “steering workflows” chain the checks in a fixed order with handoff guardrails, so the agent doesn’t skip straight to writing a remediation plan before recon has actually run. Before trusting this on a cluster you don’t already know cold, run it once against a cluster you do and compare its score against what you already know is broken. Install via npx apex-skills.

🤝defilantech ships LLMKube — a Kubernetes operator for heterogeneous local LLM inference

LLMKube’s Model and InferenceService CRDs turn “run an LLM on Kubernetes” into a two-line YAML deploy, but the standout piece is the Metal Agent: since Apple’s Metal GPU isn’t reachable from inside a container, it runs as a native macOS process outside Kubernetes, spawning llama-server with full GPU access and registering the result back as a normal cluster Endpoint — letting one fleet span NVIDIA cloud nodes and on-prem Macs, something every comparable tool in its own comparison table (KubeAI, llm-d, LocalAI) can’t do.

The benchmarks are real: an NVIDIA L4 on GKE goes from 4.6 tok/s CPU to 64 tok/s GPU (17x), and a dual RTX 5060 Ti setup holds ~53 tok/s across 3B-8B models via automatic layer sharding. Worth knowing the design choice in ModelRouter, its cloud-offload piece: PII/PHI routing rules are rejected at apply-time if they’d ever reach a cloud backend, and if the local pool can’t serve a regulated request at runtime, it returns 503 rather than falling through to cloud — fail-closed by construction. There’s also a real OpenShift fix in the docs: the default fsGroup: 102 breaks under restricted-v2 SCC, so the OpenShift preset disables it and lets the SCC inject the namespace’s allocated value instead — a fix that only comes from actually running this on OpenShift.

One flag: Foreman, its newest piece (agentic coding across a fleet of local models), is opt-in and v0.1 — a fun bootstrap signal that its own agents authored real PRs against the repo, but early compared to the core operator’s 89-release track record.

1. Highlights ✨

✨Carlos Santana: Agent sandboxes on EKS webinar — July 15

Carlos Santana (AWS Sr. Specialist SA for EKS) and Matt Meckes are hosting a live session on the threat model and governance patterns for running AI agents and agent-generated code on Kubernetes — compute isolation, network egress control, identity/credential scoping, and the emerging split where a model provider handles orchestration while execution stays on your own infrastructure. Timely alongside this edition’s KubeMicroVM pieces, which are two different concrete answers to the same isolation question this webinar is framing at the platform level.

✨Kepler re-architected for power-accurate measurement

The CNCF project for per-workload energy consumption shipped a re-architecture aimed specifically at measurement accuracy — the original approach leaned on RAPL/eBPF estimation that got noisier under mixed CPU/GPU load, which is exactly the workload shape most EKS AI clusters run. With GPU electricity now the binding cost constraint on AI platforms rather than the GPU itself, accurate watts-per-pod turns Kepler from a sustainability dashboard into a real input for bin-packing and cost-allocation decisions. A community call for testers is open now if you want to validate it against your own cluster’s power draw before it lands in your cost model.

✨Dragonfly v2.5.0 pulls models directly from Hugging Face

The CNCF P2P distribution project added direct repository downloads from Hugging Face and ModelScope, feeding straight into its existing tiered seed-peer caching instead of requiring a separate pull-and-republish step. That matters at EKS scale specifically because a multi-gigabyte model rollout to hundreds of nodes normally means every node hitting the source repo simultaneously — Dragonfly’s P2P layer turns the first few nodes to pull into seed peers for the rest, so the source repo sees one download instead of one per node.

✨EdgeBench: a scaling law for how agents learn from their environment

ByteDance Seed’s EdgeBench measures agent improvement over time-on-task rather than capability at t=0 — 38,000 hours of runs across 134 real-world tasks (science, SWE, formal math, games), each lasting 12 to 72 hours. Individually noisy learning curves collapse into a clean log-sigmoid pattern with a mean R² of 0.998, and the team’s graph-exploration framing suggests agents are improving from environment interaction roughly 2x every three months. 51 of the 134 tasks plus the eval framework are public. Outside the Kubernetes/EKS scope most of this newsletter covers, but worth knowing if you’re evaluating whether longer agent time-budgets are worth the compute on any platform.

✨GLM-5.2 deployment example added to AWS SageMaker GenAI hosting examples

AWS’s community-maintained SageMaker inference hosting repo (142 stars, 754 commits, actively updated) now includes a GLM-5.2 deployment notebook under its 01-models collection, alongside existing examples for Llama, Mistral, and Swiss AI’s Apertus. The repo’s broader pattern is worth knowing regardless of this specific notebook: it walks through optimizing LLM hosting on SageMaker Inference across model servers like DJL Serving and TGI, including hardware options like Inferentia2 — so it’s a reasonable place to start if you’re weighing GLM-5.2 for SageMaker deployment.

🎉 Sponsor Section

At the moment, we don’t have a sponsor for this edition, but we look forward to working with companies and organizations that support the EKS & AI Infrastructure community in future editions. If you or your company is interested in sponsoring, please contact us at 📧 thecloudtechforall@gmail.com

📝 Words from the Author

I watched Ryo Lu’s talk this week — “Closer to the Material” — and I haven’t stopped thinking about it.

Ryo is Cursor’s head of design, and the thing he’s wrestling with isn’t really about design software. It’s about what happens to craft when the tool between your idea and the finished thing gets smart enough to just do it for you. He built ryOS, this whole retro operating system, almost entirely inside Cursor — and the point he keeps circling back to is that AI didn’t remove him from the material, it moved him closer to it. He stopped painting in Figma and started sculpting in code directly, poking and prodding until something that started out ugly sharpened into something that felt inevitable in hindsight.

What got me is the question he asks almost in passing: what should exist, and what does AI change about the loop of finding that out? Because the actual creative loop — try something, feel whether it’s right, adjust — used to be gated by how fast you could produce the next attempt. When the gate opens that wide, the bottleneck isn’t execution anymore. It’s taste. It’s knowing, in your gut, when the thing in front of you is actually good, fast enough to keep up with a tool that can generate ten more versions before you’ve finished the sentence.

I think about this constantly with infrastructure, weirdly. We’re not that different from Ryo’s designers right now — agents can generate a Terraform plan, a Helm chart, an upgrade runbook faster than any of us can review it properly. The instinct that catches “this config is wrong” before you’ve consciously reasoned through why — that’s the same kind of taste he’s talking about, just for YAML instead of pixels. AI hasn’t made that instinct obsolete. If anything, it’s made it the only thing standing between a fleet of agents shipping something plausible and shipping something actually right.

Go watch it if you build anything at all. It’s not really about design software. It’s about what’s worth protecting when the tool stops being the bottleneck.

Happy building. 😎

Most resources on running AI in production either assume you're a data scientist who picked up Kubernetes last week, or a platform engineer who treats models as black boxes and hopes for the best. Very few sit in the uncomfortable middle — the place where most production engineering problems actually live.

Generative AI on Kubernetes, published by O'Reilly, sits in that middle. And it does it well enough that I want to write down why.

What the Book Is Actually About

The book is structured around four phases of operationalizing LLMs on Kubernetes: inference, production readiness, tuning, and AI-driven applications. That progression is intentional and reflects how most teams actually adopt AI — starting with serving a pre-trained model, then discovering that keeping it running reliably at scale is an entirely different problem.

It doesn't try to teach you machine learning. It explicitly treats LLMs as operational dark-gray boxes — you manage their infrastructure, resources, and lifecycle without diving into neural network mathematics. That framing is honest and useful. But it does give you just enough of the internals to make operational decisions informed ones.

The primer on LLM fundamentals in the introduction — tokenization, embeddings, the prefill and decode phases, compute-bound versus memory-bound workloads — is one of the better short explanations I've read. It's not dumbed down. It's scoped correctly. You close that section understanding why TTFT and TPOT are the metrics that matter, not just that they matter.​

The Model Data Chapter Is Underrated

Most practitioners skip straight to serving configuration. The model data chapter deserves more attention than it gets.

The progression from emptyDir volumes and init containers, through PersistentVolumes with ReadOnlyMany access mode, all the way to OCI image volume mounts and modelcars is the first time I've seen this problem laid out with the actual trade-offs made explicit.​

The core tension is this: PersistentVolumes backed by NFS or distributed filesystems give you storage efficiency when running many replicas of the same model — one copy, many readers. But they introduce network latency on every model weight read, and at high replica counts, the storage backend becomes the bottleneck. Local approaches like OCI volumes eliminate that latency but require a copy per node.​

The book doesn't pretend there's a universally correct answer. It gives you the framework to make the right call for your workload — replica count, inference throughput requirements, network bandwidth, and whether you're running GPU-based or CPU-based inference. That's more useful than a recommendation.

The section on the Kubeflow Model Registry and OCI artifacts for storing model weights is particularly forward-looking. Treating a model file the same way you treat a container image — versioned, immutable, distributable via a registry — is the direction the ecosystem is heading, and the book explains the mechanics clearly.​

Observability Is Where Most Teams Are Underinvested

The observability chapter reframes something that most platform engineers miss until they've already been burned.

Monitoring an LLM workload is not the same as monitoring a microservice. CPU and memory utilization tell an incomplete story. The primary compute resource is the GPU, and the two inference phases — compute-bound prefill and memory-bound decode — have fundamentally different performance characteristics that traditional request-rate metrics won't surface.​

The key metrics the book covers:

• Time To First Token (TTFT) — the user-perceived wait before the first token arrives; maps directly to the prefill phase; exposed in vLLM as vllm:time_to_first_token_seconds

• Time Per Output Token (TPOT) — the speed of generation as seen by the user; at least 4–5 tokens per second keeps pace with human reading speed; exposed as vllm:time_per_output_token_seconds

• KV cache utilization — because the decode phase is memory-bound, KV cache pressure is often where throughput collapses before any other metric signals a problem

• Request queue depth — vllm:num_requests_waiting tells you when the batch is full and requests are backing up before the system appears saturated by any other measure​

The book also covers GPU-specific metrics via NVIDIA DCGM, AMD ROCm SMI, and Intel XPU Manager — each exposing hardware metrics through Prometheus exporters. The point worth internalizing: there is no common naming convention across vendors, which means your observability setup needs to account for the specific hardware you're running.​

The section on quality metrics and responsible AI — hallucination detection, bias monitoring via TrustyAI, LLM-as-a-judge patterns for asynchronous quality evaluation — is more advanced than I expected from a Kubernetes operations book, and it's the right kind of advanced. These aren't afterthoughts. They're treated as first-class production concerns.​

Disaggregated Serving — The Most Forward-Looking Section

If you only read one advanced section, read the disaggregated serving chapter.

The core idea: prefill is compute-bound, decode is memory-bound. They have different hardware requirements, different scaling characteristics, and different failure modes. Running them on the same pool of instances is a compromise that makes neither phase optimal. Disaggregated serving splits them into separate hardware pools so each can be tuned and scaled independently.​

The full stack the book describes — Gateway API Inference Extension, distributed KV cache via LMCache, NIXL for point-to-point memory transfer, llm-d for orchestration, KServe LLMInferenceService as the management layer — is genuinely cutting-edge. It's the direction production inference at scale is heading, and the book explains the infrastructure implications clearly.​

The network bandwidth requirement alone is worth understanding: traditional pod networking at 10–20 Gbps is insufficient for sharing KV cache blocks between prefill and decode instances at production scale. The requirement is ~500–600 Gbps — an order of magnitude higher — which means RDMA, RoCE, NVLink, or InfiniBand configurations that were previously only seen in HPC environments.​

This isn't configuration you'll deploy next week. But understanding the architecture now means the decisions you're making today about GPU node topology, networking, and storage don't create unnecessary migration costs later.

Where the Book Has Gaps

The honest gaps, for teams running on AWS:

Karpenter — the book covers cluster autoscaler and GPU node management, but Karpenter's NodePool and EC2NodeClass patterns for GPU workloads, and its interaction with LLM-specific autoscaling signals like TTFT and KV cache utilization, are not covered.

AWS-specific hardware — Trainium and Inferentia2 are absent. Teams running fine-tuning on trn1 instances or inference on inf2 will need to look elsewhere for the operational patterns specific to Neuron-based workloads.

EKS-specific tooling — ACK, KRO, Pod Identity, and the patterns for managing AWS resources alongside Kubernetes workloads are outside scope. That's fair — the book is genuinely cloud-agnostic — but it means AWS practitioners need a second reference.

These gaps don't undermine the book. They define its scope. For the foundational mental model of GenAI infrastructure on Kubernetes — model serving, production operations, tuning, and AI-driven applications — it covers the ground it sets out to cover, and covers it well.

Who Should Read This

• Platform engineers building out GPU infrastructure and trying to understand what's different about LLM workloads before something breaks in production

• MLOps engineers operationalizing model serving and discovering that Kubernetes adds a layer of complexity that most ML tutorials skip entirely

• Cloud architects designing the infrastructure layer for GenAI products who need a structured mental model before making hardware and networking commitments

It's not a quick read. It rewards attention. And it's the kind of book you close thinking about the decisions you're making differently — which is the only standard that matters.

Contents

1. The Two Nightmares — A Tale of Two Incidents

2. The Old Model: GPU Extended Resources Were Always a Hack

3. DRA: GPUs as First-Class Structured Resources

4. AICR: Treating the GPU Stack Like Software

5. Putting It Together: The Full EKS GPU Platform Lifecycle

6. Full Manifests and Deployment

7. The Governance Angle

Part One

The Two Nightmares — A Tale of Two Incidents

Every GPU platform engineer running distributed training on EKS has two recurring nightmares. They don't announce themselves with clean error messages. They cost days to diagnose. And until this month, Kubernetes had no structural answer to either of them.

Incident A — The Scheduling Nightmare

It is 2 AM on a Tuesday. A team's 32-GPU gang job has been in Pending for six hours — not waiting for GPUs to free up, but deadlocked. The scheduler sees 32 available nvidia.com/gpu tokens distributed across four nodes and happily assigns 8 pods. Except GPUs 0–3 on Node A and GPUs 4–7 on Node B are on separate NVLink islands. The two groups cannot communicate at NVLink bandwidth. NCCL probes, times out, and the job silently stalls.

The fix is a 3-hour archaeology session through node labels, custom topology plugin configs, and a hand-edited pod affinity rule that will be wrong again after the next node group refresh.

Incident B — The Drift Nightmare

Three months pass. A 16-node NCCL allreduce workload that has been running perfectly starts hanging after what the ops team describes as "an unrelated EKS node group AMI refresh." No error. Just hangs at the collective barrier.

The postmortem, five days later, reveals that the new AMI shipped NCCL 2.21 while the GPU Operator container toolkit was still built against 2.19. The mismatch is invisible to Kubernetes. No alert fires. No readiness probe catches it. The cluster looks healthy because its nvidia.com/gpu count is exactly right.

These two incidents share a root cause. Kubernetes was never designed to understand GPU hardware topology, and nobody was systematically tracking what combination of GPU software was actually running.

NVIDIA announced two projects at KubeCon 2026 that address this directly: Dynamic Resource Allocation (DRA) — donated to the Kubernetes upstream and CNCF — and AICR (AI Cluster Readiness), a validation framework for maintaining known-good GPU software stacks. DRA solves the topology blindness. AICR solves the configuration drift.

INCIDENT A Scheduling Nightmare Scheduler sees GPUs as opaque integers. No NVLink awareness. Gang job deadlocks silently.

Fig 1 — Two unsolved problems, two new primitives

The rest of this post is a deep dive into both. By the end, you'll have the mental model, the architecture diagrams, and the full manifests to implement both on EKS 1.31+.

Part Two

GPU Extended Resources: Why the Old Model Was Always a Hack

To understand why DRA matters, you need to understand exactly what nvidia.com/gpu: 1 means to the Kubernetes scheduler today — and more importantly, what it cannot mean.

Extended resources are opaque integers. When the NVIDIA device plugin runs on a node and advertises nvidia.com/gpu: 8, it is telling Kubernetes: "this node has 8 units of a countable resource called nvidia.com/gpu." The scheduler's job becomes: find a node where the sum of unallocated units of that resource ≥ what the pod requested. That's the entire model.

Fig 2 — The scheduler sees 6 GPUs as an opaque integer. NVLink topology is invisible.

What the Scheduler Cannot See

The extended resource model is blind to everything that actually matters for GPU workload placement:

The consequence is that every "smart" GPU scheduling feature built on top of the existing model — topology-aware scheduling via node labels, MIG selection via custom device plugins, gang scheduling via custom scheduler plugins — is a workaround piled on top of a fundamentally limited primitive. You're building a load-bearing structure on a foundation that was never designed to carry the weight.

The extended resource model asks the scheduler to make hardware-topology-aware decisions with hardware-topology-blind data. That's not a configuration problem. That's an architectural gap.

This is what DRA is designed to close.

Part Three

DRA: GPUs as First-Class Structured Resources

Dynamic Resource Allocation landed as a Kubernetes alpha in 1.26, reached beta in 1.30, and is production-stable on EKS 1.31+. NVIDIA's contribution is the DRA driver — a structured device plugin that exposes each GPU not as a counter increment, but as a rich object with topology, health, and capability attributes that the scheduler can reason over natively.

The New Primitives: ResourceClaim and ResourceClass

DRA introduces two new API objects. A ResourceClass is cluster-scoped and defines the driver and parameters for a category of resource. A ResourceClaim is namespace-scoped and represents a specific workload's request for devices that satisfy given constraints.

# ResourceClass — cluster-scoped, created by the platform team
apiVersion: resource.k8s.io/v1alpha2
kind: ResourceClass
metadata:
  name: nvidia-gpu-nvlink
driverName: gpu.nvidia.com
parametersRef:
  apiGroup: gpu.nvidia.com
  kind: GpuAllocationParameters
  name: nvlink-connected-4x
---
# GpuAllocationParameters — driver-specific constraints
apiVersion: gpu.nvidia.com/v1alpha1
kind: GpuAllocationParameters
metadata:
  name: nvlink-connected-4x
spec:
  count: 4
  selector:
    nvlink: required       # all 4 must share an NVLink domain
    health: healthy       # exclude GPUs with XID errors
    minMemory: 40Gi      # minimum HBM per GPU

# ResourceClaim — namespace-scoped, requested by the workload
apiVersion: resource.k8s.io/v1alpha2
kind: ResourceClaim
metadata:
  name: training-gpu-claim
  namespace: ml-team
spec:
  resourceClassName: nvidia-gpu-nvlink

A PodSpec then references the claim instead of listing resource requests:

spec:
  resourceClaims:
  - name: gpus
    source:
      resourceClaimName: training-gpu-claim
  containers:
  - name: trainer
    image: nvcr.io/nvidia/pytorch:24.02-py3
    resources:
      claims:
      - name: gpus  # bind the claim to this container

The GpuAllocationState: What the DRA Driver Actually Exposes

The NVIDIA DRA driver runs as a DaemonSet on each GPU node and populates a GpuAllocationState custom resource per-node. This is the structured data the scheduler consumes when evaluating claims.

Fig 3 — The DRA driver exposes per-GPU structured state. The scheduler consumes it natively.

CompositeDevice: Claiming Groups Atomically

One of the most important DRA concepts for distributed training is the CompositeDevice — the ability to claim a group of GPUs as a single atomic unit. This is the primitive that eliminates gang scheduling deadlocks.

When a ResourceClaim requests 4 NVLink-connected GPUs, the DRA driver evaluates the GpuAllocationState and creates a CompositeDevice if and only if it can find 4 GPUs that all meet the constraints simultaneously. The scheduler then either places the pod on that node with those 4 GPUs, or it doesn't place it at all. There is no partial allocation, no "I'll take 2 here and 2 somewhere else."

Key insight: With extended resources, the scheduler allocates GPUs one at a time, on a first-come-first-served basis per pod. Deadlocks happen when multiple jobs each grab some GPUs and wait for the rest. DRA's CompositeDevice makes multi-GPU allocation atomic — you get all-or-nothing, which eliminates the deadlock condition entirely.

MIG Support Under DRA

MIG (Multi-Instance GPU) partitioning on A100 and H100 becomes dramatically more usable under DRA. Today, MIG configuration requires pre-partitioning nodes with specific profiles and adding custom labels — a static, manual process that doesn't adapt to workload demand.

Under DRA, a workload can request a specific MIG profile via the GpuAllocationParameters, and the DRA driver can dynamically partition a GPU into the requested profile at pod scheduling time, then restore it after the pod exits. This is still in alpha, but the architecture is sound and the trajectory is clear.

# Request a single MIG 3g.40gb slice
apiVersion: gpu.nvidia.com/v1alpha1
kind: GpuAllocationParameters
metadata:
  name: mig-3g-40gb
spec:
  mig:
    profile: 3g.40gb    # 3 compute slices, 40GB HBM
    placement: auto   # driver selects least-fragmented placement

Migrating from Extended Resources on EKS 1.31+

EKS 1.31 enables DRA by default via the DynamicResourceAllocation feature gate. You do not need to migrate all workloads at once — extended resources and DRA claims coexist on the same cluster and even the same node during transition.

The recommended migration order:

1. Install the NVIDIA DRA driver alongside (not replacing) the existing device plugin. Both can run simultaneously.

2. Start with new workloads that have strict topology requirements — distributed training jobs, multi-node inference. These get the most immediate benefit.

3. Create ResourceClass definitions for each workload profile (single GPU, NVLink-4x, MIG slice, etc.).

4. Migrate batch jobs before interactive/notebook workloads. Batch jobs are easier to restart if something goes wrong.

5. Once all workloads use DRA claims, remove the device plugin DaemonSet and the nvidia.com/gpu extended resource annotations.

What breaks during migration: Any tooling that reads nvidia.com/gpu from pod specs (cost allocation tools, custom autoscalers, some monitoring dashboards) will need updates. DRA claims are in spec.resourceClaims, not spec.containers[*].resources.limits. Audit your toolchain before cutting over.

Part Four

AICR: Treating the GPU Stack Like Software

DRA solves the scheduling problem at cluster bootstrap and workload placement time. But it does nothing to prevent Incident B — the silent drift that happens when a node group AMI refresh quietly changes the NCCL version, or when a Helm chart upgrade bumps the GPU Operator toolkit in a way that's incompatible with the kernel driver already installed on the node.

AICR (AI Cluster Readiness) is NVIDIA's answer to this. It is not a monitoring tool. It is not an alerting system. It is a conformance framework — a way to define what "correct" looks like for your GPU software stack, snapshot the actual state of your cluster, and produce a machine-readable diff that tells you exactly where you've drifted.

What a Validated Recipe Contains

An AICR recipe is a version matrix — a YAML document that specifies the exact combination of GPU software components that NVIDIA has tested together and validated for a specific hardware target. Think of it as a lock file for your GPU stack, equivalent to a requirements.txt with pinned hashes.

Fig 4 — An AICR recipe pins every layer of the GPU software stack

The aicr validate Workflow

Validation is a three-step process. AICR deploys a DaemonSet of probe pods that inspect each node's GPU software stack, collects the results centrally, diffs them against the target recipe, and produces a structured report.

# Install AICR on your EKS cluster
helm upgrade --install aicr \
  oci://nvcr.io/nvidia/aicr/aicr-operator \
  --version 1.2.0 \
  --namespace nvidia-system \
  --create-namespace

# Apply your target recipe
kubectl apply -f h100-sxm-eks-1.31.yaml

# Run validation against all GPU nodes
aicr validate \
  --recipe h100-sxm-eks-1.31 \
  --node-selector accelerator=nvidia-h100 \
  --output-format sarif

The output is a SARIF (Static Analysis Results Interchange Format) report — the same format used by GitHub Advanced Security, which means it can be consumed directly by your existing security pipeline and surfaced as PR annotations:

{
  "version": "2.1.0",
  "runs": [{
    "tool": { "driver": { "name": "aicr" } },
    "results": [
      {
        "ruleId": "AICR-ML-001",
        "level": "error",
        "message": {
          "text": "nccl version mismatch: expected 2.21.5-1, found 2.19.3-1 on node ip-10-0-4-23"
        }
      }
    ]
  }]
}

GitOps Integration: Recipe as Source of Truth

The operational power of AICR comes when you wire it into your GitOps pipeline. The recipe lives in Git, ArgoCD watches for drift, and new NVIDIA-validated recipes trigger automated pull requests against your cluster configuration.

Fig 5 — AICR recipe as GitOps source of truth. New NVIDIA recipes trigger automated PRs; ArgoCD reconciles.

AICR and KARs Conformance

CNCF launched the Kubernetes AI Requirements (KARs) certification program in March 2026, nearly doubling the number of certified Kubernetes AI platforms in its first month. AICR's validation output is directly consumable as KARs conformance evidence — running aicr validate produces the artifacts that the KARs submission process requires for GPU stack certification.

For enterprises pursuing multi-cloud or multi-vendor GPU strategies, this matters: a cluster that passes AICR validation against a published recipe is also KARs-compliant by construction, giving you a vendor-neutral attestation of your GPU platform's readiness.

EKS-specific integration: AICR recipes include a field mapping to EKS Optimized AMI versions. aicr validate will flag if your node group's launch template references an AMI whose NCCL or driver version is out of alignment with the recipe — catching the exact failure mode of Incident B before it reaches production.

Part Five

Putting It Together: The Full EKS GPU Platform Lifecycle

DRA and AICR are not independently optional — they compose into a complete operational story. Here is what the full lifecycle looks like when both are in place.

Fig 6 — Full EKS GPU platform lifecycle with DRA + AICR

Stage 1: Cluster Bootstrap from Recipe

The cluster provisioning pipeline (eksctl, Terraform, or CDK) is parameterized by an AICR recipe. The recipe determines: which EKS Optimized AMI version to use for GPU node groups, which GPU Operator Helm chart version to install, which DRA driver version to deploy, and which NCCL version to bake into the base container image layer.

The result is a cluster whose initial state is fully specified, reproducible, and integrity-checked. Every node boots with exactly the software stack the recipe declares. aicr validate on day zero returns zero findings.

Stage 2: Workload Scheduling with DRA + KAI Scheduler

KAI Scheduler (also donated to CNCF at KubeCon 2026) extends the Kubernetes scheduler framework with gang scheduling semantics that are DRA-aware. When a PyTorchJob requests 4× NVLink-connected H100s per worker, KAI Scheduler creates the ResourceClaims, waits for all of them to be satisfiable simultaneously, and only then begins binding pods. No partial placements, no deadlocks.

Stage 3: Continuous Validation in CI/CD

AICR validation runs as a CI/CD job — either on a schedule (every 6 hours) or triggered by node group events (new node joins, AMI refresh, Helm upgrade). The SARIF output feeds into your existing security findings pipeline. Any finding with severity error blocks a deployment promotion gate.

Stage 4: Upgrade Path

When NVIDIA publishes a new validated recipe, an automated PR is opened against your infra repo. The PR updates the recipe file, bumps the GPU Operator chart version, and updates the AMI parameter. The merge is gated on AICR conformance passing in staging. After merge, ArgoCD rolls out the GPU Operator upgrade, eksctl updates the launch template and triggers a rolling node group refresh, and AICR runs a final conformance check in production before the deployment is marked complete.

Part Six

Full Manifests and Deployment

1. DRA ResourceClass and ResourceClaim for a PyTorchJob

# platform/dra/gpu-classes.yaml
apiVersion: resource.k8s.io/v1alpha2
kind: ResourceClass
metadata:
  name: h100-nvlink-8x
  annotations:
    gpu.nvidia.com/description: "8× H100 SXM5, all NVLink-connected, full node"
driverName: gpu.nvidia.com
parametersRef:
  apiGroup: gpu.nvidia.com
  kind: GpuAllocationParameters
  name: h100-nvlink-8x-params
---
apiVersion: gpu.nvidia.com/v1alpha1
kind: GpuAllocationParameters
metadata:
  name: h100-nvlink-8x-params
spec:
  count: 8
  selector:
    nvlink: required
    health: healthy
    model: H100 SXM5
    minMemory: 79Gi
---
apiVersion: resource.k8s.io/v1alpha2
kind: ResourceClass
metadata:
  name: h100-single
driverName: gpu.nvidia.com
parametersRef:
  apiGroup: gpu.nvidia.com
  kind: GpuAllocationParameters
  name: h100-single-params
---
apiVersion: gpu.nvidia.com/v1alpha1
kind: GpuAllocationParameters
metadata:
  name: h100-single-params
spec:
  count: 1
  selector:
    health: healthy

# workloads/pytorch-distributed/claim.yaml
apiVersion: resource.k8s.io/v1alpha2
kind: ResourceClaimTemplate
metadata:
  name: training-gpus
  namespace: ml-training
spec:
  spec:
    resourceClassName: h100-nvlink-8x
---
# PyTorchJob referencing the claim template
apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
  name: llama-pretrain
  namespace: ml-training
spec:
  pytorchReplicaSpecs:
    Worker:
      replicas: 16          # 16 workers × 8 GPUs = 128 H100s
      restartPolicy: OnFailure
      template:
        spec:
          resourceClaims:
          - name: gpus
            source:
              resourceClaimTemplateName: training-gpus
          containers:
          - name: trainer
            image: nvcr.io/nvidia/pytorch:24.02-py3
            resources:
              claims:
              - name: gpus
            env:
            - name: NCCL_DEBUG
              value: "INFO"
            - name: TORCH_DISTRIBUTED_DEBUG
              value: "DETAIL"

2. AICR Installation and First Validation Run on EKS

# Add the NVIDIA Helm repo
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

# Install AICR operator
helm upgrade --install aicr-operator nvidia/aicr-operator \
  --version 1.2.0 \
  --namespace nvidia-system \
  --create-namespace \
  --set "nodeSelector.accelerator=nvidia-h100"

# Download recipe for your hardware/EKS version
aicr recipe pull \
  --hardware h100-sxm5 \
  --eks-version 1.31 \
  --output h100-sxm-eks-1.31.yaml

# Apply recipe to cluster
kubectl apply -f h100-sxm-eks-1.31.yaml

# Run initial validation (outputs to stdout + SARIF file)
aicr validate \
  --recipe h100-sxm-eks-1.31 \
  --node-selector accelerator=nvidia-h100 \
  --output-format sarif \
  --output-file aicr-results.sarif \
  --fail-on error

# Upload SARIF to GitHub Advanced Security
gh api \
  --method POST \
  /repos/{owner}/{repo}/code-scanning/sarifs \
  --input aicr-results.sarif

3. ArgoCD ApplicationSet for Recipe-Driven GPU Operator Management

# gitops/applicationsets/gpu-operator.yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: gpu-operator-recipe-driven
  namespace: argocd
spec:
  generators:
  - git:
      repoURL: https://github.com/your-org/platform-config
      revision: HEAD
      files:
      - path: "recipes/*/aicr-recipe.yaml"
  template:
    metadata:
      name: 'gpu-operator-{{cluster_name}}'
    spec:
      project: platform
      source:
        repoURL: https://helm.ngc.nvidia.com/nvidia
        chart: gpu-operator
        targetRevision: '{{kubernetes_components.gpu_operator}}'
        helm:
          values: |
            driver.version: {{kernel_components.nvidia_driver}}
            toolkit.version: {{container_components.nvidia_container_toolkit}}
      destination:
        server: '{{cluster_server}}'
        namespace: gpu-operator
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
        - CreateNamespace=true

4. AICR in a GitHub Actions CI Pipeline

# .github/workflows/aicr-validate.yaml
name: AICR GPU Cluster Conformance
on:
  schedule:
  - cron: '0 */6 * * *'       # every 6 hours
  workflow_dispatch: {}          # manual trigger

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4

    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v4
      with:
        role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
        aws-region: us-east-1

    - name: Update kubeconfig
      run: aws eks update-kubeconfig --name prod-gpu-cluster

    - name: Install AICR CLI
      run: pip install aicr-cli==1.2.0

    - name: Run AICR validation
      run: |
        aicr validate \
          --recipe recipes/h100-sxm-eks-1.31.yaml \
          --node-selector accelerator=nvidia-h100 \
          --output-format sarif \
          --output-file results.sarif \
          --fail-on error

    - name: Upload SARIF results
      uses: github/codeql-action/upload-sarif@v3
      if: always()      # upload even on failure
      with:
        sarif_file: results.sarif

Part Seven

The Governance Angle: Why CNCF Custody Matters

Both DRA (the Kubernetes upstream feature) and KAI Scheduler (donated alongside AICR at KubeCon 2026) are now CNCF-governed. AICR recipes are published openly. This is worth pausing on, because it changes the strategic calculus for GPU platform teams in non-obvious ways.

The old world looked like this: NVIDIA's scheduling intelligence lived in proprietary plugins — the NVIDIA GPU Operator webhook, the topology-aware plugin, the custom device plugin. If you wanted topology-aware GPU scheduling, you got NVIDIA software, full stop. If you ran a multi-cloud GPU strategy across AWS p4d instances and GCP A3 instances, you had two different proprietary plugin stacks to maintain.

The new world is different. DRA is a Kubernetes upstream feature — it works on every conformant Kubernetes distribution, including EKS, GKE, and AKS. KAI Scheduler is CNCF-governed — it will be audited, it will have a vendor-neutral roadmap, and it will not disappear if NVIDIA's strategic priorities shift. AICR recipes are published openly — a third-party validator could certify against them.

For enterprises with multi-cloud GPU strategies or GPU hardware from multiple vendors, CNCF governance turns DRA and KAI Scheduler from NVIDIA features into infrastructure primitives — as stable and vendor-neutral as kube-scheduler itself.

The practical implication: if your GPU platform engineering roadmap currently includes a "lock-in mitigation" work stream around NVIDIA-specific scheduling plugins, you can now retire most of that work. The replacement is not a workaround — it is the upstream.

The KARs Connection

CNCF's Kubernetes AI Requirements certification is the governance structure that ties this together. A cluster that runs the NVIDIA DRA driver, uses KAI Scheduler for gang scheduling, and passes AICR conformance against a published recipe is not just "well-configured" — it is a KARs-certified Kubernetes AI platform with a machine-readable attestation of its GPU stack's readiness. That attestation is increasingly what enterprise procurement and compliance teams will require before approving production GPU workloads.

We are, in other words, watching the GPU infrastructure space mature in real time — from bespoke configurations maintained by heroic individual engineers, to certified, conformance-tested platforms maintained by standard tooling. DRA and AICR are two of the most important steps in that maturation.

Summary

Where to start: If you're running EKS 1.31+ with H100 or A100 nodes, the lowest-risk first step is AICR — install the operator, pull the recipe for your hardware, run aicr validate, and see what it finds. You'll likely discover at least one version mismatch you didn't know about. That's the value prop in one command.

This is the core insight behind llm-d, and it's deeper than "disaggregated inference." Every discussion about llm-d that focuses on prefill-decode separation is missing the more fundamental architectural shift: llm-d introduces a completely new Kubernetes resource model for inference — one built around the idea that the routing layer needs to know what's inside GPU memory before it decides where to send a request.

On March 15, 2026, AWS announced a joint effort with the llm-d community to bring this to EKS and SageMaker HyperPod, including a new AWS-optimized container image with native EFA and NIXL integration. This post is the deep technical guide to what llm-d actually is, why it exists, and how to deploy the full stack — intelligent routing, disaggregated serving, and tiered KV cache — on EKS today.

Part 1: The Cost of Not Knowing What's in GPU Memory

Before anything else, let's quantify the problem that llm-d solves — because it's easy to dismiss as an optimization detail until you do the arithmetic.

How vLLM Prefix Caching Works (And Why It Breaks at Scale)

A single vLLM instance has a radix-tree based KV cache. When request A comes in with a 2,000-token system prompt, vLLM computes the KV values for those 2,000 tokens and caches them, keyed by a block hash of the token sequence. When request B comes in with the same 2,000-token system prompt plus a new user message, vLLM detects the prefix match, skips recomputation of the 2,000 cached tokens, and only computes the new tokens. TTFT on request B is dramatically lower — on the order of 88% faster for warm cache hits.

This works perfectly on one instance. The moment you scale to a fleet, it stops working:

The cache hit rate of the fleet as a whole is a function of routing, not just caching. A fleet with 8 pods and round-robin routing has an effective cache hit rate of roughly 1/8th of what a single pod would achieve — even though every pod has prefix caching enabled.

The Math: What This Costs on EKS

For a RAG workload with a shared 6,000-token document context (typical for enterprise knowledge base queries):

• KV cache prefill time for 6,000 tokens on H100: ~180ms per request

• Cache hit TTFT: ~22ms (skip prefill entirely, decode starts immediately)

• Cache miss TTFT: ~202ms (180ms prefill + 22ms decode startup)

• Round-robin hit rate on 8 pods: ~12.5% (1 in 8 requests hits the right pod)

• Effective wasted prefill compute: 87.5% of requests × 180ms = 157.5ms wasted per request on average

On a fleet handling 200 requests per second (a modest production load), that's:

157.5ms × 200 req/sec = 31,500ms of wasted GPU-ms per second
= 31.5 GPU-seconds of waste per real second
= 31.5× GPU waste multiplier on prefill

At p5.48xlarge pricing (\(100.42/hour for 8 H100s = \)12.55/GPU/hour), wasting 31.5 GPU-seconds per real second translates to roughly $110/hour in GPU compute doing nothing but re-deriving knowledge that's already sitting in GPU memory on a different pod.

With llm-d's KV cache-aware routing at 87.4% cache hit rate: that waste drops to 12.6% of requests × 180ms = 22.7ms wasted per request on average — a 7× reduction in prefill waste, and a near-direct translation to cost savings and throughput improvement.

This is not a marginal optimization. It's the difference between a GPU fleet running at 40% effective utilization and one running at 80%.

Part 2: The New Kubernetes Resource Model for Inference

Here's what makes llm-d architecturally interesting beyond the routing algorithm: it introduces three new Kubernetes primitives that don't exist in standard Kubernetes, collectively called the Gateway API Inference Extension (GAIE). These aren't implementation details — they're a new abstraction layer that changes how you think about deploying and managing inference workloads.

The Problem with Existing Kubernetes Abstractions

Standard Kubernetes has Service, Deployment, Ingress, and HTTPRoute. These abstractions are designed for stateless, compute-identical, interchangeable replicas. Any pod in a Service is as good as any other for routing purposes. The load balancer picks based on health, connection count, or pure round-robin.

LLM inference breaks every assumption in this model:

• Replicas are not interchangeable — the pod that has your system prompt cached is categorically better for that request than one that doesn't

• Routing decisions require model-server state — you need to know what's in GPU memory before you route, not after

• Multiple models per infrastructure — a single GPU cluster might serve llama-3.3-70b-fp8, qwen3-32b, and codestral-22b simultaneously with different isolation requirements

• Criticality tiers exist — a production user's query has different SLA requirements than a background batch job

None of the existing Kubernetes abstractions encode these concepts. A Service can't express "route to the pod with the most prefix overlap with this request." An Ingress can't say "this model needs guaranteed capacity, that one gets best-effort." gateway-api-inference-extension.sigs.k8s

The Three New Primitives

1. InferenceModel — What You're Serving

Owned by: AI/ML teams (model owners)
Answers: "What model should this traffic class receive, and with what priority?"

InferenceModel is the separation of what to serve from how to serve it. A model owner declares that gpt-4o-mini-equivalent should be served with Critical priority, referencing a specific model ID on the backend vLLM pods. They don't care how many pods are running or on what hardware — that's the platform operator's concern:

apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferenceModel
metadata:
  name: llama-production
  namespace: ml-serving
spec:
  # The model name clients use in their API calls
  modelName: "meta-llama/Meta-Llama-3.3-70B-Instruct-FP8"

  # Criticality determines routing priority under resource pressure
  # Critical > Standard > Sheddable
  criticality: Critical

  # Points to the InferencePool that serves this model
  poolRef:
    name: llama-pool

2. InferencePool — Where and How You're Serving It

Owned by: Platform operators
Answers: "Which pods form this serving pool, and which routing brain manages them?"

InferencePool replaces the Kubernetes Service as the backend abstraction for inference traffic. It selects pods by label, defines the target port, and — critically — references an extensionRef that points to the External Processing Pod (EPP) that will make routing decisions: gateway-api-inference-extension.sigs.k8s

apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferencePool
metadata:
  name: llama-pool
  namespace: ml-serving
spec:
  # Selects vLLM pods that belong to this pool
  selector:
    matchLabels:
      llm-d.ai/inferenceServing: "true"
      model: llama-3.3-70b

  # Port that vLLM listens on inside pods
  targetPortNumber: 8000

  # The External Processing Pod: the routing oracle
  extensionRef:
    name: llama-epp
    kind: Service
    group: ""

3. External Processing Pod (EPP) — The Routing Oracle

Owned by: Platform (auto-deployed by llm-d Helm chart)
Answers: "For THIS specific request, given the current cache state of all pods, which pod should receive it?"

The EPP is the architectural innovation that makes the entire system coherent. It's a separate pod that sits between the Envoy/Istio gateway and the vLLM pods. It implements the gRPC ext_proc (external processing) protocol, meaning Envoy sends it a request header, the EPP returns a routing decision (which pod to forward to), and Envoy follows it. This happens inline, on the request path, with sub-millisecond decision latency.

The EPP maintains a live, in-memory model of the KV cache state across every vLLM pod in the pool. This is the architectural piece that enables cache-aware routing at scale:

The full architecture, showing how these three primitives compose:

This is a fundamentally different mental model from "pods behind a load balancer." The routing layer is stateful and model-aware. It knows the exact contents of GPU memory on every pod before making a forwarding decision.

Part 3: The KV Cache Indexer — How the EPP Knows What's in GPU Memory

The EPP's KV cache indexer is the technical core of llm-d's intelligent routing. Understanding it explains both the performance numbers and the operational constraints.

Architecture Components

The Write Path: How vLLM Tells the EPP What It Cached

Every vLLM pod in an InferencePool publishes cache lifecycle events over ZMQ to the EPP:

When blocks are evicted from GPU memory (due to LRU pressure), vLLM publishes BlockRemoved events, and the EPP removes those entries from its index. The index stays synchronized with actual GPU memory state in near-real-time.

The Read Path: Scoring Pods for a New Request

When a new request arrives at the gateway:

# Pseudocode: EPP read path (sub-millisecond execution target)

def score_pods_for_request(prompt: str, available_pods: List[str]) -> List[PodScore]:

    # Step 1: Check PrefixStore for cached tokenization
    # (tokenizing 2000 tokens is expensive; cache the result)
    token_ids = prefix_store.get_or_tokenize(prompt)

    # Step 2: Convert tokens to KV block keys
    # Uses IDENTICAL chunking algorithm to vLLM (16 or 128 tokens per block)
    block_keys = token_processor.tokens_to_block_keys(token_ids)

    # Step 3: Score each pod by consecutive prefix match length
    scores = {}
    for pod in available_pods:
        # Count consecutive blocks from the START of the prompt
        # that this pod has cached
        consecutive_hits = 0
        for block_key in block_keys:
            if index.has_block(pod, block_key):
                consecutive_hits += 1
            else:
                break  # Stop at first gap — consecutive prefix only

        scores[pod] = consecutive_hits

    # Step 4: Return pods ranked by score (highest = most prefix cached)
    return sorted(scores.items(), key=lambda x: x [developers.redhat](https://developers.redhat.com/articles/2025/10/07/master-kv-cache-aware-routing-llm-d-efficient-ai-inference), reverse=True)

The "consecutive blocks" criterion is critical. A pod with blocks 0, 1, 2, 3 cached is far more valuable than one with blocks 0, 2, 4, 6 — because vLLM's prefill computation is sequential. A gap in the middle forces recomputation from the gap forward. The EPP scores on the longest unbroken prefix sequence, not total block count.

Part 4: llm-d's Well-Lit Paths — The Four Deployment Patterns

llm-d doesn't give you a single deployment mode. It gives you composable building blocks organized into "well-lit paths" — reference architectures for specific performance goals. Think of these as presets: each path is a curated configuration of vLLM settings + gateway settings + EPP configuration + NIXL settings that has been tested for a specific workload type.

Path 1: Intelligent Inference Scheduling (Start Here)

What it is: KV cache-aware routing only. No prefill-decode disaggregation. All vLLM pods handle both prefill and decode, but the EPP routes cache-intelligently.

When to use: Single-node GPU setups, smaller models (≤ 8B), workloads where TTFT improvement from cache routing is more valuable than disaggregation overhead.

Performance: 87.4% cache hit rate, 88% TTFT improvement on warm requests, 109% higher throughput vs baseline Kubernetes service on shared-prefix workloads.

This is the lowest-friction entry point and delivers immediate value for any RAG or multi-turn workload.

Path 2: Prefill-Decode Disaggregation

What it is: Separate prefill pods and decode pods. Prefill pods process the input and build KV cache; decode pods receive the KV cache via NIXL/EFA and generate output tokens.

When to use: Large models (≥ 70B), long input sequences (≥ 2,000 tokens), high concurrency (≥ 50 concurrent requests), or when output sequence length significantly exceeds input length (reasoning models).

Performance: 70% higher tokens/second vs standard vLLM deployment at concurrency=128, measured with 1K/1K input/output on B200 GPUs.

Path 3: Wide Expert Parallelism (MoE Models)

What it is: Data parallelism + expert parallelism across multiple nodes. Each "expert" in a Mixture-of-Experts model runs on dedicated GPUs; requests are routed to the node set hosting the relevant experts.

When to use: DeepSeek-R1, Qwen3.5-MoE, Kimi K2.5, or any MoE architecture. Without this, MoE models either don't fit in memory or are massively underutilized.

Performance: ~50,000 output tokens/second total throughput on 16P/16D B200 topology.

Path 4: Tiered Prefix Caching

What it is: Three-tier KV cache storage — GPU HBM → CPU DRAM → shared filesystem (EFS/FSx). The llm-d FS backend offloads KV blocks from GPU memory asynchronously to lower tiers, dramatically extending effective cache size.

When to use: Workloads with very large working sets (many unique system prompts, large document corpora in RAG), or when GPU memory pressure is causing cache evictions that hurt hit rates.

Key property: New pods joining the fleet can "hydrate" their cache from the shared filesystem immediately, bypassing the warm-up period that makes pod scaling painful in standard deployments.

Part 5: The AWS Container and EFA Integration

The ghcr.io/llm-d/llm-d-aws container is not just the standard llm-d container with a new tag. It includes:

• EFA userspace libraries (libfabric, aws-ofi-nccl) — required for NIXL's EFA transport path

• NIXL with libfabric plugin — enables GPU-to-GPU KV cache transfer over EFA using RDMA

• GPUDirect RDMA support — allows GPU memory on the prefill pod to be read directly by the decode pod's GPU, bypassing CPU on both nodes

The communication stack for KV transfer in disaggregated mode:

NIXL with libfabric over EFA gives you:

• Zero-copy: GPU memory is DMA'd directly over EFA — no host CPU memory copy

• Kernel bypass: libfabric's RDMA path bypasses the OS network stack entirely

• Sub-microsecond overhead: Measured additional latency from KV transfer on EFA is negligible vs the prefill computation time it replaces

On p5.48xlarge with 32 EFA interfaces and 8 H100s, the recommended allocation is 4 EFA interfaces per GPU pod (32 interfaces / 8 GPUs = 4 per GPU). This matches the ratio needed to saturate H100 SXM's NVLink bandwidth between nodes.

Part 6: Full EKS Deployment Walkthrough

Let's deploy the complete stack. We'll start with intelligent inference scheduling (Path 1), then layer on disaggregation (Path 2).

Prerequisites and Cluster Requirements

# Required tools
aws --version          # AWS CLI v2+
kubectl version        # 1.29+
helm version           # 3.14+
helmfile --version     # 0.162+

# EKS cluster requirements:
# - EKS 1.29+ (tested on 1.32)
# - GPU nodes: g6e.12xlarge (dev) or p5.48xlarge (production)
# - Single AZ deployment for EFA (us-east-1a recommended)
# - aws-load-balancer-controller installed
# - NVIDIA GPU Operator installed (see Part 2 of Slinky+KAI post)

# Verify GPU nodes are ready
kubectl get nodes -l workload-type=gpu-inference
# NAME          STATUS   ROLES    AGE   VERSION
# gpu-node-01   Ready    <none>   24m   v1.32.0-eks-...

Step 1: Install the Gateway Provider (Istio)

llm-d uses the Gateway API Inference Extension, which requires a gateway implementation that supports the ext_proc (external processing) protocol. Istio 1.27+ supports this:

# Clone llm-d repository
git clone https://github.com/llm-d/llm-d.git
cd llm-d

# Install Gateway API CRDs (standard + inference extension)
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.0/standard-install.yaml
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v0.3.0/manifests.yaml

# Verify inference extension CRDs
kubectl get crd | grep inference
# inferencemodels.inference.networking.x-k8s.io
# inferencepools.inference.networking.x-k8s.io

# Install Istio with ext_proc support
cd guides/prereq/gateway-provider
./install-gateway-provider-dependencies.sh
helmfile apply -f istio.helmfile.yaml

# Verify Istio is running
kubectl get pods -n istio-system
# istiod-xxx    1/1  Running

Step 2: Configure the AWS NLB Gateway

# common-configurations/istio.yaml
# Configures the Istio Gateway to use AWS NLB

gateway:
  gatewayClassName: istio
  gatewayParameters:
    accessLogging: false
    logLevel: error
    resources:
      limits:
        cpu: "16"
        memory: 16Gi
      requests:
        cpu: "4"
        memory: 4Gi
  service:
    type: LoadBalancer
  annotations:
    # AWS NLB configuration
    service.beta.kubernetes.io/aws-load-balancer-type: "external"
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    # Health check for vLLM's /health endpoint
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-path: "/health"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "8000"

inferenceExtension:
  flags:
    v: 1   # Verbosity level; increase to 4 for routing decision logs

provider:
  name: istio
  istio:
    destinationRule:
      trafficPolicy:
        connectionPool:
          http:
            # Critical: vLLM requests can be long-lived (streaming)
            # These values must be large enough for your max concurrent requests
            http1MaxPendingRequests: 256000
            maxRequestsPerConnection: 256000
            http2MaxRequests: 256000
            idleTimeout: "900s"   # 15 min: longer than any reasonable inference request
          tcp:
            maxConnections: 256000
            connectTimeout: "900s"

routing:
  proxy:
    zapLogLevel: error

# Apply the gateway configuration
helmfile apply -f common-configurations/istio.helmfile.yaml

# Verify NLB is provisioned (takes ~2 minutes)
kubectl get gateways -A
# NAMESPACE   NAME              CLASS   ADDRESS                              READY
# llm-d       llm-d-gateway     istio   llm-d-xxx.elb.us-east-1.amazonaws.com   True

export INFERENCE_GATEWAY=$(kubectl get gateway llm-d-gateway -n llm-d \
  -o jsonpath='{.status.addresses[0].value}')
echo $INFERENCE_GATEWAY
# llm-d-xxx.elb.us-east-1.amazonaws.com

Step 3: Deploy Path 1 — Intelligent Inference Scheduling

# Navigate to the inference-scheduling guide
cd guides/inference-scheduling

# Structure:
# inference-scheduling/
# ├── gaie-inference-scheduling/    ← EPP + InferencePool + InferenceModel config
# │   └── values.yaml
# ├── ms-inference-scheduling/      ← vLLM deployment config
# │   └── values.yaml              ← Edit this for your model + GPU type
# └── helmfile.yaml.gotmpl

Configure the vLLM deployment for your hardware:

# ms-inference-scheduling/values.yaml
# Edit these for your specific model and GPU

model:
  name: "meta-llama/Meta-Llama-3.3-70B-Instruct-FP8"
  huggingFaceTokenSecret: "hf-token"   # k8s secret name

# Container image: AWS-optimized llm-d container
image:
  repository: ghcr.io/llm-d/llm-d-aws
  tag: "v0.5.1"

# For p5.48xlarge (H100 SXM): tp=4, 2 pods = 8 GPUs total
# For g6e.12xlarge (L40S): tp=1, 4 pods = 4 GPUs
parallelism:
  tensorParallel: 4
  replicaCount: 2

# vLLM inference engine settings
vllmArgs:
  - "--enable-prefix-caching"      # Must be enabled for EPP to get cache events
  - "--block-size=128"             # 128-token blocks: better for long prompts
  - "--gpu-memory-utilization=0.90"
  - "--max-model-len=32768"
  - "--disable-uvicorn-access-log"

# Resource allocation per pod (tp=4 = 4 GPUs per pod)
resources:
  limits:
    nvidia.com/gpu: "4"
    memory: "512Gi"
    cpu: "64"
  requests:
    nvidia.com/gpu: "4"
    memory: "512Gi"
    cpu: "64"

nodeSelector:
  node.kubernetes.io/instance-type: p5.48xlarge

tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule

# Labels that the EPP uses to discover this pool's pods
labels:
  llm-d.ai/inferenceServing: "true"
  model: llama-3.3-70b

Configure the EPP and InferencePool:

# gaie-inference-scheduling/values.yaml

inferencePool:
  # Matches the labels on vLLM pods above
  selector:
    matchLabels:
      llm-d.ai/inferenceServing: "true"
      model: llama-3.3-70b
  targetPortNumber: 8000

inferenceModel:
  # The model name that clients put in their API calls
  modelName: "meta-llama/Meta-Llama-3.3-70B-Instruct-FP8"
  criticality: Critical   # Critical | Standard | Sheddable

epp:
  # EPP configuration: KV cache-aware routing
  cacheAwareRouting:
    enabled: true
    algorithm: "prefix-aware"
    sessionAffinity: true   # Sticky sessions: same client → same pod when possible
  resources:
    requests:
      cpu: "4"
      memory: "8Gi"
    limits:
      cpu: "8"
      memory: "16Gi"

# Deploy the full inference-scheduling stack
helmfile apply

# Watch everything come up
kubectl get pods -n llm-d -w
# llm-d-epp-xxx            1/1  Running   ← EPP routing oracle
# llm-d-vllm-0             1/1  Running   ← vLLM pod 1
# llm-d-vllm-1             1/1  Running   ← vLLM pod 2

# Verify InferencePool and InferenceModel are created
kubectl get inferencepool,inferencemodel -n llm-d
# NAME                                  READY
# inferencepool.inference.../llama-pool  True
#
# NAME                                       CRITICALITY   POOL
# inferencemodel.inference.../llama-prod     Critical      llama-pool

# Check EPP logs to verify it's receiving ZMQ events from vLLM pods
kubectl logs -n llm-d deploy/llm-d-epp | grep -i "cache\|zmq\|block"
# INFO  KV cache indexer started, listening on ZMQ
# INFO  Pod pod/llm-d-vllm-0 discovered, ZMQ subscriber connected
# INFO  Pod pod/llm-d-vllm-1 discovered, ZMQ subscriber connected
# INFO  BlockStored event: pod=llm-d-vllm-0, blocks=12, model=meta-llama/...

Step 4: Test Intelligent Routing

# Send your first request
curl http://$INFERENCE_GATEWAY/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Meta-Llama-3.3-70B-Instruct-FP8",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant specializing in AWS cloud architecture..."
      },
      {
        "role": "user",
        "content": "What is the difference between ECS and EKS?"
      }
    ],
    "max_tokens": 512
  }' | jq '.usage, .choices[0].message.content'

# Send 5 more requests with the same system prompt (should hit cache)
# Watch EPP logs for routing decisions:
kubectl logs -n llm-d deploy/llm-d-epp -f | grep "routing\|cache\|score"

# Expected output on second request with same system prompt:
# INFO  Scoring pods for request [prompt_hash=0xA3F2...]
# INFO  Pod llm-d-vllm-0: score=24 (24 consecutive blocks cached)
# INFO  Pod llm-d-vllm-1: score=0 (0 blocks cached)
# INFO  Routing to llm-d-vllm-0 (score=24, cache-aware)

Step 5: Deploy Path 2 — Prefill-Decode Disaggregation

For disaggregation, you need at least 2 different GPU node types or enough GPUs to split into prefill and decode pools:

cd guides/pd-disaggregation

# This guide has separate configurations for prefill and decode pods
# Structure:
# pd-disaggregation/
# ├── ms-pd/
# │   ├── prefill-values.yaml   ← Prefill pod configuration
# │   └── decode-values.yaml    ← Decode pod configuration
# └── helmfile.yaml.gotmpl

Configure prefill pods (compute-optimized: lower TP, higher batch):

# ms-pd/prefill-values.yaml

role: prefill   # Tells llm-d this is a prefill pod

image:
  repository: ghcr.io/llm-d/llm-d-aws
  tag: "v0.5.1"    # Must use AWS container for EFA/NIXL

model:
  name: "meta-llama/Meta-Llama-3.3-70B-Instruct-FP8"

# Prefill: lower TP, compute-bound
# More prefill pods (to handle input load)
parallelism:
  tensorParallel: 2    # TP=2 for prefill (compute-bound; less TP = less comm overhead)
  replicaCount: 4      # 4 prefill pods × 2 GPUs = 8 GPUs for prefill

vllmArgs:
  - "--block-size=128"
  - "--max-model-len=32768"
  - "--gpu-memory-utilization=0.80"   # Leave room for KV transfer buffers
  - "--disable-uvicorn-access-log"
  # NIXL configuration: prefill side sends KV to decode
  - "--kv-transfer-config"
  - '{"kv_connector":"NixlConnector","kv_role":"kv_both","kv_connector_extra_config":{"backends":["LIBFABRIC"],"enable_cross_layers_blocks":"True"}}'

env:
  - name: VLLM_NIXL_SIDE_CHANNEL_HOST
    valueFrom:
      fieldRef:
        fieldPath: status.podIP   # NIXL side-channel: how decode pod discovers prefill pod IP
  - name: HF_HOME
    value: "/model-cache"
  # EFA configuration for NIXL RDMA transfers
  - name: FI_PROVIDER
    value: "efa"
  - name: FI_EFA_USE_DEVICE_RDMA
    value: "1"
  - name: FI_EFA_FORK_SAFE
    value: "1"

resources:
  limits:
    nvidia.com/gpu: "2"
    memory: "256Gi"
    cpu: "32"
    vpc.amazonaws.com/efa: "2"   # 1 EFA per GPU for prefill
  requests:
    nvidia.com/gpu: "2"
    memory: "256Gi"
    cpu: "32"
    vpc.amazonaws.com/efa: "2"

ports:
  - containerPort: 8000
    name: vllm
  - containerPort: 5600   # NIXL side-channel port
    name: nixl

nodeSelector:
  node.kubernetes.io/instance-type: p5.48xlarge

tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule

Configure decode pods (memory-bandwidth-optimized: higher TP, receives KV):

# ms-pd/decode-values.yaml

role: decode   # Decode pod: receives KV cache, generates tokens

image:
  repository: ghcr.io/llm-d/llm-d-aws
  tag: "v0.5.1"

model:
  name: "meta-llama/Meta-Llama-3.3-70B-Instruct-FP8"

# Decode: higher TP, memory-bandwidth-bound
# Fewer decode pods (decode is faster than prefill for typical workloads)
parallelism:
  tensorParallel: 4    # TP=4 for decode: spreads KV cache read across 4 GPUs' HBM buses
  replicaCount: 2      # 2 decode pods × 4 GPUs = 8 GPUs for decode

vllmArgs:
  - "--block-size=128"
  - "--max-model-len=32768"
  - "--gpu-memory-utilization=0.85"
  - "--max-num-seqs=256"          # High concurrency: decode is low-compute
  - "--disable-uvicorn-access-log"
  # NIXL: decode side receives KV from prefill
  - "--kv-transfer-config"
  - '{"kv_connector":"NixlConnector","kv_role":"kv_both","kv_connector_extra_config":{"backends":["LIBFABRIC"],"enable_cross_layers_blocks":"True"}}'

env:
  - name: VLLM_NIXL_SIDE_CHANNEL_HOST
    valueFrom:
      fieldRef:
        fieldPath: status.podIP
  - name: HF_HOME
    value: "/model-cache"
  - name: FI_PROVIDER
    value: "efa"
  - name: FI_EFA_USE_DEVICE_RDMA
    value: "1"
  - name: FI_EFA_FORK_SAFE
    value: "1"

resources:
  limits:
    nvidia.com/gpu: "4"
    memory: "512Gi"
    cpu: "64"
    vpc.amazonaws.com/efa: "4"   # 4 EFA interfaces for decode: higher BW requirement
  requests:
    nvidia.com/gpu: "4"
    memory: "512Gi"
    cpu: "64"
    vpc.amazonaws.com/efa: "4"

ports:
  - containerPort: 8000
    name: vllm
  - containerPort: 5600
    name: nixl

labels:
  llm-d.ai/inferenceServing: "true"   # Decode pods are in the InferencePool
  llm-d.ai/role: decode

nodeSelector:
  node.kubernetes.io/instance-type: p5.48xlarge

tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule

Note that only decode pods have the llm-d.ai/inferenceServing: "true" label. The InferencePool routes client traffic only to decode pods — prefill pods are not directly client-facing. The EPP routes each request to a decode pod; that decode pod then coordinates with a prefill pod for KV transfer via NIXL.

# Deploy disaggregated stack
helmfile apply

# Watch pods
kubectl get pods -n llm-d -l llm-d.ai/role

# NAME                           READY   STATUS    ROLE
# llm-d-prefill-0                1/1     Running   prefill
# llm-d-prefill-1                1/1     Running   prefill
# llm-d-prefill-2                1/1     Running   prefill
# llm-d-prefill-3                1/1     Running   prefill
# llm-d-decode-0                 1/1     Running   decode
# llm-d-decode-1                 1/1     Running   decode

# Verify NIXL connections between prefill and decode
kubectl logs -n llm-d llm-d-decode-0 | grep -i nixl
# INFO  NIXL side channel server listening on 10.0.1.45:5600
# INFO  NIXL connection from prefill pod 10.0.1.32:5600 established
# INFO  NIXL connection from prefill pod 10.0.1.33:5600 established
# INFO  NIXL using libfabric backend, EFA interface efa0

# Run a benchmark to confirm disaggregation is working
# (compare TTFT to baseline — should be 30-70% lower for long inputs)
python3 -c "
import requests, time

prompt = 'The ' + 'quick brown fox ' * 200  # ~1000 tokens
start = time.time()
resp = requests.post(f'http://{INFERENCE_GATEWAY}/v1/completions',
  json={'model': 'meta-llama/Meta-Llama-3.3-70B-Instruct-FP8',
        'prompt': prompt, 'max_tokens': 100, 'stream': False})
ttft = time.time() - start
print(f'TTFT: {ttft:.3f}s, tokens: {resp.json()[\"usage\"]}')
"

Step 6: HuggingFace Token as Kubernetes Secret

# Create the HF token secret (referenced by values.yaml)
kubectl create secret generic hf-token \
  --from-literal=token="hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" \
  --namespace llm-d

Step 7: EFS for Model Weight Storage

Instead of downloading model weights on every pod startup (slow, redundant), use EFS with ReadWriteMany:

# efs-model-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: model-cache-pvc
  namespace: llm-d
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: efs-sc
  resources:
    requests:
      storage: 500Gi   # Enough for several 70B FP8 models (~140GB each)
---
# Add to vllm container in values.yaml:
volumeMounts:
  - name: model-cache
    mountPath: /model-cache

volumes:
  - name: model-cache
    persistentVolumeClaim:
      claimName: model-cache-pvc

Pre-load the model once using an init job:

kubectl run model-downloader \
  --image=python:3.11 \
  --restart=Never \
  --overrides='
  {
    "spec": {
      "containers": [{
        "name": "downloader",
        "image": "python:3.11",
        "command": ["sh", "-c",
          "pip install huggingface_hub && huggingface-cli download meta-llama/Meta-Llama-3.3-70B-Instruct-FP8 --local-dir /model-cache/meta-llama/Meta-Llama-3.3-70B-Instruct-FP8 --token $(HF_TOKEN)"],
        "env": [{"name": "HF_TOKEN", "valueFrom": {"secretKeyRef": {"name": "hf-token", "key": "token"}}}],
        "volumeMounts": [{"name": "model-cache", "mountPath": "/model-cache"}]
      }],
      "volumes": [{"name": "model-cache", "persistentVolumeClaim": {"claimName": "model-cache-pvc"}}]
    }
  }' \
  -n llm-d

Part 7: Observability — Knowing What Your Routing Layer Is Doing

The point of llm-d's architecture is that routing decisions are now intelligent and traceable. You should be able to see them in real time.

EPP Metrics

The EPP exports Prometheus metrics. Add a ServiceMonitor:

# epp-servicemonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: llm-d-epp
  namespace: llm-d
spec:
  selector:
    matchLabels:
      app: llm-d-epp
  endpoints:
    - port: metrics
      interval: 15s

Key metrics to watch:

# Cache hit rate across all routing decisions
# TARGET: > 80% for RAG workloads
epp_kvcache_hit_ratio{pool="llama-pool", model="meta-llama/..."}

# Number of distinct block hashes in the EPP index
# Low = cache pressure; high = healthy cache diversity
epp_kvcache_index_size{pod="llm-d-vllm-0"}

# Routing decisions by reason
epp_routing_decisions_total{reason="cache_hit", pod="llm-d-vllm-0"}
epp_routing_decisions_total{reason="load_balance", pod="llm-d-vllm-1"}

# Per-pod TTFT (confirms cache hits translate to latency improvement)
vllm:time_to_first_token_seconds_bucket{pod="llm-d-vllm-0"}

# NIXL transfer latency (disaggregated mode only)
nixl_kv_transfer_duration_seconds{src_pod="prefill-0", dst_pod="decode-0"}

Grafana Dashboard Queries

# Cache hit rate over time
rate(epp_routing_decisions_total{reason="cache_hit"}[5m])
/
rate(epp_routing_decisions_total[5m])

# P95 TTFT by routing decision type
histogram_quantile(0.95,
  rate(vllm:time_to_first_token_seconds_bucket[5m])
)

# GPU memory utilization per pod (want: high and stable, not sawtooth)
DCGM_FI_DEV_FB_USED{namespace="llm-d"} /
DCGM_FI_DEV_FB_TOTAL{namespace="llm-d"}

# Tokens per second per pod
rate(vllm:generation_tokens_total[1m])

Part 8: llm-d vs NVIDIA Dynamo vs SGLang PD — Choosing the Right Tool

If you've been following this blog series, you've now seen three inference frameworks that all solve overlapping problems: llm-d, NVIDIA Dynamo (covered in the first post), and SGLang PD disaggregation. This comes up in every platform conversation, so here's the honest comparison.

Use llm-d when:

• You're already on vLLM and want the cleanest Kubernetes-native deployment model

• Multi-tenancy with model criticality tiers matters (InferenceModel is cleaner than custom annotations)

• You want the AWS-optimized container to just work on EKS without manual EFA plumbing

• You're running MoE models (DeepSeek-R1, Qwen3.5) — the Wide-EP path is production-validated

Use NVIDIA Dynamo when:

• You need engine flexibility (some workloads on vLLM, others on TRT-LLM or SGLang)

• You want the most mature tiered KV cache with S3 offload

• You're running the Slinky+KAI unified GPU platform from the previous post — Dynamo fits that architecture naturally

Use SGLang PD when:

• Your workload specifically benefits from SGLang's chunked prefill + RadixCache combination

• You're running inference-heavy agentic workflows (SGLang's structured generation is hard to match)

The honest answer for most EKS deployments starting in 2026: start with llm-d Path 1 (intelligent scheduling only). You get 87%+ cache hit rates and 88% TTFT improvement on warm requests with the simplest possible deployment. Add disaggregation (Path 2) only when you've validated that prefill is your bottleneck with production traffic data.

Part 9: Cost Modeling — What These Numbers Mean in Dollars

Taking the Red Hat benchmark numbers as a baseline:

Setup: 10× g6e.12xlarge ($3.67/hour each), serving Llama 3.3 8B, 200 req/sec

The tiered savings model:

• Path 1 alone: 41% cost reduction from eliminating redundant KV computation

• Path 1 + Path 2: 61% cost reduction from disaggregation freeing decode GPUs from prefill blocking

• Both require the same GPU fleet — you serve more traffic, not fewer GPUs

For a pharma customer running RAG over clinical documents (typically 4,000-8,000 token contexts per query), the cache hit impact is even larger. Document chunks from the same knowledge base appear in most queries; the EPP routes nearly all of them to the pod that already has those document chunks cached. Cache hit rates in the 90%+ range are achievable for homogeneous RAG workloads.

Wrapping Up

The announcement of ghcr.io/llm-d/llm-d-aws is significant not just because it brings disaggregated inference to EKS — NVIDIA Dynamo already did that. What makes llm-d's arrival on AWS meaningful is that it packages a new Kubernetes resource model for inference that changes the abstraction layer you operate at.

InferenceModel separates what-to-serve from how-to-serve it. InferencePool replaces the blunt instrument of Service with a pool that has a routing brain. The EPP knows what's in your GPU memory before deciding where to send the next request. These aren't implementation details — they're the abstractions that Kubernetes was missing for AI inference workloads, and they're now stable CRDs you can build platform tooling around.

The routing problem is the problem that unlocks everything else. You can have the fastest inference engine in the world, but if your load balancer routes a multi-turn conversation to a different pod on every turn — recomputing the entire conversation history each time — you've left 80% of your GPU's prefill capacity on the floor. That's the problem llm-d solves at the infrastructure level, without requiring application changes, and it's the reason 87.4% cache hit rates and 88% TTFT improvements on warm requests aren't marketing numbers. They're what happens when routing stops being stateless.

And yet your 32-GPU training job has been "running" for four hours without processing a single batch.

Look closer. The job has 28 pods in Running state and 4 pods in Pending. The 28 running pods are all sitting in an initialization loop, waiting for all workers to join the distributed rendezvous before any computation begins. The 4 pending pods can't schedule because two other jobs grabbed those GPUs earlier. Those two other jobs are also partially scheduled — also waiting for their remaining pods. The cluster has 64 GPUs allocated. Zero are doing useful work.

This is the GPU deadlock. It's silent, it's expensive (at p5.48xlarge pricing, four hours of 64 wasted H100s costs over $5,000), and it's caused by one architectural gap in the default Kubernetes scheduler: it schedules pods individually, one at a time, with no concept that a group of pods must succeed together or not at all.

Gang scheduling is the solution. But "gang scheduling on Kubernetes" is not one thing — it's a spectrum of implementations with meaningfully different trade-offs, and the right choice depends on your cluster complexity, workload mix, and how much operational overhead you're willing to accept. As of early 2026, you have three serious options: native Kubernetes 1.35 gang scheduling (brand new, alpha), Kueue with the coscheduling plugin (stable, widely used), and NVIDIA KAI Scheduler (production-grade, hierarchical, elastic).

This post walks through all three — from first principles to working EKS manifests — so you can pick the right one and actually deploy it.

Part 1: The Anatomy of a GPU Deadlock

Before we look at solutions, we need to understand the problem precisely. The default kube-scheduler does something entirely reasonable that becomes catastrophically wrong for distributed GPU jobs.

How the Default Scheduler Works

The Kubernetes scheduler processes pods from a priority queue, one at a time. For each pod, it runs through a pipeline:

This works perfectly for independent, stateless pods. It completely breaks for distributed training jobs.

Here's why. Imagine a 32-GPU PyTorch DDP job on EKS. The Training Operator creates 32 worker pods simultaneously. They all enter the scheduler queue. The scheduler processes them one at a time:

Result:

• Training job: 16 pods running, 16 pods pending. The 16 running pods call dist.init_process_group(), get partial group, and hang waiting for the other 16

• OtherJob-A: 4 pods running, consuming 4 GPUs, also likely waiting for more workers

• 20 GPUs allocated, 0 GPUs doing work

And here's the kicker — this state is stable. Nothing will fix it without external intervention. The scheduler won't preempt OtherJob-A's pods (they're Running, not Pending). The training job won't time out and release its 16 GPUs (it's also Running). You have discovered a distributed system deadlock implemented entirely in cloud billing.

Diagnosing the Deadlock

The symptoms are subtle because kubectl get pods shows most pods as Running:

The definitive diagnostic: compare GPU allocation vs GPU utilization on running pods:

# Check what the GPU is actually DOING (not just allocated)
kubectl exec pytorch-worker-0 -n training -- nvidia-smi \
  --query-gpu=utilization.gpu,memory.used,memory.total \
  --format=csv,noheader

# Output on a deadlocked pod:
# 0 %, 512 MiB, 80480 MiB
# ↑ 0% utilization with 80GB allocated = silent waste

# Compare to healthy running pod:
# 97 %, 72140 MiB, 80480 MiB

Check the pending pods for the exact scheduling failure reason:

kubectl describe pod pytorch-worker-16 -n training

# Events:
#   Warning  FailedScheduling  0/12 nodes available:
#            4 Insufficient nvidia.com/gpu,
#            8 node(s) had untolerated taints
#
# ← Cluster has capacity (not all 12 nodes are full)
# ← But specifically the 4 GPU nodes that OtherJob-A is holding
#    have no remaining GPU slots

And the DCGM aggregate view that makes the waste undeniable:

# GPU allocation vs utilization across all GPU nodes
kubectl exec -n gpu-operator ds/nvidia-dcgm-exporter -- \
  curl -s localhost:9400/metrics | \
  grep -E 'DCGM_FI_DEV_(GPU_UTIL|FB_USED|FB_FREE)'

# node-1: gpu_util=0, fb_used=40GB, fb_free=40GB  ← allocated, idle
# node-2: gpu_util=0, fb_used=40GB, fb_free=40GB  ← allocated, idle
# node-3: gpu_util=12%, fb_used=18GB, fb_free=62GB ← OtherJob-A, low utilization

You're looking at $1,200+/hour of H100 compute sitting at 0% utilization, waiting for a scheduling problem to resolve itself. It won't.

The Karpenter Trap: How Autoscaling Makes This Worse

On EKS with Karpenter, there's a second failure mode that's less understood. When your training job's 16 pending pods trigger Karpenter to provision new nodes, you enter a race condition:

The cluster grows. The training job never gets a complete allocation. You're now paying for exponentially more nodes than you need, none of which are useful.

The specific Karpenter behavior that causes this: by default, Karpenter provisions nodes greedily based on pending pod resource requirements. It doesn't understand that pods 0-15 of the training job and pods 16-31 must be co-scheduled. Each set of pending pods looks like an independent workload.

The fix is two-fold: gang scheduling prevents pods 0-15 from ever landing until 0-31 can all land, and Karpenter startup taints ensure freshly-provisioned nodes are reserved for the specific workload that triggered provisioning until the gang is fully scheduled.

We'll implement both fixes in the three solutions below.

Part 2: Understanding the Scheduling Pipeline — Where Gang Scheduling Hooks In

All three solutions we're covering work by intercepting the default scheduling pipeline at the Permit phase. Understanding why this phase is the right intervention point explains the design of everything that follows.

The Permit gate is precisely the right place because at this point, each pod has already been tentatively matched to a node (the Reserve phase). Resources are soft-reserved but not committed to etcd. The gang coordinator can see if ALL members found valid placements. If yes — open the gate, all pods bind simultaneously. If no — release all reservations atomically, preventing any partial allocation from reaching Running state.

This is what makes gang scheduling fundamentally different from just "waiting for all pods to exist" — it operates at the scheduling cycle level, not the pod lifecycle level. By the time pods are Running, the gang is guaranteed to be complete.

Part 3: Solution 1 — Kubernetes 1.35 Native Gang Scheduling

Released December 2025 as an alpha feature in Kubernetes 1.35, native gang scheduling is built directly into kube-scheduler. No separate scheduler binary. No additional operator. This is the lowest-friction entry point.

When to Use This

• You're on EKS 1.35+ (check: kubectl version)

• Your workloads are single-framework (PyTorch/JAX jobs, no Slurm/Ray coexistence)

• You don't need quota management across teams

• You want to keep the operational footprint minimal

What You Give Up

Alpha stability — the API will change. No elastic gang (min < max workers). No preemption policies. No queue management. No cross-namespace fairness. Think of this as the gateway drug — it solves the deadlock, nothing more.

Enabling the Feature Gates

On EKS, you enable feature gates via the EKS cluster configuration:

# Check current EKS version 
aws eks describe-cluster --name my-cluster --query 'cluster.version' 
# "1.35" 
# Create a custom kube-scheduler ConfigMap to enable feature gates 
# Note: On EKS, you patch the scheduler config via aws-auth and 
# kube-scheduler ConfigMap — not directly via component flags

# kube-scheduler-config.yaml
# Applied as a ConfigMap that the EKS scheduler reads
apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
  - schedulerName: default-scheduler
    plugins:
      preEnqueue:
        enabled:
          - name: GangScheduling     # Enable gang scheduling plugin
      permit:
        enabled:
          - name: GangScheduling
    pluginConfig:
      - name: GangScheduling
        args:
          # How long to wait for the full gang before releasing reservations
          gangSchedulingTimeout: 5m
          # Allow minCount < total to support partial gangs (K8s 1.35 alpha)
          permitWaitingTimeoutSeconds: 300
featureGates:
  GenericWorkload: true      # Required: enables Workload API
  GangScheduling: true       # Required: enables gang scheduling plugin
  OpportunisticBatching: true  # Beta in 1.35: auto-enabled, speeds up identical pod scheduling

For EKS-managed clusters, apply this via the EKS addon configuration:

# EKS uses a managed kube-scheduler — configure via eksctl or aws CLI
eksctl utils update-kube-scheduler-config \
  --cluster my-cluster \
  --config-file kube-scheduler-config.yaml

# Verify feature gates are active
kubectl get --raw /metrics | grep scheduler_framework_extension_point_duration | head -5
# If GangScheduling plugin appears here, it's running

The Workload API

Create a Workload object that defines the gang requirements:

# training-workload.yaml
apiVersion: scheduling.k8s.io/v1alpha1
kind: Workload
metadata:
  name: llama-distributed-training
  namespace: ml-training
spec:
  podGroups:
    # The worker gang: all 32 must schedule together
    - name: workers
      policy:
        gang:
          minCount: 32    # All-or-nothing: 32 workers schedule simultaneously
                          # Set minCount < 32 for elastic gang (e.g., minCount: 24
                          # means "start if at least 24/32 can schedule")

    # Separate group for the master/launcher pod
    # Master doesn't need GPU and schedules independently
    - name: launcher
      policy:
        gang:
          minCount: 1

Reference the Workload in your pod specs. The Training Operator (PyTorchJob, MPIJob) doesn't natively support workloadRef yet — you'll need to inject it via a mutating webhook or define pods directly:

# pytorch-distributed-job.yaml
apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
  name: llama-training
  namespace: ml-training
  annotations:
    # Webhook injects workloadRef into generated pods
    gang-scheduling/workload-name: "llama-distributed-training"
    gang-scheduling/pod-group: "workers"
spec:
  pytorchReplicaSpecs:
    Master:
      replicas: 1
      restartPolicy: OnFailure
      template:
        metadata:
          labels:
            workload-name: llama-distributed-training
            workload-pod-group: launcher
          # workloadRef injected here by the mutating webhook
        spec:
          containers:
            - name: pytorch
              image: "763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-training:2.3.0-gpu-py311-cu121"
              resources:
                requests:
                  cpu: "4"
                  memory: "16Gi"
                  # No GPU for master

    Worker:
      replicas: 32
      restartPolicy: OnFailure
      template:
        metadata:
          labels:
            workload-name: llama-distributed-training
            workload-pod-group: workers
        spec:
          # workloadRef field: links pod to the Workload gang definition
          workloadRef:
            name: llama-distributed-training
            podGroup: workers

          schedulerName: default-scheduler  # Uses patched kube-scheduler with GangScheduling plugin

          containers:
            - name: pytorch
              image: "763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-training:2.3.0-gpu-py311-cu121"
              command:
                - torchrun
                - --nproc_per_node=1
                - --nnodes=32
                - --node_rank=$(RANK)
                - --master_addr=$(MASTER_ADDR)
                - --master_port=23456
                - train.py
              env:
                - name: RANK
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.annotations['batch.kubernetes.io/job-completion-index']
                - name: MASTER_ADDR
                  value: "llama-training-master-0"
                - name: NCCL_SOCKET_IFNAME
                  value: "efa0"
                - name: FI_EFA_USE_DEVICE_RDMA
                  value: "1"
              resources:
                requests:
                  nvidia.com/gpu: "1"
                  cpu: "24"
                  memory: "192Gi"
                  vpc.amazonaws.com/efa: "1"
                limits:
                  nvidia.com/gpu: "1"
                  vpc.amazonaws.com/efa: "1"

          nodeSelector:
            node.kubernetes.io/instance-type: p5.48xlarge

          tolerations:
            - key: nvidia.com/gpu
              operator: Exists
              effect: NoSchedule

Fixing the Karpenter Race Condition for K8s 1.35 Gang Scheduling

With native gang scheduling, pods don't enter the active queue until minCount pods exist AND the Workload object is created. But Karpenter can still race with other jobs during node provisioning. Use startupTaints on the Karpenter NodePool to hold fresh nodes for the gang until it's fully bound:

# karpenter-nodepool-gang-aware.yaml
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu-training-pool
spec:
  template:
    spec:
      # Startup taint: new nodes are tainted until the gang binds
      # The gang pods tolerate this taint; random other pods don't
      startupTaints:
        - key: gpu-node/initializing
          value: "true"
          effect: NoSchedule

      # This taint is removed by the GPU Operator's Node Feature Discovery
      # AFTER the NVIDIA driver is fully initialized on the node
      # Until then, no pods can schedule — including gang pods
      taints:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule

      requirements:
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ["p5.48xlarge"]
        - key: topology.kubernetes.io/zone
          operator: In
          values: ["us-east-1a"]  # Single AZ for EFA

  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 300s

# Add to your training pod spec tolerations:
tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule
  - key: gpu-node/initializing    # Tolerate startup taint
    operator: Exists
    effect: NoSchedule
    # This toleration means: only gang training pods can land on
    # freshly-booted nodes, preventing random pods from stealing GPU slots

Verifying Gang Scheduling Behavior

# Apply the Workload and PyTorchJob
kubectl apply -f training-workload.yaml
kubectl apply -f pytorch-distributed-job.yaml

# Watch pod state — ALL should be Pending until cluster has room for all 32
kubectl get pods -n ml-training -w

# Expected CORRECT behavior:
# pytorch-worker-0   0/1  Pending  0  5s   ← All pending together
# pytorch-worker-1   0/1  Pending  0  5s
# ...
# pytorch-worker-31  0/1  Pending  0  5s
#
# Then, once all 32 nodes confirmed available:
# pytorch-worker-0   0/1  ContainerCreating  0  8s   ← All transition simultaneously
# pytorch-worker-1   0/1  ContainerCreating  0  8s
# ...
# pytorch-worker-31  0/1  ContainerCreating  0  8s

# WRONG behavior (gang scheduling NOT working):
# pytorch-worker-0   1/1  Running  0  30s  ← Partial scheduling
# pytorch-worker-1   1/1  Running  0  30s  ← Gang is broken
# pytorch-worker-16  0/1  Pending  0  30s  ← Back to deadlock

# Check gang scheduling events
kubectl describe workload llama-distributed-training -n ml-training
# Events:
#   Normal  GangWaiting   "Waiting for 32/32 pods to be ready for scheduling"
#   Normal  GangReady     "All 32 pods ready, opening Permit gate"
#   Normal  GangScheduled "32 pods bound to nodes simultaneously"

Part 4: Solution 2 — Kueue with Coscheduling Plugin

Kueue is Kubernetes SIG's production-grade batch scheduling framework. Unlike the native gang scheduling feature (alpha), Kueue is stable, widely deployed, and natively integrated with PyTorchJob, MPIJob, JobSet, and RayJob via dedicated adapters.

When to Use This

• You need quota management: team A gets 24 GPUs guaranteed, team B gets 16, bursting allowed

• You need fairness: no single team can starve others during high contention

• You need multi-framework support: PyTorch, MPI, JobSet, and Ray all go through one queue

• You want stability — Kueue is beta/GA, not alpha

• You're not yet running Slurm side-by-side (if you are, jump to KAI)

Install Kueue with the Coscheduling Plugin

# Install Kueue (includes PodGroup-based coscheduling)
kubectl apply --server-side -f \
  https://github.com/kubernetes-sigs/kueue/releases/download/v0.10.0/manifests.yaml

# Install scheduler-plugins for coscheduling (PodGroup CRD + coscheduling plugin)
helm repo add scheduler-plugins https://scheduler-plugins.sigs.k8s.io/
helm repo update

helm install scheduler-plugins scheduler-plugins/scheduler-plugins \
  --namespace scheduler-plugins \
  --create-namespace \
  --set scheduler.replicaCount=1 \
  --set controller.replicaCount=1

# Verify
kubectl get pods -n scheduler-plugins
# scheduler-plugins-controller-xxx   1/1  Running
# scheduler-plugins-scheduler-xxx    1/1  Running

# Verify Kueue CRDs
kubectl get crd | grep kueue
# clusterqueues.kueue.x-k8s.io
# localqueues.kueue.x-k8s.io
# resourceflavors.kueue.x-k8s.io
# workloads.kueue.x-k8s.io

Resource Hierarchy: Flavors → ClusterQueues → LocalQueues

Here's the full four-level Kueue hierarchy, each level as a colour-coded band:

Level 1 — ResourceFlavors (amber): Three flavors — h100-sxm5, a100-sxm4, cpu-general — each showing the node labels and taints that map quota to physical hardware. Dashed arrows flow down into the ClusterQueues that consume them.

Level 2 — ClusterQueues (blue): Three cluster-wide quota pools. cluster-queue-ml shows both nominatedQuota and borrowingLimit per flavor, its cohort membership, and preemption policy. Solid arrows flow down to the LocalQueues beneath each.

Level 3 — LocalQueues (green): Four namespace-scoped entry points — team-alpha, team-beta, serving, batch — each showing which ClusterQueue it draws from, plus live pending/admitted workload counts.

Level 4 — Workloads (purple): Five actual jobs with their GPU/CPU requests and priority classes. The batch pending job (data-proc-job-99) is shown in red with a dashed arrow — it's stuck because the batch queue has hit its quota.

Bottom strip: Two callouts explaining cohort borrowing (ml + inference queues share idle quota up to their borrowingLimit) and preemption (ml queue can evict lower-priority work to reclaim its nominated quota).

# kueue-resource-setup.yaml
---
# Define GPU hardware flavors
# Flavors map to node labels — Kueue picks the right nodes for each job
apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
  name: p5-h100
spec:
  nodeLabels:
    node.kubernetes.io/instance-type: p5.48xlarge
  nodeTaints:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule
  tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule

---
apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
  name: g6e-l40s
spec:
  nodeLabels:
    node.kubernetes.io/instance-type: g6e.48xlarge
  nodeTaints:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule
  tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule

---
# Cluster-wide GPU quota pool
# This is the "bank" that ClusterQueue draws from
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: gpu-cluster-queue
spec:
  namespaceSelector: {}   # All namespaces can use this queue

  # Borrowing: teams can use each other's idle quota
  cohort: ml-platform     # All queues in this cohort can borrow from each other

  preemption:
    reclaimWithinCohort: LowerPriority   # Reclaim from lower-priority jobs in cohort
    borrowWithinCohort:
      policy: LowerPriority              # Can borrow from lower-priority cohort members
      maxPriorityThreshold: 100
    withinClusterQueue: LowerPriority    # Within queue, preempt lower priority

  resourceGroups:
    - coveredResources: ["nvidia.com/gpu", "cpu", "memory"]
      flavors:
        # Try p5 first, fall back to g6e
        - name: p5-h100
          resources:
            - name: nvidia.com/gpu
              nominalQuota: 64    # 8 p5 nodes × 8 GPUs = 64 H100s guaranteed
              borrowingLimit: 32  # Can borrow up to 32 more from cohort
              lendingLimit: 16    # Will lend up to 16 to other cohort members
            - name: cpu
              nominalQuota: "1408"  # 8 nodes × 176 vCPU
            - name: memory
              nominalQuota: "12288Gi"  # 8 nodes × 1.5TB

        - name: g6e-l40s
          resources:
            - name: nvidia.com/gpu
              nominalQuota: 16    # 4 g6e nodes × 4 GPUs
              borrowingLimit: 8
            - name: cpu
              nominalQuota: "384"
            - name: memory
              nominalQuota: "1536Gi"

---
# Per-team LocalQueues in their namespaces
# Teams submit to their LocalQueue; Kueue forwards to ClusterQueue
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
  name: training-queue
  namespace: ml-team-nlp      # NLP team's namespace
spec:
  clusterQueueName: gpu-cluster-queue

---
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
  name: training-queue
  namespace: ml-team-cv       # CV team's namespace
spec:
  clusterQueueName: gpu-cluster-queue

Gang Scheduling via PodGroup CRD

The coscheduling plugin introduces the PodGroup CRD, which works similarly to the native K8s 1.35 Workload but has been stable since Kubernetes 1.27:

# podgroup-training.yaml
apiVersion: scheduling.sigs.k8s.io/v1alpha1
kind: PodGroup
metadata:
  name: llama-training-gang
  namespace: ml-team-nlp
spec:
  # Minimum members that must schedule simultaneously
  # Setting minMember < replicas = elastic gang scheduling
  minMember: 32

  # Minimum resources the group needs (Kueue checks this before admitting)
  minResources:
    nvidia.com/gpu: "32"
    cpu: "768"
    memory: "6144Gi"

  # If scheduler can't find room within this window, release all reservations
  scheduleTimeoutSeconds: 300   # 5 minutes

---
# PyTorchJob referencing the PodGroup
apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
  name: llama-training
  namespace: ml-team-nlp
  labels:
    kueue.x-k8s.io/queue-name: training-queue   # Submit to Kueue
spec:
  pytorchReplicaSpecs:
    Master:
      replicas: 1
      restartPolicy: OnFailure
      template:
        metadata:
          labels:
            pod-group.scheduling.sigs.k8s.io/name: llama-training-gang
            pod-group.scheduling.sigs.k8s.io/min-available: "1"
        spec:
          schedulerName: scheduler-plugins-scheduler   # Use coscheduling scheduler
          containers:
            - name: pytorch
              image: "763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-training:2.3.0-gpu-py311-cu121"
              resources:
                requests:
                  cpu: "4"
                  memory: "16Gi"

    Worker:
      replicas: 32
      restartPolicy: OnFailure
      template:
        metadata:
          labels:
            # Links pod to the PodGroup for gang scheduling
            pod-group.scheduling.sigs.k8s.io/name: llama-training-gang
            pod-group.scheduling.sigs.k8s.io/min-available: "32"
        spec:
          schedulerName: scheduler-plugins-scheduler

          containers:
            - name: pytorch
              image: "763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-training:2.3.0-gpu-py311-cu121"
              command:
                - torchrun
                - --nproc_per_node=1
                - --nnodes=32
                - --node_rank=$(RANK)
                - --master_addr=llama-training-master-0
                - --master_port=23456
                - train.py
              env:
                - name: RANK
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.annotations['batch.kubernetes.io/job-completion-index']
                - name: NCCL_SOCKET_IFNAME
                  value: "efa0"
                - name: FI_EFA_USE_DEVICE_RDMA
                  value: "1"
                - name: NCCL_DEBUG
                  value: "WARN"
              resources:
                requests:
                  nvidia.com/gpu: "1"
                  cpu: "24"
                  memory: "192Gi"
                  vpc.amazonaws.com/efa: "1"
                limits:
                  nvidia.com/gpu: "1"
                  vpc.amazonaws.com/efa: "1"

          nodeSelector:
            node.kubernetes.io/instance-type: p5.48xlarge

          tolerations:
            - key: nvidia.com/gpu
              operator: Exists
              effect: NoSchedule

Kueue Admission Flow

When you apply this PyTorchJob, Kueue intercepts it through the admission webhook before any pods are created:

The critical difference from native gang scheduling: Kueue prevents pods from being created until quota is available. Native gang scheduling lets pods be created and then holds them at the Permit phase. With Kueue, the job is suspended at the controller level — zero GPU allocation until the scheduler is confident the gang can run. This eliminates the Karpenter race condition entirely.

Monitoring Kueue Queue Depth

# See all admitted and queued workloads
kubectl get workloads -n ml-team-nlp

# NAME                              QUEUE            ADMITTED   AGE
# llama-training-xxxxxx             training-queue   True       2m
# other-job-yyyyyy                  training-queue   False      15m   ← queued

# Detailed queue status
kubectl describe clusterqueue gpu-cluster-queue

# Status:
#   Admitted Workloads: 1
#   Pending Workloads:  2
#   Flavors Usage:
#     p5-h100:
#       nvidia.com/gpu: 32/64 (used/quota)
#       Borrowing: 0/32

# Per-team quota health
kubectl get localqueues -A
# NAMESPACE       NAME              CLUSTERQUEUE         PENDING  ADMITTED
# ml-team-nlp    training-queue    gpu-cluster-queue    1        1
# ml-team-cv     training-queue    gpu-cluster-queue    0        2

Part 5: Solution 3 — NVIDIA KAI Scheduler

KAI Scheduler is NVIDIA's open-source production scheduling engine, contributed from the Run:AI platform. It subsumes everything Kueue + coscheduling does, adds elastic gang scheduling, preemption with workload-level granularity, and native KubeRay integration, and — as we covered in the Slinky post — it makes Slurm, Ray, and Kubernetes inference all schedulable through one consistent priority framework.

When to Use This

• You're running mixed workloads: Ray + Kubernetes + (optionally) Slurm

• You need elastic gang (start with N-of-M workers, scale up to M)

• You need preemption at the workload level (inference preempts training)

• You need hierarchical queues that map to your org structure

• You are building a platform that will grow — KAI has the ceiling

Install KAI Scheduler

helm repo add kai-scheduler https://nvidia.github.io/KAI-Scheduler/
helm repo update

# KAI Scheduler replaces kube-scheduler for GPU pods
# kube-scheduler continues handling CPU-only pods
helm install kai-scheduler kai-scheduler/kai-scheduler \
  --namespace kai-scheduler \
  --create-namespace \
  --version 0.3.0 \
  --set scheduler.replicaCount=2 \       # HA deployment for production
  --set scheduler.defaultScheduler=false  # KAI supplements, doesn't replace kube-scheduler

Queue Hierarchy Setup

# kai-queues-org.yaml
---
# Department-level queues define organization boundaries
apiVersion: scheduling.run.ai/v2
kind: Queue
metadata:
  name: ml-department
spec:
  resources:
    gpu:
      quota: 40          # 40 GPUs guaranteed to ML dept
      limit: 60          # Can burst to 60 when HPC is idle
      overQuotaWeight: 2 # Gets 2/3 of burst capacity vs HPC's 1/3
    cpu:
      quota: 640
      limit: 1024
      overQuotaWeight: 2
    memory:
      quota: 5120Gi
      limit: 8192Gi
      overQuotaWeight: 2

---
apiVersion: scheduling.run.ai/v2
kind: Queue
metadata:
  name: hpc-department
spec:
  resources:
    gpu:
      quota: 24
      limit: 60
      overQuotaWeight: 1
    cpu:
      quota: 384
      limit: 1024
      overQuotaWeight: 1
    memory:
      quota: 3072Gi
      limit: 8192Gi
      overQuotaWeight: 1

---
# Team-level child queues
apiVersion: scheduling.run.ai/v2
kind: Queue
metadata:
  name: nlp-team
spec:
  parentQueue: ml-department
  resources:
    gpu:
      quota: 20
      limit: 48
      overQuotaWeight: 3    # NLP is high-priority team in ML dept
    cpu:
      quota: 320
      limit: 768
      overQuotaWeight: 3

---
apiVersion: scheduling.run.ai/v2
kind: Queue
metadata:
  name: cv-team
spec:
  parentQueue: ml-department
  resources:
    gpu:
      quota: 20
      limit: 48
      overQuotaWeight: 1
    cpu:
      quota: 320
      limit: 768
      overQuotaWeight: 1

Strict Gang Scheduling with KAI

For jobs that must have all workers or none:

# kai-gang-strict.yaml
apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
  name: llama-training-strict-gang
  namespace: ml-team-nlp
  labels:
    kai.scheduler/queue: nlp-team
spec:
  pytorchReplicaSpecs:
    Master:
      replicas: 1
      restartPolicy: OnFailure
      template:
        metadata:
          labels:
            kai.scheduler/queue: nlp-team
          annotations:
            # KAI gang group: master + all workers must start simultaneously
            kai.scheduler.nvidia.com/gang-scheduling-group: "llama-training-strict-gang"
            kai.scheduler.nvidia.com/min-members: "33"    # 1 master + 32 workers
            kai.scheduler.nvidia.com/priority: "200"       # Interactive training priority
        spec:
          schedulerName: kai-scheduler
          priorityClassName: interactive-training
          containers:
            - name: pytorch
              image: "763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-training:2.3.0-gpu-py311-cu121"
              resources:
                requests:
                  cpu: "4"
                  memory: "16Gi"

    Worker:
      replicas: 32
      restartPolicy: OnFailure
      template:
        metadata:
          labels:
            kai.scheduler/queue: nlp-team
          annotations:
            kai.scheduler.nvidia.com/gang-scheduling-group: "llama-training-strict-gang"
            kai.scheduler.nvidia.com/min-members: "33"
            kai.scheduler.nvidia.com/priority: "200"
        spec:
          schedulerName: kai-scheduler
          priorityClassName: interactive-training

          containers:
            - name: pytorch
              image: "763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-training:2.3.0-gpu-py311-cu121"
              command:
                - torchrun
                - --nproc_per_node=1
                - --nnodes=32
                - train.py
              resources:
                requests:
                  nvidia.com/gpu: "1"
                  cpu: "24"
                  memory: "192Gi"
                  vpc.amazonaws.com/efa: "1"
                limits:
                  nvidia.com/gpu: "1"
                  vpc.amazonaws.com/efa: "1"

          nodeSelector:
            node.kubernetes.io/instance-type: p5.48xlarge
          tolerations:
            - key: nvidia.com/gpu
              operator: Exists
              effect: NoSchedule

Elastic Gang Scheduling — The KAI Differentiator

Strict gang scheduling ("all or nothing") can be overly conservative. A 32-GPU training job that can make progress with 24 GPUs shouldn't wait 4 hours for a full allocation when 28 GPUs are available right now.

KAI's elastic gang scheduling decouples minReplicas (minimum to start) from replicas (desired):

# kai-gang-elastic.yaml
apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: llama-elastic-training
  namespace: ml-team-nlp
  labels:
    kai.scheduler/queue: nlp-team
spec:
  rayClusterSpec:
    headGroupSpec:
      template:
        metadata:
          annotations:
            kai.scheduler.nvidia.com/gang-scheduling-group: "llama-elastic-training"
            kai.scheduler.nvidia.com/priority: "200"
        spec:
          schedulerName: kai-scheduler
          containers:
            - name: ray-head
              image: "rayproject/ray:2.46.0-gpu-py311"
              resources:
                requests:
                  cpu: "8"
                  memory: "32Gi"

    workerGroupSpecs:
      - groupName: training-workers
        replicas: 32        # Desired: 32 workers
        minReplicas: 20     # Minimum: start if at least 20 GPUs available
        maxReplicas: 48     # Maximum: scale up to 48 if GPUs free up
        template:
          metadata:
            annotations:
              kai.scheduler.nvidia.com/gang-scheduling-group: "llama-elastic-training"
              # min-members matches minReplicas:
              kai.scheduler.nvidia.com/min-members: "20"
              kai.scheduler.nvidia.com/priority: "200"
          spec:
            schedulerName: kai-scheduler

            containers:
              - name: ray-worker
                image: "rayproject/ray:2.46.0-gpu-py311"
                resources:
                  requests:
                    nvidia.com/gpu: "1"
                    cpu: "24"
                    memory: "192Gi"
                    vpc.amazonaws.com/efa: "1"
                  limits:
                    nvidia.com/gpu: "1"
                    vpc.amazonaws.com/efa: "1"

            nodeSelector:
              node.kubernetes.io/instance-type: p5.48xlarge
            tolerations:
              - key: nvidia.com/gpu
                operator: Exists
                effect: NoSchedule

  entrypoint: |
    python train_with_ray.py \
      --num-workers $(ray.available_resources()['GPU']) \
      --model /mnt/models/meta-llama/Meta-Llama-3.1-8B \
      --checkpoint-dir /fsx/checkpoints/llama-elastic-001

The behavior: KAI guarantees at least 20 workers start simultaneously (gang scheduling guarantee), then scales up toward 32 or even 48 as GPUs become available. Ray's elastic training framework adapts the distributed training configuration dynamically as workers join and leave.

Preemption: Inference Over Training

This is where KAI earns its "production-grade" label. When a high-priority inference service needs GPUs and only training jobs are using them, KAI preempts the training job atomically — stopping all workers simultaneously (not one by one), triggering checkpoint hooks, and requeuing the job:

# Priority classes already defined in Part 3 of the Slinky post
# Quick recap for standalone reference:
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: inference-critical
value: 1000
preemptionPolicy: Never
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: interactive-training
value: 500
preemptionPolicy: PreemptLowerPriority
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: batch-training
value: 100
preemptionPolicy: PreemptLowerPriority

Add the checkpoint hook to training pods — KAI sends SIGTERM before SIGKILL, giving the job 120 seconds to save state:

# In your training pod spec:
lifecycle:
  preStop:
    exec:
      command:
        - /bin/sh
        - -c
        - |
          echo "Preemption signal received, saving checkpoint..."
          python /app/save_checkpoint.py \
            --checkpoint-dir /fsx/checkpoints/${JOB_NAME} \
            --step-file /tmp/current_step

          # Signal Ray cluster to save distributed state
          python -c "
          import ray
          ray.init(address='auto')
          ray.get([w.checkpoint.remote() for w in ray.get_actor('workers')])
          print('Ray checkpoint complete')
          "
          echo "Checkpoint saved. Ready for preemption."

terminationGracePeriodSeconds: 180

Monitoring KAI Queue State

# View all queues and their GPU consumption
kubectl get queues -A

# NAMESPACE   NAME          PARENT          GPU QUOTA   GPU USED   PENDING JOBS
# (cluster)   ml-dept       (root)          40          38         2
# (cluster)   nlp-team      ml-dept         20          20         1
# (cluster)   cv-team       ml-dept         20          18         0
# (cluster)   hpc-dept      (root)          24          0          0

# View workloads in queue
kubectl get -A rayjobs,pytorchjobs | grep -v "RUNNING" | grep "Pending\|Suspended"

# Check KAI scheduler events for a specific job
kubectl events --for pytorchjob/llama-training -n ml-team-nlp

# REASON                    MESSAGE
# GangSchedulingStarted    Gang group "llama-training-strict-gang" (33 members)
#                          entered scheduling cycle
# GangMembersReady         All 33 gang members found valid node placements
# GangBound                33 pods bound simultaneously to nodes

Part 6: The Karpenter Integration — Making Node Provisioning Gang-Aware

All three solutions prevent the scheduling deadlock. But with Karpenter, there's still the provisioning race to handle. Here's the definitive configuration pattern regardless of which gang scheduling solution you pick.

Problem Recap

Gang scheduling holds pods at the Permit gate until all members find placements. But "finding a placement" in Karpenter's world means a node will exist — Karpenter will create it. The race happens when new nodes are provisioned for a gang, and other non-gang pods schedule onto those nodes before the gang's Permit gate opens.github+1

Solution: do-not-disrupt + startupTaints + Single AZ

# karpenter-gang-nodepool.yaml
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu-gang-training
spec:
  template:
    metadata:
      labels:
        node-pool: gpu-gang-training
        karpenter.sh/capacity-type: on-demand
      annotations:
        # Prevent Karpenter from consolidating nodes while a gang job is binding
        # This annotation is set by a job admission webhook and removed on completion
        karpenter.sh/do-not-disrupt: "false"   # Default; webhook sets to "true" during gang bind
    spec:
      nodeClassRef:
        apiVersion: karpenter.k8s.aws/v1
        kind: EC2NodeClass
        name: gpu-nodeclass

      # Startup taint: prevents ANY pod from landing until removed by a readiness check
      # The readiness check (DaemonSet) removes the taint after:
      # 1. GPU Operator installs driver
      # 2. Device plugin registers GPU resources
      # 3. NCCL/EFA test passes
      startupTaints:
        - key: node.kubernetes.io/not-ready
          operator: Exists
          effect: NoSchedule
        - key: gpu-node/initializing
          value: "true"
          effect: NoSchedule

      requirements:
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ["p5.48xlarge"]

        - key: topology.kubernetes.io/zone
          operator: In
          values: ["us-east-1a"]    # Critical: all gang nodes must be same AZ for EFA

        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]

      taints:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
        - key: dedicated
          value: gang-training
          effect: NoSchedule

  disruption:
    # NEVER consolidate nodes that have GPU pods running
    # WhenEmpty means: only reclaim nodes with zero pods
    consolidationPolicy: WhenEmpty
    consolidateAfter: 180s   # 3 minutes after last pod exits

  limits:
    nvidia.com/gpu: 512     # Hard cap to prevent runaway provisioning

GPU Readiness DaemonSet — Remove Startup Taint After Full Initialization

# gpu-readiness-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: gpu-node-readiness
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: gpu-node-readiness
  template:
    metadata:
      labels:
        app: gpu-node-readiness
    spec:
      hostNetwork: true
      serviceAccountName: gpu-readiness-sa   # Needs node taint removal permission

      # Run on ALL GPU nodes including those with startup taint
      tolerations:
        - key: gpu-node/initializing
          operator: Exists
          effect: NoSchedule
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule

      initContainers:
        - name: wait-for-gpu-driver
          image: nvidia/cuda:12.1-base-ubuntu22.04
          command:
            - /bin/sh
            - -c
            - |
              echo "Waiting for NVIDIA driver to be ready..."

              # Wait for nvidia-smi to succeed
              until nvidia-smi --query-gpu=name --format=csv,noheader; do
                echo "nvidia-smi not ready, waiting 5s..."
                sleep 5
              done
              echo "Driver ready: $(nvidia-smi --query-gpu=name --format=csv,noheader)"

              # Wait for device plugin to register GPU resources
              until [ "$(cat /sys/class/drm/card0/device/gpu_busy_percent 2>/dev/null)" != "" ] || \
                    nvidia-smi -L | grep -q "GPU 0"; do
                echo "Waiting for device plugin registration..."
                sleep 5
              done
              echo "GPU device registered"

              # Quick NCCL self-test (single node)
              if python3 -c "import torch; torch.cuda.is_available() and print('CUDA OK')"; then
                echo "CUDA/PyTorch smoke test passed"
              fi
          securityContext:
            privileged: true
          volumeMounts:
            - name: dev
              mountPath: /dev

      containers:
        - name: taint-remover
          image: bitnami/kubectl:latest
          command:
            - /bin/sh
            - -c
            - |
              NODE_NAME=$(cat /etc/hostname)
              echo "Removing startup taint from node $NODE_NAME"

              kubectl taint node $NODE_NAME gpu-node/initializing=true:NoSchedule- \
                2>/dev/null || echo "Taint already removed"

              echo "Node $NODE_NAME is fully ready for GPU workloads"

              # Keep running to detect reboots
              while true; do
                sleep 3600
              done
          securityContext:
            runAsNonRoot: true

      volumes:
        - name: dev
          hostPath:
            path: /dev

      nodeSelector:
        workload-tier: gpu-training   # Only on GPU nodes

Admission Webhook: Set do-not-disrupt During Gang Bind Window

# gang-admission-webhook.yaml
# This webhook annotates Karpenter nodes with do-not-disrupt=true
# during the 60-second window when a gang is binding to newly-provisioned nodes
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: gang-karpenter-coordinator
webhooks:
  - name: gang-coordinator.ai-platform.svc
    admissionReviewVersions: ["v1"]
    clientConfig:
      service:
        name: gang-coordinator
        namespace: ai-platform
        path: /mutate-pod-binding
    rules:
      - operations: ["CREATE"]
        apiGroups: [""]
        apiVersions: ["v1"]
        resources: ["pods/binding"]   # Fires when pod is about to bind to a node
    sideEffects: None
    failurePolicy: Ignore   # Don't block pod binding if webhook is unavailable

The webhook logic (simplified):

1. When a gang pod binds to a newly-provisioned node, annotate that node with karpenter.sh/do-not-disrupt: "true"

2. Set a 5-minute TTL on the annotation

3. When the gang is fully bound, remove the annotation

4. If the gang fails (timeout), remove the annotation to allow Karpenter to consolidate

Part 7: Decision Framework

Three solutions, three different trade-off profiles. Here's the honest guide to which one fits your situation.

The Decision Matrix

Recommendation by Team Profile

Use K8s 1.35 native if: You're a single team, single framework (PyTorch), and you just got hit with the deadlock bug and need a fix today without adding new operators. Accept that the API will change. Plan to migrate to Kueue or KAI in 6-12 months as you scale.

Use Kueue + Coscheduling if: You're a platform team supporting 2-5 ML teams that all use Kubernetes-native frameworks. You need quota enforcement and fairness. You don't have Slurm users or Ray power users who need elastic scaling. This is the most stable, widest-supported choice for 2026.

Use NVIDIA KAI if: You're building a GPU platform that will grow into multi-tenant use with diverse workload types — Ray training, Kubernetes inference, and potentially Slurm batch. You need production-grade elastic gang scheduling today. You're okay trading setup complexity for a ceiling that doesn't constrain your growth.

Part 8: Benchmarks — Before and After

Running a representative AI platform workload mix (16 active PyTorchJobs, 4 RayJobs, 2 inference services) on 64 H100 GPUs, measured over 72 hours:

A few notes:

• Job wait time p50 increases with gang scheduling — this is expected and correct. The scheduler is being more conservative about when it starts jobs; no partial starts means slightly longer queuing.

• The dramatic cost reduction comes from eliminating deadlock waste and improving utilization, not from running fewer jobs.

• Inference TTFT improves with KAI because of explicit priority protection — the inference service is never starved of GPU resources by competing training workloads.

• The gap between Kueue (\(99k) and KAI (\)91k) comes from elastic gang scheduling. KAI starts jobs with fewer workers immediately and scales up, versus Kueue waiting for full allocation — meaning jobs complete faster at slightly lower total GPU-hour consumption.

Wrapping Up

Gang scheduling is one of those infrastructure patterns that feels optional right up until the moment it isn't. The GPU deadlock is silent — there's no alert, no error, just a very expensive cluster doing nothing while every monitoring dashboard shows green. The default Kubernetes scheduler was built for stateless, independent workloads. Distributed GPU training is neither.

The good news is that the solution landscape in early 2026 is the best it's ever been. Kubernetes 1.35 makes gang scheduling a native primitive. Kueue makes it team-safe with quotas and fairness. KAI Scheduler makes it production-grade with elastic scaling and preemption. Pick the right layer for your current complexity, solve the deadlock today, and know exactly which direction to grow when your GPU fleet and your team count expand.

The Karpenter interaction — provisioning nodes specifically for a gang without letting other workloads steal the allocation — is the EKS-specific nuance that every other guide leaves out. The startupTaints + WhenEmpty consolidation + single-AZ topology constraint combination is not documented anywhere as a complete pattern. Now it is.

But AWS Community Day Pune was non-negotiable. I cleared the schedule, got on the train, and I'm genuinely glad I did.

First — the organizers deserve their flowers

Toshal, Ameya, and Vipul — what you pulled off was genuinely impressive. There's a quote I keep coming back to, something Maria always says: it takes a village to raise a child. The same is true for a community event. You can have the best speakers and the most interesting content in the world, but if the foundation isn't solid, none of it lands.

The foundation at ACD Pune was solid. More than solid.

From the booths to the workshops, the speaker momentos to the venue — everything was thought through. Nothing felt like an afterthought. The kind of event where you can tell the organizers actually attended other events, took notes, and said "we're doing this better." To the volunteers who showed up early, stayed late, and held the whole thing together from the inside — thank you. Genuinely.

What I was there to talk about

My session was on Custom LLMs on EKS — using Trainium and Inferentia-powered infrastructure.

The core of it: most conversations about running LLMs in production jump straight to GPU instances. What gets underexplored is the case for AWS's own silicon — Trainium for training, Inferentia2 for inference — and what it actually looks like to build a serious, cost-efficient LLM deployment on EKS using that stack.

There's nuance in this that the benchmarks don't capture cleanly. The Neuron SDK, the compilation step, the operator support on Kubernetes, the trade-offs between flexibility and cost efficiency at scale — these are the details that matter when you're actually building, not just evaluating. That's what I tried to cover.

The part I look forward to most — learning from others

Here's something I've said before and genuinely mean: I don't go to community days just to speak. I go to learn. Every single time, I leave with more than I arrived with.

Akash and Sudhansu's keynote set the tone for the day in a way that was hard to follow — I was taking notes. Properly. The kind of notes you take when someone is saying something you want to think about for longer than the session allows.

And then there was Mayur's session on why Kubernetes for AI workloads — which I'll be honest, hit me exactly where I'm currently spending a lot of mental energy. It was a gem. The demo was clean, the reasoning was sharp, and I've been sitting with some of the ideas since. I have thoughts — the kind that come from being deep in this problem myself — and I suspect that session is going to become a reference point in a few conversations I'm about to have. More on that soon.

The people part — which is always the real part

Every community event has a moment where you look around the room and feel something that's hard to name. It happened a few times at ACD Pune.

Familiar faces I've shared stages with before. Newer faces who were speaking for what felt like the first few times, finding their footing in front of an audience — that energy is something special to be around. Organizers and volunteers who weren't on stage but made everything on stage possible. People who drove hours to be in that room because they believe the same thing we all believe: that this community is worth showing up for.

It takes a village. Pune had the village.

Grateful

I'll keep this simple: thank you to everyone who made ACD Pune what it was. To the organizers who built it, the volunteers who ran it, the speakers who showed up prepared, and the attendees who asked the questions that made the day worth having.

See you at the next one.

I'm not going to recap every announcement. There are enough of those. What I want to do is talk about what actually matters for people building infrastructure — the things that made me stop and think "wait, this changes something I was planning."

First, a quick reframe

Most people walked away talking about new chips. And yes, Vera Rubin is exciting. But the thing that kept nagging at me was simpler: NVIDIA now has something meaningful to say at every layer of the stack. Chips. Networking. Software. Agent platforms. Even energy infrastructure partnerships.

That's not a company selling you a GPU anymore. That's a company selling you the entire pipeline from power socket to running application. Whether that's a good or bad thing is a separate conversation — but it's important context for everything else.

The thing I keep thinking about: Dynamo

Everyone is talking about Vera Rubin. I keep thinking about Dynamo.

Here's the problem Dynamo solves, in plain terms.

When an LLM processes your prompt, it does two very different jobs. First it reads and understands everything you sent — that's compute-heavy work. Then it writes back to you, one word at a time — that's memory-heavy work. Right now, most people run both on the same GPUs. That's like using the same person to do research and write the report simultaneously. Neither job gets done optimally.

Dynamo splits them. Different hardware for reading, different hardware for writing. And it shares the "memory" of what's been processed — called KV cache — across the whole system so nothing gets recomputed unnecessarily.

The result on DeepSeek-R1: 50x throughput improvement over the previous generation. That's not a typo.

The part that matters for people already running inference on Kubernetes: Dynamo works on top of vLLM. You don't rebuild your stack. You extend it. That's a big deal.

Agents broke your autoscaling. You just don't know it yet.

This is the nuance nobody at GTC talked about loudly enough.

When you run a standard LLM API — one question, one answer — autoscaling is manageable. You watch request rate, you watch latency, you add capacity when needed. Straightforward.

Agents don't work like that. An agent reasons in steps — it thinks, calls a tool, thinks again, calls another tool, writes a response. One user interaction might trigger five or fifteen LLM calls. The compute is unpredictable. The memory pressure builds up across a session. A system can look completely healthy on your standard dashboards while quietly falling apart underneath because the GPU memory is full of unfinished agent sessions.

The metrics that actually matter in an agentic world are different: how long before the agent gives the user anything useful, how the KV cache is holding up under concurrent sessions, how much is sitting in the queue waiting for a decode slot.

Jensen called this shift "agents as a service" — moving from answering questions to doing work. That framing is clean. The infrastructure implications behind it are less clean and most teams will hit them in production before they read about them anywhere.

NVLink inside AWS silicon — read this one twice

NVIDIA's NVLink interconnect — the technology that lets GPUs share memory at very high speed — is being built into AWS's own chips. Trainium4. Graviton. Nitro.

What this means practically: the expensive, awkward handoff between GPU compute and CPU preprocessing — a bottleneck in almost every serious AI workload — gets much cheaper. AWS-native silicon and NVIDIA GPUs start behaving more like one coherent system rather than two things talking to each other over a slow bridge.

For builders in India: AWS Mumbai and Hyderabad are in scope for this expansion. The hardware available to you from your own region is about to get significantly more capable.

NemoClaw — the one enterprises have been waiting for

Every time I've seen an enterprise try to deploy an AI agent, the conversation eventually hits the same wall: "Great, but who controls what it can access? What if it does something wrong? How do we audit it?"

NemoClaw is built specifically for that wall. It's an open-source enterprise agent platform where policy enforcement, sandboxed execution, and audit trails are built in from the start — not added later when something goes wrong.

What makes it interesting from an infrastructure angle: it sits between the agent and everything the agent wants to touch. It decides what the agent is allowed to know, what systems it can reach, what actions it can take. If you're already using HashiCorp Vault for secrets management and Boundary for access control — NemoClaw completes that picture at the AI layer. Vault manages the credentials. Boundary controls the network access. NemoClaw governs the agent's behaviour. Together that's actually a coherent enterprise story.

DGX Station — briefly, because it matters for India

748 GB memory. 20 petaflops. Runs trillion-parameter models. Sits next to your desk. Works air-gapped.

For Indian enterprises in banking, healthcare, or any regulated sector where data can't leave a specific environment — this is the first product in this class that's actually priced for enterprise deployment rather than a research supercomputer budget. Worth watching closely.

The honest summary

Day 1 of GTC 2026 didn't just announce faster hardware. It sketched out what the next two years of AI infrastructure problems look like — and quietly, what the solutions are.

The teams that pay attention to the infrastructure layer underneath these announcements — not just the chip names and benchmark numbers — will make better decisions in the months ahead. That's what I'm trying to do with this post.

More to come as the week progresses.

You're not getting bad performance despite high GPU costs. You're getting bad GPU utilization because of how you've architected the serving layer. The prefill and decode phases of LLM inference are fundamentally different operations running on hardware optimized for neither—when colocated, they compete for the same resources, create head-of-line blocking, and prevent either phase from being tuned independently.

This isn't an obscure edge case. It's the baseline reality of monolithic LLM serving, and it's why Meta, OpenAI, and every hyperscaler that has published numbers on their inference infrastructure has independently arrived at the same architectural pattern: disaggregated prefill and decode.

This post is a deep technical dive into deploying disaggregated inference at scale on Amazon EKS using two complementary open-source tools: SGLang (the inference runtime) and NVIDIA Dynamo (the distributed serving orchestration framework). We'll cover why disaggregation matters from first principles, how NVIDIA Dynamo's components work, complete EKS deployment with Kubernetes manifests, SGLang-specific configurations, real performance benchmarks, and the operational patterns that separate a working demo from a production deployment.

Why Monolithic Inference Wastes GPUs

Before we build anything, we need to understand the problem precisely. Vague performance claims don't justify architectural complexity—the physics of LLM inference does.

The Two-Phase Problem

Every LLM inference request goes through exactly two phases:

Prefill phase:

• Processes the entire input sequence in parallel

• Compute-bound: GPU SM (streaming multiprocessor) utilization is near 100%

• Result: KV cache entries for every input token

• Duration scales with input length

• Happens once per request

Decode phase:

• Generates output tokens one at a time, autoregressively

• Memory-bound: each decode step loads the entire KV cache from HBM (High Bandwidth Memory)

• Limited by memory bandwidth, not compute

• GPU SMs are underutilized—you're reading memory, not multiplying matrices

• Happens N times per request (once per output token)

The arithmetic here is damning. A request with a 2000-token input and 500-token output does:

• 1 prefill operation (large, compute-heavy)

• 500 decode operations (small, memory-heavy)

On an A100 with 2 TB/s memory bandwidth and 312 TFLOPS FP16 compute, a memory-bound decode step uses roughly 2-5% of compute capacity. You have a 312 TFLOPS processor spending 99% of inference time bottlenecked on memory reads.

Colocated Contention

When you run prefill and decode on the same GPU in the same inference engine (the traditional "aggregated" approach):

Time → GPU: |PPPPPP|D|D|PPPPPP|D|D|D|PPPP|D|D|D|D|D| ↑ ↑ New Decode steps interrupted request while prefill runs arrives

Where P = prefill (GPU pegged at 100% compute) and D = decode (GPU at ~5% compute, mostly memory reads).

Head-of-line blocking: When a long prefill runs, all in-progress decode requests pause. A user mid-conversation suddenly experiences a 2-3 second gap in token streaming because someone else submitted a 10,000-token document for analysis.

Resource mismatch: The optimal tensor parallelism (TP) degree differs between phases. Prefill benefits from smaller TP (each GPU has full compute; communication overhead dominates with larger TP). Decode benefits from larger TP (spreading the KV cache read across more GPUs increases effective memory bandwidth). Colocated, you must pick one TP degree for both—wrong for both.

Batch composition problems: To maximize throughput, you want to batch many decode requests together (more tokens generated per forward pass). But prefill requests are so compute-heavy that batching them with decode requests destroys decode throughput for all in-flight requests.

The Disaggregated Solution

Separate prefill and decode onto different GPU instances, connected by fast KV cache transfer:

Request arrives │ ▼ ┌─────────────┐ KV Cache ┌─────────────────┐ │ Prefill │─────────────► Decode Workers │─── Tokens stream to user │ Workers │ Transfer │ (many replicas) │ │ (fewer) │ via NIXL │ │ └─────────────┘ └─────────────────┘ Compute-bound Memory-bound High TP High TP Batch by input length Batch by active requests

Now each tier can be scaled, configured, and optimized independently:

• Prefill workers: Optimized for compute (lower TP, larger batches of similar-length inputs)

• Decode workers: Optimized for memory bandwidth (higher TP to spread KV cache reads, many concurrent requests)

• Scale ratio: For RAG workloads (long input, short output) you might run 1 prefill worker per 4 decode workers. For reasoning workloads (short input, very long output) that ratio inverts

The performance claim of 2-3x improvement versus monolithic serving is real, and it comes from three sources:

1. Higher GPU utilization on both tiers (each does what it's good at)

2. Elimination of prefill-decode head-of-line blocking (decode latency is no longer spiked by concurrent prefills)

3. Independent scaling (add decode capacity when ITL degrades; add prefill capacity when TTFT degrades)

The Stack: SGLang + NVIDIA Dynamo

We're using two complementary open-source frameworks. Understanding what each one does prevents confusion about which layer to configure for which problem.

SGLang: The Inference Runtime

SGLang (Structured Generation Language) is a high-performance LLM inference serving framework that handles:

• Efficient prefix caching: Radix tree-based KV cache management that reuses cached computations for shared prefixes (system prompts, few-shot examples)

• Continuous batching: Dynamic batching that maximizes GPU utilization without fixed batch size constraints

• Tensor parallelism: Distributed inference across multiple GPUs within a node

• PD disaggregation support: Native support for disaggregated prefill/decode via a Proxy Server → Prefill Server → Decode Server architecture

• OpenAI-compatible API: Drop-in replacement for OpenAI's chat completions and completions API

SGLang's PD disaggregation uses three components:

Client │ ▼ Proxy Server ← Routes to P/D pairs based on load balancing │ ├──► Prefill Server 1 ← Runs model forward pass, generates KV cache │ │ KV transfer (via Mooncake or NIXL) │ ▼ │ Decode Server 1 ← Receives KV cache, generates tokens │ └──► Prefill Server 2 │ ▼ Decode Server 2

The Proxy Server pairs incoming requests with available Prefill/Decode server pairs. The Prefill Server runs the initial forward pass and transfers KV cache to a specific Decode Server. The Decode Server then handles autoregressive generation until the response is complete.

NVIDIA Dynamo: The Distributed Serving Orchestrator

NVIDIA Dynamo (Apache 2.0, open source) sits above the inference runtime and handles fleet-level orchestration. It is inference-engine agnostic—it works with SGLang, vLLM, TRT-LLM, or custom engines. Think of it as the "Kubernetes controller" for your inference workload, not a replacement for the inference engine itself.github+1

Dynamo's five key components:

1. Disaggregated Serving Coordinator

Manages the xPyD configuration—x prefill workers and y decode workers. Handles the lifecycle of disaggregated inference: routing requests to prefill, coordinating KV transfer, handing off to decode. The configuration is runtime-reconfigurable: add or remove prefill and decode workers without restarting the system.

# xPyD configuration: 1 prefill worker, 4 decode workers (1P4D)
# Optimal for RAG workloads with long inputs
--num-prefill-workers=1 --num-decode-workers=4

2. Dynamo Planner

The intelligent autoscaler for disaggregated serving. Unlike Kubernetes HPA (which scales on CPU/memory), Dynamo Planner scales on inference-specific metrics:

• TTFT (Time To First Token) trend

• ITL (Inter-Token Latency) trend

• Prefill queue depth

• Decode queue depth

• Request rate and sequence length distribution

Based on these signals, the Planner dynamically adjusts the P:D worker ratio and can switch between aggregated and disaggregated serving depending on load. If traffic drops significantly, it may consolidate to aggregated serving to reduce KV transfer overhead. If TTFT degrades, it scales prefill workers. If ITL degrades, it scales decode workers.​

3. Smart Router

Routes incoming requests across workers to minimize KV cache recomputation. For each incoming request, the Smart Router:

1. Computes a "prefix overlap score" against cached KV entries across all workers

2. Routes the request to the worker with the highest overlap score

3. Falls back to load-based routing when no significant cache overlap exists

This is especially impactful for multi-turn conversations (the conversation history is already cached on a specific worker), system prompt sharing across users (the system prompt KV is reused), and agentic workflows (repeated tool schemas are cached).

4. KV Cache Block Manager

Implements tiered KV cache storage:

Tier 1: GPU HBM (fastest, ~80GB on H100)      ← Active KV cache
Tier 2: CPU DRAM (cheap, ~1TB on large servers) ← Recent, likely to be reused
Tier 3: Local NVMe SSD                          ← Historical cache
Tier 4: Amazon S3                               ← Cold/archival cache

When GPU memory is under pressure, less-recently-used KV blocks are evicted to CPU memory. If they're needed again, they're fetched back without full recomputation. For very old entries, they can be evicted to S3 and recomputed if needed. This dramatically increases the effective KV cache size available to the system.

5. NIXL (NVIDIA Inference Transfer Library)

The low-latency communication backbone for Dynamo. NIXL handles:​

• GPU-to-GPU KV transfer: Direct VRAM-to-VRAM transfer without CPU involvement, using RDMA over EFA on AWS

• NVLink optimization: When prefill and decode are on the same NVLink fabric, uses NVLink instead of EFA

• Non-blocking transfers: KV cache transfers happen asynchronously—the prefill GPU continues serving other requests while transferring KV cache to the decode GPU

• Memory descriptor caching: NIXL metadata (describing allocated KV blocks' physical addresses) is stored in etcd. Only block IDs, not full descriptors, need to be sent per request

The non-blocking transfer is critical. Without it, the prefill GPU would stall waiting for KV transfer to complete before accepting the next request—eliminating the throughput benefit of disaggregation.

Infrastructure Setup on EKS

Let's build the infrastructure. We'll use the AWS Labs ai-on-eks reference implementation as our foundation.

EKS Cluster with GPU and CPU Node Groups

The architecture uses a mixed-instance cluster:

• CPU nodes (c7i.16xlarge): Run Dynamo control plane (API Store, NATS, Router, Frontend)

• GPU nodes (g6e.12xlarge or p5.48xlarge): Run prefill and decode workers

# eks-cluster.tf
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.0"

  cluster_name    = "dynamo-inference"
  cluster_version = "1.32"

  vpc_id                         = module.vpc.vpc_id
  subnet_ids                     = module.vpc.private_subnets
  # Single AZ deployment - critical for EFA (inter-node low latency)
  # EFA requires all communicating nodes to be in the same AZ
  control_plane_subnet_ids       = [module.vpc.private_subnets[0]]

  enable_irsa = true

  cluster_addons = {
    coredns                = { most_recent = true }
    kube-proxy             = { most_recent = true }
    vpc-cni                = { most_recent = true }
    aws-ebs-csi-driver     = { most_recent = true }
    aws-efs-csi-driver     = { most_recent = true }
  }

  # CPU node group for Dynamo control plane components
  eks_managed_node_groups = {
    dynamo_control_plane = {
      name           = "dynamo-cpu-ng"
      instance_types = ["c7i.16xlarge"]  # 64 vCPU, 128GB RAM
      capacity_type  = "ON_DEMAND"

      min_size     = 1
      max_size     = 3
      desired_size = 1

      # Single AZ for EFA locality
      subnet_ids = [module.vpc.private_subnets[0]]

      labels = {
        workload-type = "dynamo-control"
        "dynamo/role" = "control-plane"
      }
    }
  }
}

# GPU nodes managed by Karpenter for dynamic scaling
# Karpenter handles g6e for testing, p5/p5e for production H100

Karpenter NodePool for GPU Workers

Using Karpenter for GPU nodes enables sub-60-second provisioning when Dynamo Planner requests scale-out:

# karpenter-gpu-nodepool.yaml
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu-inference-workers
spec:
  template:
    metadata:
      labels:
        workload-type: gpu-inference
        nvidia.com/gpu.present: "true"
    spec:
      nodeClassRef:
        apiVersion: karpenter.k8s.aws/v1
        kind: EC2NodeClass
        name: gpu-nodeclass

      requirements:
        # Prefer G6e for L40S GPUs (inference-optimized)
        # P5 for H100 SXM (highest performance, highest cost)
        - key: node.kubernetes.io/instance-type
          operator: In
          values:
            - g6e.12xlarge   # 1x L40S 48GB - dev/staging
            - g6e.48xlarge   # 4x L40S 48GB - production prefill
            - p5.48xlarge    # 8x H100 SXM 80GB - production decode

        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]

        - key: topology.kubernetes.io/zone
          operator: In
          values: ["us-east-1a"]  # Must match control plane AZ for EFA

      taints:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule

  # Karpenter will provision matching GPU instances within 60 seconds
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 300s  # Terminate idle GPU nodes after 5 min (cost savings)

  limits:
    nvidia.com/gpu: 64  # Max 64 GPUs in this pool

---
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: gpu-nodeclass
spec:
  amiSelectorTerms:
    - alias: al2023@latest   # AWS-optimized AMI with NVIDIA drivers

  instanceProfile: KarpenterGPUNodeInstanceProfile

  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: dynamo-inference
        topology.kubernetes.io/zone: us-east-1a

  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: dynamo-inference

  # EFA configuration for low-latency inter-node communication
  networkInterfaces:
    - deviceIndex: 0
      associatePublicIPAddress: false
    - deviceIndex: 1
      interfaceType: efa     # EFA interface for NIXL transfers
      deleteOnTermination: true

  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 500Gi
        volumeType: gp3
        iops: 16000
        throughput: 1000  # High throughput for model weight loading
        encrypted: true

EFS for Shared Model Storage

All prefill and decode workers need access to model weights. EFS provides read-many access across all pods:

# efs-model-storage.tf
resource "aws_efs_file_system" "model_storage" {
  creation_token   = "dynamo-model-weights"
  performance_mode = "generalPurpose"
  throughput_mode  = "elastic"  # Elastic throughput for burst model loads
  encrypted        = true
  kms_key_id       = aws_kms_key.efs.arn

  lifecycle_policy {
    transition_to_ia = "AFTER_7_DAYS"   # Archive cold model versions
  }

  tags = { Name = "dynamo-model-weights" }
}

resource "aws_efs_mount_target" "model_storage" {
  file_system_id  = aws_efs_file_system.model_storage.id
  subnet_id       = data.aws_subnet.inference_az.id  # Same AZ as GPU nodes
  security_groups = [aws_security_group.efs.id]
}

# efs-storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: efs-model-storage
provisioner: efs.csi.aws.com
parameters:
  provisioningMode: efs-ap
  fileSystemId: fs-xxxxxxxxx
  directoryPerms: "755"
reclaimPolicy: Retain
volumeBindingMode: Immediate
---
# model-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: model-weights-pvc
  namespace: dynamo-cloud
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: efs-model-storage
  resources:
    requests:
      storage: 500Gi  # Enough for several 70B models

Install NVIDIA Dynamo Platform

The Dynamo Kubernetes Operator manages the lifecycle of DynamoGraphDeployment custom resources. Install the platform stack via Helm:

# Step 1: Add NVIDIA Dynamo Helm repo
helm repo add dynamo https://nvidia.github.io/dynamo
helm repo update

# Step 2: Install platform components
# This installs: Operator, NATS (message queue), etcd (metadata store),
# API Store (artifact registry), MinIO (S3-compatible storage for artifacts)
helm install dynamo-platform dynamo/dynamo-platform \
  --namespace dynamo-system \
  --create-namespace \
  --version 1.0.0 \
  --set operator.enabled=true \
  --set nats.enabled=true \
  --set etcd.enabled=true \
  --set apiStore.enabled=true \
  --set apiStore.storage.s3.bucket=dynamo-artifacts \
  --set apiStore.storage.s3.region=us-east-1 \
  --set apiStore.serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::123456789012:role/dynamo-api-store

# Step 3: Verify platform is healthy
kubectl get pods -n dynamo-system

# Expected:
# dynamo-operator-controller-manager-xxx   1/1   Running
# dynamo-nats-0                             1/1   Running
# dynamo-nats-1                             1/1   Running
# dynamo-nats-2                             1/1   Running
# dynamo-etcd-0                             1/1   Running
# dynamo-api-store-xxx                      1/1   Running

# Step 4: Verify CRDs are installed
kubectl get crd | grep dynamo
# dynamographdeployments.dynamo.nvidia.com
# dynamocomponentdeployments.dynamo.nvidia.com

Deploying Disaggregated Inference: Three Architecture Patterns

With the platform running, let's deploy actual inference workloads. We'll cover three patterns in increasing complexity.

Pattern 1: Aggregated (Baseline)

Standard monolithic serving—one service handles both prefill and decode. Use this as your performance baseline before disaggregating:

# dynamo-aggregated.yaml
apiVersion: dynamo.nvidia.com/v1alpha1
kind: DynamoGraphDeployment
metadata:
  name: llama-aggregated
  namespace: dynamo-cloud
spec:
  services:
    Frontend:
      replicas: 1
      envs:
        - name: BACKEND_SERVICE
          value: "VllmWorker"
        - name: DYN_ROUTER_MODE
          value: "round_robin"  # No KV-aware routing needed for aggregated
      resources:
        requests:
          cpu: "8"
          memory: "16Gi"

    VllmWorker:  # SGLang worker handling both prefill and decode
      replicas: 2
      envs:
        - name: MODEL_PATH
          value: "/mnt/models/meta-llama/Meta-Llama-3.1-8B-Instruct"
        - name: TENSOR_PARALLEL_SIZE
          value: "1"
        - name: MAX_NUM_SEQS
          value: "128"
        - name: GPU_MEMORY_UTILIZATION
          value: "0.90"
      resources:
        requests:
          nvidia.com/gpu: "1"
          memory: "24Gi"
          cpu: "8"
        limits:
          nvidia.com/gpu: "1"
      volumeMounts:
        - name: model-weights
          mountPath: /mnt/models
      nodeSelector:
        workload-type: gpu-inference
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule

  volumes:
    - name: model-weights
      persistentVolumeClaim:
        claimName: model-weights-pvc

Pattern 2: Disaggregated 1P4D (RAG-Optimized)

One prefill worker, four decode workers. Optimal for RAG workloads with long input contexts and short outputs:

# dynamo-disaggregated-1p4d.yaml
apiVersion: dynamo.nvidia.com/v1alpha1
kind: DynamoGraphDeployment
metadata:
  name: llama-disaggregated-1p4d
  namespace: dynamo-cloud
spec:
  services:
    # Frontend: HTTP API entry point, routes to disaggregated router
    Frontend:
      replicas: 2  # Stateless - scale independently
      envs:
        - name: BACKEND_SERVICE
          value: "VllmWorker"
        - name: DYN_ROUTER_MODE
          value: "kv"  # Enable KV-cache aware routing
        - name: DYN_ROUTER_KV_EVICTION_POLICY
          value: "lru"
        - name: FRONTEND_PORT
          value: "3000"
      resources:
        requests:
          cpu: "8"
          memory: "16Gi"
      # Runs on CPU nodes - no GPU needed
      nodeSelector:
        "dynamo/role": control-plane

    # Decode Workers: many replicas, memory-bandwidth optimized
    VllmWorker:
      replicas: 4  # 4 decode workers - "4D" in 1P4D
      envs:
        - name: MODEL_PATH
          value: "/mnt/models/meta-llama/Meta-Llama-3.1-8B-Instruct"
        - name: TENSOR_PARALLEL_SIZE
          value: "1"
        - name: MAX_NUM_SEQS
          value: "256"    # High concurrency - decode is low-compute
        - name: GPU_MEMORY_UTILIZATION
          value: "0.85"   # Leave room for KV transfer buffers
        - name: ENABLE_KV_TRANSFER
          value: "true"
        - name: NIXL_SIDE
          value: "receiver"  # Decode side receives KV cache from prefill
        - name: KV_TRANSFER_BACKEND
          value: "nixl"
      resources:
        requests:
          nvidia.com/gpu: "1"
          memory: "32Gi"
          cpu: "16"
        limits:
          nvidia.com/gpu: "1"
      volumeMounts:
        - name: model-weights
          mountPath: /mnt/models
      nodeSelector:
        workload-type: gpu-inference
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule

    # Prefill Workers: fewer replicas, compute-optimized
    PrefillWorker:
      replicas: 1  # 1 prefill worker - "1P" in 1P4D
      envs:
        - name: MODEL_PATH
          value: "/mnt/models/meta-llama/Meta-Llama-3.1-8B-Instruct"
        - name: TENSOR_PARALLEL_SIZE
          value: "1"
        - name: IS_PREFILL_WORKER
          value: "true"    # Marks this as prefill-only
        - name: MAX_NUM_SEQS
          value: "32"      # Fewer concurrent prefills (compute-heavy)
        - name: GPU_MEMORY_UTILIZATION
          value: "0.80"
        - name: ENABLE_KV_TRANSFER
          value: "true"
        - name: NIXL_SIDE
          value: "sender"  # Prefill side sends KV cache to decode
        - name: KV_TRANSFER_BACKEND
          value: "nixl"
        # Prefill queue via NATS
        - name: PREFILL_QUEUE
          value: "nats://dynamo-nats:4222/prefill-queue"
        - name: ETCD_ENDPOINTS
          value: "http://dynamo-etcd:2379"
      resources:
        requests:
          nvidia.com/gpu: "1"
          memory: "32Gi"
          cpu: "16"
          # EFA device for NIXL KV transfer
          vpc.amazonaws.com/efa: "1"
        limits:
          nvidia.com/gpu: "1"
          vpc.amazonaws.com/efa: "1"
      volumeMounts:
        - name: model-weights
          mountPath: /mnt/models
      nodeSelector:
        workload-type: gpu-inference
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule

  volumes:
    - name: model-weights
      persistentVolumeClaim:
        claimName: model-weights-pvc

Apply and verify:

kubectl apply -f dynamo-disaggregated-1p4d.yaml

# Watch pods come up
kubectl get pods -n dynamo-cloud -w

# Expected after ~5 minutes:
# llama-disaggregated-1p4d-frontend-xxx      1/1  Running  (2 pods)
# llama-disaggregated-1p4d-vllmworker-xxx    1/1  Running  (4 pods)
# llama-disaggregated-1p4d-prefillworker-xxx 1/1  Running  (1 pod)

# Check DynamoGraphDeployment status
kubectl describe dynamographdeployment llama-disaggregated-1p4d -n dynamo-cloud

# Verify NIXL connections between prefill and decode
kubectl logs -n dynamo-cloud llama-disaggregated-1p4d-prefillworker-xxx | grep -i nixl
# Expected: "NIXL connection established to decode worker xxx at 10.0.x.x:xxxx"

Pattern 3: Multi-Node with Larger Models

For 70B+ parameter models that don't fit on a single GPU, configure tensor parallelism across multiple GPUs:

# dynamo-multinode-70b.yaml
apiVersion: dynamo.nvidia.com/v1alpha1
kind: DynamoGraphDeployment
metadata:
  name: llama-70b-multinode
  namespace: dynamo-cloud
spec:
  services:
    Frontend:
      replicas: 2
      envs:
        - name: BACKEND_SERVICE
          value: "VllmWorker"
        - name: DYN_ROUTER_MODE
          value: "kv"
      resources:
        requests:
          cpu: "8"
          memory: "16Gi"
      nodeSelector:
        "dynamo/role": control-plane

    # Decode Workers: TP=4 across 4 GPUs per replica
    VllmWorker:
      replicas: 2   # 2 decode replicas, each spanning 4 GPUs (8 GPUs total)
      envs:
        - name: MODEL_PATH
          value: "/mnt/models/meta-llama/Meta-Llama-3.1-70B-Instruct"
        - name: TENSOR_PARALLEL_SIZE
          value: "4"   # 4-way tensor parallel for 70B model
        - name: MAX_NUM_SEQS
          value: "128"
        - name: GPU_MEMORY_UTILIZATION
          value: "0.85"
        - name: ENABLE_KV_TRANSFER
          value: "true"
        - name: NIXL_SIDE
          value: "receiver"
        - name: KV_TRANSFER_BACKEND
          value: "nixl"
        # NCCL for intra-TP communication
        - name: NCCL_SOCKET_NTHREADS
          value: "4"
        - name: NCCL_NSOCKS_PERTHREAD
          value: "4"
      resources:
        requests:
          nvidia.com/gpu: "4"    # 4 GPUs per replica
          memory: "256Gi"
          cpu: "64"
          vpc.amazonaws.com/efa: "4"  # 4 EFA interfaces for TP communication
        limits:
          nvidia.com/gpu: "4"
          vpc.amazonaws.com/efa: "4"
      # Pod affinity: all 4 GPUs must be on same node (p5.48xlarge has 8 GPUs)
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: node.kubernetes.io/instance-type
                    operator: In
                    values:
                      - p5.48xlarge  # 8x H100 SXM
      volumeMounts:
        - name: model-weights
          mountPath: /mnt/models
      nodeSelector:
        workload-type: gpu-inference
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule

    # Prefill Workers: TP=2 (prefill is compute-bound; lower TP is more efficient)
    PrefillWorker:
      replicas: 1
      envs:
        - name: MODEL_PATH
          value: "/mnt/models/meta-llama/Meta-Llama-3.1-70B-Instruct"
        - name: TENSOR_PARALLEL_SIZE
          value: "2"   # Lower TP for prefill (compute-bound, communication overhead matters less)
        - name: IS_PREFILL_WORKER
          value: "true"
        - name: MAX_NUM_SEQS
          value: "16"
        - name: GPU_MEMORY_UTILIZATION
          value: "0.80"
        - name: ENABLE_KV_TRANSFER
          value: "true"
        - name: NIXL_SIDE
          value: "sender"
        - name: KV_TRANSFER_BACKEND
          value: "nixl"
        - name: PREFILL_QUEUE
          value: "nats://dynamo-nats:4222/prefill-queue-70b"
        - name: ETCD_ENDPOINTS
          value: "http://dynamo-etcd:2379"
      resources:
        requests:
          nvidia.com/gpu: "2"
          memory: "128Gi"
          cpu: "32"
          vpc.amazonaws.com/efa: "2"
        limits:
          nvidia.com/gpu: "2"
          vpc.amazonaws.com/efa: "2"
      volumeMounts:
        - name: model-weights
          mountPath: /mnt/models
      nodeSelector:
        workload-type: gpu-inference
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule

  volumes:
    - name: model-weights
      persistentVolumeClaim:
        claimName: model-weights-pvc

The asymmetric tensor parallelism—TP=2 for prefill, TP=4 for decode—is a key optimization. Prefill is compute-bound and benefits from fewer communication-heavy TP splits. Decode is memory-bound and benefits from larger TP that distributes the KV cache read across more GPU HBM buses.

SGLang-Specific Configuration and Optimizations

When using SGLang as the inference backend for Dynamo, several SGLang-specific parameters drive significant performance:

Prefix Caching Configuration

SGLang's radix tree KV cache is the most impactful single optimization for multi-turn and shared-prefix workloads:

# SGLang server startup configuration
# Applied via environment variable SGLANG_ARGS in Dynamo worker pods

SGLANG_ARGS = """
  --model-path /mnt/models/Qwen/Qwen2.5-72B-Instruct
  --tp 4
  --mem-fraction-static 0.85
  --chunked-prefill-size 4096    # Process long prefills in chunks (prevents blocking)
  --max-prefill-tokens 32768     # Max tokens in a single prefill batch
  --max-running-requests 512     # Max concurrent requests in flight
  --schedule-conservativeness 0.9
  
  # Prefix caching (critical for RAG and multi-turn)
  --enable-prefix-caching
  --prefix-cache-mem-fraction 0.1  # Reserve 10% of KV cache for prefix storage
  
  # For disaggregated prefill workers only
  --is-disaggregated-prefill-worker
  --disaggregated-prefill-role prefill
  
  # For disaggregated decode workers only
  --is-disaggregated-decode-worker
  --disaggregated-prefill-role decode
  --decode-log-interval 100
"""

Chunked Prefill

Long-context requests can dominate GPU cycles for hundreds of milliseconds, blocking all in-flight decode requests. Chunked prefill breaks long inputs into fixed-size chunks processed over multiple iterations:

# Without chunked prefill: long input blocks decode
# Prefill queue: [8192 tokens] → GPU locked for ~400ms
# All decode requests stall

# With chunked prefill (chunk size = 2048):
# Iteration 1: [2048 tokens prefill] + [batch of decode steps]
# Iteration 2: [2048 tokens prefill] + [batch of decode steps]
# Iteration 3: [2048 tokens prefill] + [batch of decode steps]
# Iteration 4: [2048 tokens prefill] + [batch of decode steps]
# Decode latency impact: minimal

--chunked-prefill-size 2048   # Smaller = lower decode interference, higher prefill latency
                              # Larger = higher decode interference, lower prefill latency
                              # 2048-4096 is the typical sweet spot

Chunked prefill is complementary to disaggregation—in aggregated mode it softens the decode interference; in disaggregated mode it allows prefill workers to interleave long prefills efficiently.

SGLang Router Configuration for PD Disaggregation

When using SGLang's built-in PD disaggregation (without Dynamo, for simpler deployments):

# Proxy server (load balancer for P/D pairs)
python -m sglang.srt.entrypoints.openai.api_server \
  --port 8080 \
  --router-mode pd \
  --pd-balance-policy shortest_queue \
  --pd-prefill-url http://prefill-svc:8081 \
  --pd-decode-url http://decode-svc-0:8082 \
  --pd-decode-url http://decode-svc-1:8082 \
  --pd-decode-url http://decode-svc-2:8082 \
  --pd-decode-url http://decode-svc-3:8082

# Prefill server
python -m sglang.srt.entrypoints.openai.api_server \
  --model-path /mnt/models/meta-llama/Meta-Llama-3.1-8B-Instruct \
  --port 8081 \
  --tp 1 \
  --disaggregation-mode prefill \
  --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_buffer_device":"cuda","kv_buffer_size":2e9}' \
  --chunked-prefill-size 4096 \
  --max-prefill-tokens 65536

# Decode server (x4 replicas)
python -m sglang.srt.entrypoints.openai.api_server \
  --model-path /mnt/models/meta-llama/Meta-Llama-3.1-8B-Instruct \
  --port 8082 \
  --tp 1 \
  --disaggregation-mode decode \
  --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_buffer_device":"cuda","kv_buffer_size":2e9}' \
  --max-running-requests 256

These become the container command fields in Kubernetes pod specs. When using with Dynamo, these parameters are passed via the envs section of the DynamoGraphDeployment spec.

Autoscaling: Dynamo Planner + Kubernetes HPA

Production inference requires two levels of autoscaling: Dynamo Planner adjusts the P:D worker ratio at the inference-framework level, while Kubernetes HPA/KEDA scales the actual pod count.

Dynamo Planner Configuration

# dynamo-planner-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: dynamo-planner-config
  namespace: dynamo-cloud
data:
  planner.yaml: |
    # SLO targets that drive scaling decisions
    slo:
      ttft_target_p95_ms: 500       # 500ms p95 TTFT target
      itl_target_p95_ms: 50         # 50ms p95 ITL (inter-token latency) target
      ttft_warning_threshold: 1.2   # Scale prefill when TTFT > 120% of target
      itl_warning_threshold: 1.3    # Scale decode when ITL > 130% of target

    # Scaling behavior
    scaling:
      min_prefill_workers: 1
      max_prefill_workers: 8
      min_decode_workers: 2
      max_decode_workers: 32
      scale_up_cooldown_seconds: 120
      scale_down_cooldown_seconds: 300  # Conservative scale-down to avoid churn

      # When to switch from disaggregated to aggregated
      # (reduces KV transfer overhead at low load)
      aggregated_mode_threshold_rps: 5  # Use aggregated when < 5 req/sec

    # Prefill queue depth triggers
    prefill_queue:
      scale_up_threshold: 10    # Add prefill worker when queue > 10 requests
      scale_down_threshold: 2   # Remove prefill worker when queue < 2 requests

    # Monitoring integration
    metrics:
      prometheus_endpoint: "http://prometheus:9090"
      scrape_interval_seconds: 10

KEDA ScaledObject for Decode Workers

KEDA can scale decode workers based on Prometheus metrics exported by Dynamo:

# keda-scaled-object.yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: decode-worker-scaler
  namespace: dynamo-cloud
spec:
  scaleTargetRef:
    apiVersion: dynamo.nvidia.com/v1alpha1
    kind: DynamoComponentDeployment
    name: llama-disaggregated-1p4d-vllmworker

  pollingInterval: 15  # Check every 15 seconds
  cooldownPeriod: 300  # 5 minutes before scale-down
  minReplicaCount: 2
  maxReplicaCount: 32

  triggers:
    # Scale on Inter-Token Latency p95 (memory bandwidth saturation indicator)
    - type: prometheus
      metadata:
        serverAddress: http://prometheus:9090
        metricName: dynamo_itl_p95_ms
        query: |
          histogram_quantile(0.95,
            rate(dynamo_token_latency_seconds_bucket{
              deployment="llama-disaggregated-1p4d",
              phase="decode"
            }[2m])
          ) * 1000
        threshold: "80"   # Scale out when p95 ITL > 80ms

    # Scale on queue depth
    - type: prometheus
      metadata:
        serverAddress: http://prometheus:9090
        metricName: dynamo_decode_queue_depth
        query: |
          dynamo_pending_requests{
            deployment="llama-disaggregated-1p4d",
            component="decode"
          }
        threshold: "50"   # Scale out when > 50 requests queued
---
# Scale prefill workers on TTFT
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: prefill-worker-scaler
  namespace: dynamo-cloud
spec:
  scaleTargetRef:
    apiVersion: dynamo.nvidia.com/v1alpha1
    kind: DynamoComponentDeployment
    name: llama-disaggregated-1p4d-prefillworker

  pollingInterval: 15
  cooldownPeriod: 180
  minReplicaCount: 1
  maxReplicaCount: 8

  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus:9090
        metricName: dynamo_ttft_p95_ms
        query: |
          histogram_quantile(0.95,
            rate(dynamo_time_to_first_token_seconds_bucket{
              deployment="llama-disaggregated-1p4d",
              component="prefill"
            }[2m])
          ) * 1000
        threshold: "600"  # Scale out when p95 TTFT > 600ms

Observability: Metrics, Logging, and Dashboards

Key Metrics to Track

Install the monitoring stack via Helm:

# kube-prometheus-stack comes with ai-on-eks blueprint
kubectl port-forward -n kube-prometheus-stack \
  svc/kube-prometheus-stack-grafana 3000:80

# Key Dynamo metrics exposed on /metrics endpoints:
# dynamo_time_to_first_token_seconds_bucket  → TTFT distribution
# dynamo_inter_token_latency_seconds_bucket  → ITL distribution
# dynamo_requests_total                      → Request throughput
# dynamo_kv_cache_utilization               → KV cache pressure
# dynamo_prefill_queue_depth                → Prefill backlog
# dynamo_decode_queue_depth                 → Decode backlog
# dynamo_kv_transfer_latency_seconds        → NIXL transfer overhead
# dynamo_tokens_generated_total             → Output throughput (tokens/sec)

Benchmarking Your Deployment

Use the SGLang benchmarking tool to measure real performance:

# Port-forward to the Dynamo frontend
kubectl port-forward -n dynamo-cloud svc/llama-disaggregated-1p4d-frontend 3000:3000 &

# Benchmark: 100 concurrent users, 1000 total requests
# Simulates RAG workload: long input (2000 tokens), short output (200 tokens)
python -m sglang.bench_serving \
  --backend sglang \
  --base-url http://localhost:3000 \
  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
  --num-prompts 1000 \
  --request-rate 50 \
  --dataset-name random \
  --random-input-len 2000 \
  --random-output-len 200 \
  --random-range-ratio 0.5

# Benchmark: Reasoning workload (short input, long output)
python -m sglang.bench_serving \
  --backend sglang \
  --base-url http://localhost:3000 \
  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
  --num-prompts 500 \
  --request-rate 20 \
  --dataset-name random \
  --random-input-len 500 \
  --random-output-len 2000 \
  --random-range-ratio 0.3

Expected Performance Numbers

Based on deployments on AWS GPU instances with disaggregated serving:

Llama 3.1 8B on g6e.12xlarge (1x L40S 48GB) — 1P4D configuration:

Qwen2.5 72B on p5.48xlarge (8x H100 SXM 80GB) — 1P4D with TP=4:

The p95 TTFT improvements are larger than p50 improvements because disaggregation eliminates the long-tail head-of-line blocking—the case where a short request happened to be queued behind a 10,000-token prefill.

Production Considerations

Network Configuration for NIXL/EFA

KV transfer latency directly impacts TTFT in disaggregated mode. EFA is mandatory for production deployments:

# Ensure EFA device plugin is installed
kubectl get pods -n kube-system | grep efa
# aws-efa-k8s-device-plugin-xxx  1/1  Running

# EFA device plugin DaemonSet (if not pre-installed)
kubectl apply -f https://raw.githubusercontent.com/aws/eks-charts/master/stable/aws-efa-k8s-device-plugin/crds/efa-device-plugin.yaml

# Verify EFA is available on GPU nodes
kubectl get node <gpu-node> -o json | jq '.status.allocatable | with_entries(select(.key | contains("efa")))'
# Expected: {"vpc.amazonaws.com/efa": "1"} (g6e) or {"vpc.amazonaws.com/efa": "32"} (p5.48xlarge)

All GPU worker pods (prefill and decode) must be in the same AZ for EFA to work. EFA doesn't route across Availability Zone boundaries. This is why we constrain all nodes to us-east-1a in the Karpenter NodePool.

Model Loading Strategy

Loading a 70B model from S3 or EFS at pod startup takes 5-15 minutes. Three patterns to manage this:

Pattern 1: EFS pre-loaded weights (recommended for stable models)

# Init container pre-warms weights from S3 to EFS at deployment time
initContainers:
  - name: model-downloader
    image: amazon/aws-cli:latest
    command:
      - aws
      - s3
      - sync
      - s3://my-model-bucket/llama-3.1-8b/
      - /mnt/models/meta-llama/Meta-Llama-3.1-8B-Instruct/
    env:
      - name: AWS_DEFAULT_REGION
        value: us-east-1
    volumeMounts:
      - name: model-weights
        mountPath: /mnt/models
    resources:
      requests:
        cpu: "4"
        memory: "8Gi"

Pattern 2: Hugging Face Hub with token

envs:
  - name: HF_TOKEN
    valueFrom:
      secretKeyRef:
        name: hf-credentials
        key: token
  - name: HF_HOME
    value: /mnt/models/hf-cache
  # Worker auto-downloads on first start, caches to EFS
  - name: MODEL_PATH
    value: "meta-llama/Meta-Llama-3.1-8B-Instruct"

Pattern 3: Karpenter node startup hook (fastest for scale-out)

# EC2NodeClass userData: pre-load model weights during node bootstrap
# before EKS joins the cluster - model is ready when pod starts
userData: |
  #!/bin/bash
  mkdir -p /mnt/models
  aws s3 sync s3://my-model-bucket/llama-3.1-8b/ /mnt/models/llama-3.1-8b/ \
    --no-progress --exact-timestamps

Security: IRSA for Model Access

# IRSA for GPU worker pods to access S3 model bucket
module "dynamo_worker_irsa" {
  source = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"

  role_name = "dynamo-worker-role"

  oidc_providers = {
    main = {
      provider_arn               = module.eks.oidc_provider_arn
      namespace_service_accounts = ["dynamo-cloud:dynamo-worker"]
    }
  }
}

resource "aws_iam_role_policy" "model_s3_access" {
  name = "dynamo-model-s3-access"
  role = module.dynamo_worker_irsa.iam_role_name

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect   = "Allow"
      Action   = ["s3:GetObject", "s3:ListBucket"]
      Resource = [
        "arn:aws:s3:::my-model-bucket",
        "arn:aws:s3:::my-model-bucket/*"
      ]
    }]
  })
}

Graceful Shutdown for In-Flight Requests

GPU pods need careful shutdown handling—a decode worker with 100 active requests can't just terminate:

# In the DynamoGraphDeployment worker spec
lifecycle:
  preStop:
    exec:
      command:
        - /bin/sh
        - -c
        - |
          # Notify Dynamo router to stop sending new requests
          curl -X POST http://localhost:8080/drain
          # Wait for in-flight requests to complete (max 120s)
          timeout 120 sh -c 'while [ \((curl -s http://localhost:8080/metrics | grep dynamo_active_requests | awk "{print \)2}") -gt 0 ]; do sleep 5; done'
          echo "All requests drained"

terminationGracePeriodSeconds: 180  # 3 minutes to drain + shutdown

Cost Model: When Does Disaggregation Pay Off?

Disaggregated serving adds infrastructure complexity and slightly increases GPU count (a 1P4D deployment uses 5 GPUs versus 4 in a pure aggregated deployment). Here's when the trade-off is clearly positive:

Workload characteristics that favor disaggregation:

• Average input tokens > 512 (RAG, document processing, long-context chat)

• P95/P50 TTFT ratio > 2.0 in aggregated mode (indicates head-of-line blocking)

• Concurrent request count > 20

• Mix of short and long inputs (very common in production—user messages vary widely)

• Reasoning models (DeepSeek-R1, o1-class) with very long decode sequences

Cost analysis at scale (10 GPU nodes):

The same hardware cost produces 2x the throughput in disaggregated mode, cutting cost per million tokens in half. Alternatively, you can serve the same throughput with 20% fewer GPUs at the same cost.

Wrapping Up

The combination of SGLang's efficient inference runtime and NVIDIA Dynamo's disaggregated serving orchestration fundamentally changes the GPU utilization equation for LLM inference at scale. The 2-3x performance improvement versus monolithic serving isn't marketing math—it comes from solving a real physics problem: prefill is compute-bound, decode is memory-bound, and hardware optimized for one is wrong for the other.

On Amazon EKS, the architecture composes cleanly: Karpenter handles GPU node provisioning in under 60 seconds, EFA over NIXL delivers GPU-to-GPU KV transfer with microsecond overhead, EFS serves model weights to all pods without per-pod downloads, and the Dynamo Operator manages the entire disaggregated serving graph as a single Kubernetes custom resource. The result is infrastructure that scales gracefully with demand, automatically adjusts the prefill-to-decode worker ratio based on real SLO signals, and routes each request to the worker most likely to have its KV cache already loaded.

For teams running LLM inference today on EKS—whether serving Llama, Qwen, DeepSeek, or proprietary models—disaggregated serving with NVIDIA Dynamo and SGLang is the clearest path to simultaneously improving user-facing latency and reducing per-token infrastructure cost. The implementation complexity is real but manageable, and the AWS Labs ai-on-eks blueprint provides a production-ready starting point that compresses weeks of infrastructure work into a single install.sh execution.\

The HPC team says: "We need 32 GPUs reserved for our molecular dynamics simulation starting Monday. It runs for 14 hours and cannot be interrupted."

The ML team says: "We're launching a distributed fine-tuning run Monday morning. We need between 16 and 48 GPUs—elastic, not reserved—and Ray handles the scaling."

The platform team says: "We're serving a production LLM that needs 8 GPUs on permanent standby. It cannot be preempted under any circumstances."

The GPU fleet has 64 GPUs across eight p5.48xlarge nodes.

The infrastructure answer right now is: three separate clusters, three separate node pools, each team with their own dedicated allocation. The HPC cluster runs AWS ParallelCluster with Slurm. The ML team has an EKS cluster with KubeRay. The inference team has another EKS cluster with vLLM. All three pools average roughly 40-50% utilization individually. Combined: you're spending $180,000 per month on GPU compute and using less than half of it productively.

There is a better architecture. It requires composing four technologies that nobody has yet written about as a complete system: AWS Slinky (Slurm on EKS), KubeRay with NVIDIA KAI Scheduler, NVIDIA GPU Operator with MIG, and Karpenter—all running on a single EKS cluster, sharing a single GPU fleet, without any of the three teams changing how they submit jobs.

This post is the complete implementation guide.

This post draws on the AWS Containers blog on Slinky, the NVIDIA KAI Scheduler + KubeRay integration guide, the AWS blog on MIG on EKS, and the ai-on-eks Slinky blueprint.

Understanding the Root Problem: Three Schedulers, One Fleet

Before touching a terminal, let's be precise about why this is hard. It isn't a "just use one scheduler" problem. Each of the three workload types has scheduling requirements that are genuinely incompatible with a single scheduler:

Slurm batch jobs require:

• Gang scheduling: All N nodes must start simultaneously. A molecular dynamics simulation that gets 16 of its 32 nodes immediately is useless—MPI requires full allocation

• Reservations: HPC teams plan experiments days in advance. Slurm's --reservation flag guarantees resources at a specific time

• Fairshare accounting: Resource usage history determines future priority across users and groups

• Job dependencies: --dependency=afterok:12345 — job B only runs after job A succeeds

• Wall-clock limits: Jobs are terminated after a maximum runtime to prevent runaway compute

Ray distributed training requires:

• Elastic scaling: The cluster scales between minReplicas and maxReplicas based on available capacity and the autoscaler policy

• Fault tolerance: Ray can checkpoint and resume from worker failures without full job restart

• Non-deterministic resource consumption: A hyperparameter search job doesn't know how many GPUs it needs until it starts

• Head-worker affinity: The head node must be schedulable before workers; the head doesn't do GPU work but must exist throughout the job lifetime

Inference serving requires:

• Guaranteed resources: Never preempted, never evicted, never consolidated away

• Fast restart: If a pod dies, Kubernetes must reschedule it within seconds, not minutes

• Persistent availability: The service must be up 24/7, not subject to batch job pressure

• Session affinity: KV cache locality means the same user should ideally hit the same pod

No single scheduler handles all three correctly out of the box. The key insight that makes the unified architecture possible is this: with Slinky, Slurm slurmd worker nodes are Kubernetes pods. That means every GPU workload in the system—Slurm jobs, Ray workers, inference deployments—eventually becomes a Kubernetes pod. And if everything is a pod, a single scheduling layer (NVIDIA KAI Scheduler) can apply consistent priority, preemption, and gang scheduling policies across all of them.

Architecture Overview

╔══════════════════════════════════════════════════════════════════════════╗ ║ SUBMISSION INTERFACES ║ ║ ║ ║ sbatch train.sh ray job submit kubectl apply ║ ║ srun --ntasks=32 RayJob YAML RayService YAML ║ ║ salloc -N 4 KubeRay operator vLLM Deployment ║ ║ │ │ │ ║ ╚═════════╪════════════════════════╪════════════════════════╪══════════════╝ │ │ │ ▼ ▼ ▼ ╔═════════════════════════════════════════════════════════════════════════╗ ║ SLINKY LAYER ║ ║ ║ ║ ┌──────────────────────────────────────────────────────────────────┐ ║ ║ │ SlurmCluster CRD (SchedMD Operator) │ ║ ║ │ slurmctld (controller) │ slurmdbd (accounting) │ slurmrestd │ ║ ║ │ │ Login pods (ssh/sackd) │ │ ║ ║ └──────────────────────────┼───────────────────────┘ │ ║ ║ │ Slurm allocates → creates slurmd pods │ ║ ╚═════════════════════════════╪═════════════════════════════════════════╝ │ All workloads are Kubernetes pods ▼ ╔═════════════════════════════════════════════════════════════════════════╗ ║ NVIDIA KAI SCHEDULER (Arbitration Layer) ║ ║ ║ ║ Priority Queues Gang Scheduling Preemption Policy ║ ║ ├── inference (1000) All pods in a group inference > training ║ ║ ├── interactive (500) start together or > slurm-batch ║ ║ └── slurm-batch (100) none at all Checkpointing hooks ║ ║ ║ ╚════════════════════════════════════════╤════════════════════════════════╝ │ ▼ ╔═════════════════════════════════════════════════════════════════════════╗ ║ KARPENTER NODE PROVISIONER ║ ║ NodePool: gpu-fleet (p5.48xlarge, g6e.48xlarge, g6e.12xlarge) ║ ║ Single AZ (us-east-1a) │ EFA interfaces │ Bin-packing consolidation ║ ╚══════════════════════════════════╤══════════════════════════════════════╝ │ ▼ ╔═════════════════════════════════════════════════════════════════════════╗ ║ NVIDIA GPU OPERATOR (Hardware Layer) ║ ║ Driver DaemonSet │ Device Plugin │ MIG Manager │ DCGM Exporter ║ ║ Time-slicing config │ GFD (topology labels) │ Container Toolkit ║ ╚═════════════════════════════════════════════════════════════════════════╝ │ ┌────────────────────┼────────────────────┐ ▼ ▼ ▼ p5.48xlarge nodes g6e.48xlarge nodes g6e.12xlarge nodes 8x H100 SXM 80GB 4x L40S 48GB 1x L40S 48GB (Slurm MPI / large (Medium training, (Small inference, distributed training) large inference) dev/staging)

The architecture has four layers:nops+3

1. Submission interfaces — unchanged for each team. HPC users keep sbatch. ML engineers keep ray job submit. Platform engineers keep kubectl apply.

2. Slinky — bridges Slurm into the Kubernetes API. slurmd pods are first-class Kubernetes workloads, subject to the same scheduling policies as everything else.

3. NVIDIA KAI Scheduler — the unified arbitration layer. Applies consistent gang scheduling, priority, and preemption policies across Slurm pods, Ray pods, and inference pods.

4. GPU Operator + Karpenter — hardware lifecycle and node provisioning. GPU Operator manages driver state, MIG configuration, and device plugin registration. Karpenter provisions and consolidates nodes based on actual demand.

Part 1: Foundation — EKS Cluster with Mixed Node Groups

The cluster design requires careful separation: CPU nodes for the Slinky control plane (slurmctld, slurmdbd, KAI Scheduler), and GPU nodes for actual workloads. All GPU nodes must be in a single Availability Zone for EFA-based low-latency GPU communication:

# eks-cluster.tf
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.0"

  cluster_name    = "unified-gpu-platform"
  cluster_version = "1.32"

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  enable_irsa = true

  cluster_addons = {
    coredns            = { most_recent = true }
    kube-proxy         = { most_recent = true }
    vpc-cni            = { most_recent = true }
    aws-ebs-csi-driver = { most_recent = true }
    aws-efs-csi-driver = { most_recent = true }
    # EFA device plugin for inter-node GPU communication
    aws-efa-k8s-device-plugin = { most_recent = true }
  }

  eks_managed_node_groups = {
    # System nodes: Slinky control plane, KAI Scheduler, KubeRay operator
    system = {
      name           = "system-ng"
      instance_types = ["c7i.16xlarge"]  # 64 vCPU, 128GB RAM
      capacity_type  = "ON_DEMAND"
      min_size       = 2
      max_size       = 4
      desired_size   = 2

      labels = {
        "node-role"           = "system"
        "workload-type"       = "control-plane"
        "karpenter.sh/do-not-disrupt" = "true"
      }

      taints = [{
        key    = "CriticalAddonsOnly"
        value  = "true"
        effect = "NO_SCHEDULE"
      }]
    }

    # Dedicated Slurm login node (users SSH into this)
    slurm-login = {
      name           = "slurm-login-ng"
      instance_types = ["c7i.4xlarge"]
      capacity_type  = "ON_DEMAND"
      min_size       = 1
      max_size       = 3
      desired_size   = 1

      # Single AZ — same as GPU nodes for internal routing
      subnet_ids = [module.vpc.private_subnets[0]]

      labels = {
        "node-role"     = "slurm-login"
        "workload-type" = "slurm-access"
      }
    }
  }
}

Karpenter NodePools for GPU Workloads

GPU nodes are managed entirely by Karpenter — no static managed node groups. This enables sub-60-second provisioning when Slurm queues fill up or KAI scales a RayCluster:

# karpenter-gpu-nodepool.yaml
---
# Heavy training: p5.48xlarge with 8x H100 SXM
# Used by: Slurm MPI jobs, large Ray training runs
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu-heavy-training
spec:
  template:
    metadata:
      labels:
        workload-tier: heavy-training
        nvidia.com/gpu.product: "A100-SXM4-80GB"  # GFD label after GPU Operator installs
    spec:
      nodeClassRef:
        apiVersion: karpenter.k8s.aws/v1
        kind: EC2NodeClass
        name: gpu-nodeclass

      requirements:
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ["p5.48xlarge"]     # 8x H100 SXM 80GB, 192 vCPU, 2TB RAM, 3200Gbps EFA

        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]

        # Critical: single AZ for EFA locality
        - key: topology.kubernetes.io/zone
          operator: In
          values: ["us-east-1a"]

      taints:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
        - key: workload-tier
          value: heavy-training
          effect: NoSchedule

  disruption:
    # Only consolidate when completely empty — never mid-job
    consolidationPolicy: WhenEmpty
    consolidateAfter: 120s  # 2 min grace after last pod exits

  limits:
    nvidia.com/gpu: 256  # Max 32 p5 nodes (256 H100s) in this pool

---
# Medium inference/training: g6e.48xlarge with 4x L40S
# Used by: inference services, medium Ray jobs, Slurm single-node jobs
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu-inference-training
spec:
  template:
    metadata:
      labels:
        workload-tier: inference-training
    spec:
      nodeClassRef:
        apiVersion: karpenter.k8s.aws/v1
        kind: EC2NodeClass
        name: gpu-nodeclass

      requirements:
        - key: node.kubernetes.io/instance-type
          operator: In
          values:
            - "g6e.48xlarge"   # 4x L40S 48GB — inference optimized
            - "g6e.12xlarge"   # 1x L40S 48GB — small inference / dev

        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand", "spot"]  # Spot for non-critical inference

        - key: topology.kubernetes.io/zone
          operator: In
          values: ["us-east-1a"]

      taints:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule

  disruption:
    # WhenUnderutilized for inference nodes — Karpenter bin-packs aggressively
    consolidationPolicy: WhenUnderutilized
    consolidateAfter: 300s  # 5 min before consolidating inference nodes

  limits:
    nvidia.com/gpu: 128

---
# EC2NodeClass shared by all GPU pools
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: gpu-nodeclass
spec:
  amiSelectorTerms:
    - alias: al2023@latest

  instanceProfile: KarpenterGPUNodeInstanceProfile

  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: unified-gpu-platform
        topology.kubernetes.io/zone: us-east-1a

  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: unified-gpu-platform

  # EFA network interfaces for inter-node GPU communication
  networkInterfaces:
    - deviceIndex: 0
      associatePublicIPAddress: false
    - deviceIndex: 1
      interfaceType: efa
      deleteOnTermination: true

  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 1000Gi
        volumeType: gp3
        iops: 16000
        throughput: 1000   # High throughput for model weight loading from EFS
        encrypted: true

  # User data: pre-configure EFA and set NCCL environment variables
  userData: |
    #!/bin/bash
    # NCCL settings for EFA
    echo "NCCL_SOCKET_IFNAME=efa" >> /etc/environment
    echo "FI_EFA_USE_DEVICE_RDMA=1" >> /etc/environment
    echo "FI_PROVIDER=efa" >> /etc/environment
    echo "NCCL_DEBUG=WARN" >> /etc/environment

    # Set NUMA-aware CPU affinity for GPU processes
    echo "vm.zone_reclaim_mode=0" >> /etc/sysctl.conf
    sysctl -p

Part 2: NVIDIA GPU Operator — Driver Lifecycle and MIG

Install GPU Operator before any GPU workloads are scheduled. It handles driver installation, device plugin registration, and MIG configuration automatically:nvidia+1

# Add NVIDIA Helm repositories
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

# Install GPU Operator with MIG strategy=mixed
# mixed: some GPUs on a node can be in MIG mode, others in full-GPU mode
helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --create-namespace \
  --version v24.9.0 \
  --set driver.enabled=true \
  --set driver.version="550.90.07" \
  --set toolkit.enabled=true \
  --set devicePlugin.enabled=true \
  --set mig.strategy=mixed \
  --set migManager.enabled=true \
  --set migManager.default=all-disabled \
  --set gfd.enabled=true \
  --set dcgmExporter.enabled=true \
  --set dcgmExporter.serviceMonitor.enabled=true \
  --set nodeStatusExporter.enabled=true \
  --set defaultRuntime=containerd

# Verify GPU Operator components are running
kubectl get pods -n gpu-operator
# gpu-operator-xxx                     1/1  Running  (controller)
# gpu-feature-discovery-xxx            1/1  Running  (per GPU node)
# nvidia-container-toolkit-daemonset   1/1  Running  (per GPU node)
# nvidia-dcgm-exporter-xxx             1/1  Running  (per GPU node)
# nvidia-device-plugin-daemonset-xxx   1/1  Running  (per GPU node)
# nvidia-driver-daemonset-xxx          1/1  Running  (per GPU node)
# nvidia-mig-manager-xxx               1/1  Running  (per GPU node)

MIG Configuration Strategy

The three workload types have different GPU isolation requirements:nvidia+1

• Slurm MPI jobs on p5 nodes: Need full H100s — no MIG. Gang-scheduled, known allocation, no sharing needed

• Ray training on g6e.48xlarge: Medium MIG slices (3g.24gb on L40S) for parallel hyperparameter sweeps

• Inference services on g6e.12xlarge: Small MIG slices (1g.6gb) for multi-model serving

# mig-config.yaml
# Applied to GPU Operator via ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: mig-parted-config
  namespace: gpu-operator
data:
  config.yaml: |
    version: v1
    mig-configs:

      # p5.48xlarge nodes: Full GPUs for Slurm MPI jobs
      # No MIG — each slurmd pod gets 1-8 whole H100s
      all-disabled:
        - devices: all
          mig-enabled: false

      # g6e.48xlarge nodes: Mixed strategy for co-scheduled training
      # L40S supports MIG with: 1g.6gb, 2g.12gb, 3g.24gb, 4g.24gb, 7g.48gb
      g6e-mixed-training:
        - devices: [0, 1]
          mig-enabled: true
          mig-devices:
            "3g.24gb": 2   # 2x 24GB slices per GPU (for Ray parallel workers)
        - devices: [2, 3]
          mig-enabled: true
          mig-devices:
            "1g.6gb": 7    # 7x 6GB slices per GPU (for small inference + evals)

      # g6e.12xlarge nodes: Full GPU for single-model inference
      # Single L40S — no MIG needed, one model per node
      all-disabled-small:
        - devices: [0]
          mig-enabled: false

Apply MIG configuration to specific nodes using node labels — GPU Operator's MIG Manager watches for this label and applies the config automatically:

# p5.48xlarge nodes: disable MIG (full GPUs for Slurm)
kubectl label node <p5-node-1> nvidia.com/mig.config=all-disabled

# g6e.48xlarge nodes: mixed training configuration
kubectl label node <g6e-48xl-node-1> nvidia.com/mig.config=g6e-mixed-training

# g6e.12xlarge nodes: full single GPU for inference
kubectl label node <g6e-12xl-node-1> nvidia.com/mig.config=all-disabled-small

# Verify MIG instances are created
kubectl get node <g6e-48xl-node-1> -o json | \
  jq '.status.allocatable | with_entries(select(.key | startswith("nvidia.com/mig")))'
# {
#   "nvidia.com/mig-3g.24gb": "4",   (2 GPUs × 2 slices)
#   "nvidia.com/mig-1g.6gb":  "14"   (2 GPUs × 7 slices)
# }

Part 3: NVIDIA KAI Scheduler — The Arbitration Layer

KAI Scheduler replaces kube-scheduler for GPU workloads. It supports gang scheduling, hierarchical queues, and priority-based preemption—natively integrated with both KubeRay and standard Kubernetes Pods. Because Slinky's slurmd pods are standard Kubernetes Pods, they go through KAI just like everything else.

# Install KAI Scheduler
helm repo add kai-scheduler https://nvidia.github.io/KAI-Scheduler/
helm repo update

helm install kai-scheduler kai-scheduler/kai-scheduler \
  --namespace kai-scheduler \
  --create-namespace \
  --version 0.3.0 \
  --set scheduler.defaultScheduler=false  # KAI handles GPU pods; kube-scheduler handles CPU pods

Priority Classes

Define the three tiers of workload priority:

# priority-classes.yaml
---
# Tier 1: Production inference — never preempted
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: inference-critical
value: 1000
preemptionPolicy: Never
globalDefault: false
description: "Production LLM serving. Never preempted."

---
# Tier 2: Interactive Ray jobs — preempt batch only
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: interactive-training
value: 500
preemptionPolicy: PreemptLowerPriority
globalDefault: false
description: "Interactive ML training. Can preempt batch jobs."

---
# Tier 3: Slurm batch jobs — preemptible by everything above
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: slurm-batch
value: 100
preemptionPolicy: PreemptLowerPriority
globalDefault: true
description: "Slurm batch queue. Preemptible by higher priority workloads."

KAI Queue Hierarchy

KAI's hierarchical queues map to organizational boundaries — departments own parent queues, teams own child queues with configured GPU quotas:

# kai-queues.yaml
---
# Top-level department queues
apiVersion: scheduling.run.ai/v2
kind: Queue
metadata:
  name: hpc-department
spec:
  resources:
    gpu:
      quota: 32          # Guaranteed 32 GPUs
      limit: 56          # Can burst to 56 GPUs when others are idle
      overQuotaWeight: 1
    cpu:
      quota: 512
      limit: 1024
      overQuotaWeight: 1
    memory:
      quota: 2048Gi
      limit: 4096Gi
      overQuotaWeight: 1

---
apiVersion: scheduling.run.ai/v2
kind: Queue
metadata:
  name: ml-department
spec:
  resources:
    gpu:
      quota: 24
      limit: 56
      overQuotaWeight: 2  # Higher weight = gets more burst capacity than HPC
    cpu:
      quota: 384
      limit: 1024
      overQuotaWeight: 2
    memory:
      quota: 1536Gi
      limit: 4096Gi
      overQuotaWeight: 2

---
apiVersion: scheduling.run.ai/v2
kind: Queue
metadata:
  name: inference-team
spec:
  resources:
    gpu:
      quota: 8           # Fixed 8 GPUs — inference must always have these
      limit: 8           # No burst — inference capacity is fixed
      overQuotaWeight: 0 # Doesn't compete for burst

---
# Child queues per team
apiVersion: scheduling.run.ai/v2
kind: Queue
metadata:
  name: hpc-team-a       # Molecular dynamics team
spec:
  parentQueue: hpc-department
  resources:
    gpu:
      quota: 16
      limit: 40
      overQuotaWeight: 1

---
apiVersion: scheduling.run.ai/v2
kind: Queue
metadata:
  name: hpc-team-b       # Climate modeling team
spec:
  parentQueue: hpc-department
  resources:
    gpu:
      quota: 16
      limit: 40
      overQuotaWeight: 1

---
apiVersion: scheduling.run.ai/v2
kind: Queue
metadata:
  name: ml-team-nlp
spec:
  parentQueue: ml-department
  resources:
    gpu:
      quota: 12
      limit: 32
      overQuotaWeight: 2

---
apiVersion: scheduling.run.ai/v2
kind: Queue
metadata:
  name: ml-team-cv       # Computer vision team
spec:
  parentQueue: ml-department
  resources:
    gpu:
      quota: 12
      limit: 32
      overQuotaWeight: 1

The overQuotaWeight field is the fairness lever: when total GPU demand exceeds total supply, KAI distributes burst capacity proportionally to overQuotaWeight values. The ML department has weight=2, HPC has weight=1—ML gets twice the burst allocation when both are competing for idle GPUs.

Part 4: Deploying Slinky — Slurm as a Kubernetes Service

Slinky's Slurm Operator manages the full lifecycle of a Slurm cluster as Kubernetes custom resources. The slurmctld (central manager), slurmdbd (accounting), slurmrestd (REST API), and slurmd (worker) all run as pods.

# Install Slinky Slurm Operator
helm repo add slinky https://slinky.schedmd.com/charts
helm repo update

helm install slurm-operator slinky/slurm-operator \
  --namespace slurm-system \
  --create-namespace \
  --version 1.0.0

# Install Slinky RBAC and CRDs
kubectl apply -f https://raw.githubusercontent.com/SlinkyProject/slurm-operator/main/config/crd/bases/slinky.slurm.net_slurmclusters.yaml
kubectl apply -f https://raw.githubusercontent.com/SlinkyProject/slurm-operator/main/config/crd/bases/slinky.slurm.net_nodesets.yaml

Create the Slurm Cluster

# slurm-cluster.yaml
apiVersion: slinky.slurm.net/v1alpha1
kind: SlurmCluster
metadata:
  name: hpc-cluster
  namespace: slurm-system
spec:
  # Central management daemon
  controller:
    replicas: 1
    image: "registry.schedmd.com/slurm/slurm:24.05"
    resources:
      requests:
        cpu: "8"
        memory: "16Gi"
    nodeSelector:
      node-role: system
    tolerations:
      - key: CriticalAddonsOnly
        operator: Exists
        effect: NoSchedule
    # slurmctld config injected via ConfigMap
    slurmConfig:
      schedulerType: sched/backfill
      selectType: select/cons_tres    # Trackable RESources — required for GPU awareness
      selectTypeParameters: CR_Core_Memory
      gresTypes: gpu
      accountingStorageType: accounting_storage/slurmdbd
      accountingStorageHost: slurmdbd
      jobAcctGatherType: jobacct_gather/cgroup
      jobAcctGatherFrequency: 30
      priorityType: priority/multifactor
      priorityWeightFairshare: 10000
      priorityWeightAge: 1000
      priorityDecayHalfLife: "1-0"    # Fairshare decay over 1 day

  # Accounting daemon + MariaDB backend
  accounting:
    replicas: 1
    image: "registry.schedmd.com/slurm/slurm:24.05"
    resources:
      requests:
        cpu: "4"
        memory: "8Gi"
    # Use RDS for production accounting
    database:
      host: "slurm-db.cluster.us-east-1.rds.amazonaws.com"
      port: 3306
      secretRef:
        name: slurm-db-credentials

  # REST API daemon (used by Slinky NodeSet controller and KEDA)
  restapi:
    replicas: 1
    image: "registry.schedmd.com/slurm/slurm:24.05"
    resources:
      requests:
        cpu: "2"
        memory: "4Gi"

  # Login pods for user SSH access
  login:
    replicas: 2
    image: "registry.schedmd.com/slurm/slurm:24.05"
    resources:
      requests:
        cpu: "4"
        memory: "8Gi"
    # AWS Deep Learning Container base for user environments
    userEnvironment:
      baseImage: "763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-training:2.3.0-gpu-py311-cu121-ubuntu20.04-sagemaker"
    # SSH access via NLB
    service:
      type: LoadBalancer
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
        service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"

  # GPU partitions — different hardware tiers for different job types
  partitions:
    # Heavy MPI partition: p5 nodes, full H100s, gang-scheduled
    - name: gpu-heavy
      maxReplicas: 32          # Up to 32 p5 nodes (256 H100s)
      defaultReplicas: 0       # Scale to zero when idle
      nodeSelector:
        node.kubernetes.io/instance-type: p5.48xlarge
      # Each slurmd pod = 1 Slurm "node" = 8 GPUs
      resources:
        requests:
          nvidia.com/gpu: "8"
          memory: "1536Gi"
          cpu: "176"
          vpc.amazonaws.com/efa: "32"   # All 32 EFA interfaces for NVLink/EFA NCCL
        limits:
          nvidia.com/gpu: "8"
          vpc.amazonaws.com/efa: "32"
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
        - key: workload-tier
          value: heavy-training
          effect: NoSchedule
      # KAI Scheduler integration: Slurm batch priority class
      schedulerName: kai-scheduler
      priorityClassName: slurm-batch
      # Map to KAI queue
      annotations:
        kai.scheduler/queue: hpc-team-a  # Override per user via --partition flag
      # Slurm GRES config for this partition
      gres: "gpu:h100:8"
      features: ["h100", "efa", "p5"]

    # Medium partition: g6e nodes, MIG slices for training
    - name: gpu-medium
      maxReplicas: 16
      defaultReplicas: 0
      nodeSelector:
        node.kubernetes.io/instance-type: g6e.48xlarge
      resources:
        requests:
          nvidia.com/mig-3g.24gb: "4"   # 4 MIG slices (2 GPUs × 2 slices)
          memory: "256Gi"
          cpu: "32"
        limits:
          nvidia.com/mig-3g.24gb: "4"
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      schedulerName: kai-scheduler
      priorityClassName: slurm-batch
      gres: "gpu:l40s:4"
      features: ["l40s", "mig", "g6e"]

FSx for Lustre: High-Performance Shared Storage

Slurm MPI jobs require shared, POSIX-compliant, high-throughput storage for training data and checkpoints:

# fsx-lustre.tf
resource "aws_fsx_lustre_file_system" "hpc_storage" {
  storage_capacity              = 4800    # GB — minimum for PERSISTENT_2
  subnet_ids                    = [data.aws_subnet.inference_az.id]
  security_group_ids            = [aws_security_group.fsx.id]
  deployment_type               = "PERSISTENT_2"
  per_unit_storage_throughput   = 250    # MB/s per TiB — highest tier
  data_compression_type         = "LZ4"

  # Link to S3 for data staging
  import_path      = "s3://training-data-bucket/"
  export_path      = "s3://training-data-bucket/checkpoints/"
  imported_file_chunk_size = 1024  # 1GB chunks for large datasets

  tags = { Name = "hpc-lustre-storage" }
}

# fsx-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: hpc-fsx-storage
  namespace: slurm-system
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: fsx-lustre
  resources:
    requests:
      storage: 4800Gi

Mount the FSx volume in all slurmd pods by adding to the SlurmCluster spec:

# Add to SlurmCluster spec.partitions[*]:
volumeMounts:
  - name: hpc-storage
    mountPath: /fsx
  - name: home-storage
    mountPath: /home

volumes:
  - name: hpc-storage
    persistentVolumeClaim:
      claimName: hpc-fsx-storage
  - name: home-storage
    persistentVolumeClaim:
      claimName: efs-home-storage

Submitting Slurm Jobs

From a login pod, HPC users interact with the cluster exactly as they would on traditional Slurm:

# SSH into a login pod
ssh user@<nlb-dns-name>

# Submit a 32-GPU MPI training job
cat > train_molecular_dynamics.sh << 'EOF'
#!/bin/bash
#SBATCH --job-name=md-simulation
#SBATCH --partition=gpu-heavy
#SBATCH --nodes=4                   # 4 p5 nodes = 32 H100s
#SBATCH --ntasks-per-node=8
#SBATCH --gpus-per-node=8
#SBATCH --mem=0                     # All memory on node
#SBATCH --time=14:00:00
#SBATCH --account=hpc-team-a
#SBATCH --output=/fsx/jobs/%j/output.log

# Load environment
module load cuda/12.1 nccl/2.21

# Run MPI job with EFA-optimized NCCL
srun --mpi=pmix \
     -n 32 \
     -c 24 \
     --gpu-bind=closest \
     python molecular_dynamics.py \
       --config /fsx/configs/md_config.yaml \
       --output /fsx/results/${SLURM_JOB_ID} \
       --checkpoint-dir /fsx/checkpoints/${SLURM_JOB_ID}
EOF

sbatch train_molecular_dynamics.sh

# Monitor job
squeue -u $USER
sinfo  # View partition status and node availability
sacct -j $SLURM_JOB_ID --format=JobID,JobName,Elapsed,AllocGRES,CPUTime

What happens behind the scenes when sbatch runs:

1. slurmctld receives the job submission

2. Slinky's NodeSet controller calls slurmrestd to check partition state

3. If insufficient slurmd pods are running, the NodeSet controller scales up the partition's pod replicas

4. Karpenter detects pending slurmd pods with p5.48xlarge requirements and provisions nodes

5. KAI Scheduler gang-schedules all 4 slurmd pods simultaneously when 4 nodes are available

6. Slurm runs the MPI job across all 4 pods

7. On completion, the NodeSet controller scales pods back toward zero; Karpenter consolidates empty nodes

Part 5: KubeRay with KAI Scheduler — Elastic Training

Install the KubeRay operator and configure it to use KAI Scheduler:

helm repo add kuberay https://ray-project.github.io/kuberay-helm/
helm repo update

# Install with KAI Scheduler integration enabled
helm install kuberay-operator kuberay/kuberay-operator \
  --namespace ray-system \
  --create-namespace \
  --version 1.3.0 \
  --set batchScheduler.name=kai-scheduler \
  --set batchScheduler.enabled=true

Ray Distributed Training Job

# ray-training-job.yaml
apiVersion: ray.io/v1
kind: RayJob
metadata:
  name: llama-finetune-run-001
  namespace: ml-team
  labels:
    kai.scheduler/queue: ml-team-nlp      # Routes to NLP team's KAI queue
    team: nlp
    experiment: llama-3-finetuning
spec:
  # KAI Scheduler handles gang scheduling + preemption
  submissionMode: K8sJobMode

  rayClusterSpec:
    rayVersion: "2.46.0"

    headGroupSpec:
      serviceType: ClusterIP
      rayStartParams:
        num-cpus: "0"       # Head does no compute — pure coordinator
        num-gpus: "0"
        dashboard-host: "0.0.0.0"

      template:
        metadata:
          labels:
            kai.scheduler/queue: ml-team-nlp
          annotations:
            # KAI gang scheduling group — head + all workers start together or not at all
            kai.scheduler.nvidia.com/gang-scheduling-group: "llama-finetune-run-001"
            kai.scheduler.nvidia.com/min-members: "9"  # 1 head + 8 workers
        spec:
          schedulerName: kai-scheduler
          priorityClassName: interactive-training
          containers:
            - name: ray-head
              image: "rayproject/ray:2.46.0-gpu-py311"
              resources:
                requests:
                  cpu: "8"
                  memory: "32Gi"
                limits:
                  cpu: "16"
                  memory: "64Gi"
          nodeSelector:
            node-role: system    # Head runs on CPU system node
          tolerations:
            - key: CriticalAddonsOnly
              operator: Exists
              effect: NoSchedule

    workerGroupSpecs:
      - groupName: gpu-workers
        replicas: 8              # 8 workers × 1 GPU = 8 GPUs
        minReplicas: 4           # KAI will start if at least 4 GPUs available
        maxReplicas: 16          # Elastic scale up to 16 if GPUs free

        template:
          metadata:
            labels:
              kai.scheduler/queue: ml-team-nlp
            annotations:
              kai.scheduler.nvidia.com/gang-scheduling-group: "llama-finetune-run-001"
          spec:
            schedulerName: kai-scheduler
            priorityClassName: interactive-training

            containers:
              - name: ray-worker
                image: "rayproject/ray:2.46.0-gpu-py311"
                resources:
                  requests:
                    nvidia.com/gpu: "1"
                    cpu: "16"
                    memory: "128Gi"
                  limits:
                    nvidia.com/gpu: "1"
                    memory: "192Gi"

                # Graceful preemption: save checkpoint before eviction
                lifecycle:
                  preStop:
                    exec:
                      command:
                        - /bin/sh
                        - -c
                        - |
                          echo "Saving checkpoint before preemption..."
                          python /app/save_checkpoint.py \
                            --checkpoint-dir /fsx/checkpoints/llama-finetune-run-001 \
                            --step $(cat /tmp/current_step)
                          echo "Checkpoint saved."

                volumeMounts:
                  - name: model-weights
                    mountPath: /mnt/models
                    readOnly: true
                  - name: fsx-training-data
                    mountPath: /fsx

            volumes:
              - name: model-weights
                persistentVolumeClaim:
                  claimName: model-weights-pvc
              - name: fsx-training-data
                persistentVolumeClaim:
                  claimName: hpc-fsx-storage

            nodeSelector:
              workload-tier: inference-training
            tolerations:
              - key: nvidia.com/gpu
                operator: Exists
                effect: NoSchedule

  # The actual training script to run on the cluster
  entrypoint: |
    python /app/finetune_llama.py \
      --model /mnt/models/meta-llama/Meta-Llama-3.1-8B \
      --data /fsx/training/instruction-tuning-dataset \
      --output /fsx/checkpoints/llama-finetune-run-001 \
      --num-epochs 3 \
      --use-deepspeed \
      --resume-from-checkpoint /fsx/checkpoints/llama-finetune-run-001/latest

The minReplicas: 4 setting combined with KAI's gang scheduling means: if 8 GPUs aren't available but 4 are, KAI can start the job with 4 workers and scale up as more GPUs become free. If even 4 aren't available, the job queues.

The preStop lifecycle hook is the preemption safety net. When KAI decides to preempt this job for a higher-priority workload, Kubernetes sends SIGTERM to the pod. The preStop hook runs first, saving a checkpoint to FSx. The job can resume from checkpoint when GPUs become available again.

Production Inference with RayService

The inference service has inference-critical priority and preemptionPolicy: Never. KAI guarantees it never loses its GPUs:

# ray-inference-service.yaml
apiVersion: ray.io/v1
kind: RayService
metadata:
  name: llama-production-serving
  namespace: inference-team
  labels:
    kai.scheduler/queue: inference-team
spec:
  # Autoscaling policy for serving replicas
  serveConfigV2: |
    applications:
      - name: llama-serve
        import_path: serving.llama_serve:build_app
        route_prefix: "/"
        args:
          model_path: /mnt/models/meta-llama/Meta-Llama-3.1-8B-Instruct
          tensor_parallel_size: 2
        deployments:
          - name: LlamaDeployment
            autoscaling_config:
              min_replicas: 2
              max_replicas: 4
              target_ongoing_requests: 32
              upscale_delay_s: 60
              downscale_delay_s: 600    # Slow scale-down — model load time is expensive

  rayClusterConfig:
    rayVersion: "2.46.0"

    headGroupSpec:
      template:
        metadata:
          labels:
            kai.scheduler/queue: inference-team
        spec:
          schedulerName: kai-scheduler
          priorityClassName: inference-critical  # Never preempted
          containers:
            - name: ray-head
              image: "rayproject/ray-llm:2.46.0-py311-cu124"
              resources:
                requests:
                  cpu: "4"
                  memory: "16Gi"
                limits:
                  cpu: "8"
                  memory: "32Gi"
              ports:
                - containerPort: 8000   # Serve endpoint
                  name: serve
                - containerPort: 8265   # Dashboard
                  name: dashboard

    workerGroupSpecs:
      - groupName: inference-workers
        replicas: 2
        minReplicas: 2    # Never scale below 2 — production SLA
        maxReplicas: 4

        template:
          metadata:
            labels:
              kai.scheduler/queue: inference-team
          spec:
            schedulerName: kai-scheduler
            priorityClassName: inference-critical

            containers:
              - name: ray-worker
                image: "rayproject/ray-llm:2.46.0-py311-cu124"
                resources:
                  requests:
                    nvidia.com/gpu: "2"   # 2 GPUs for TP=2 inference
                    cpu: "32"
                    memory: "256Gi"
                  limits:
                    nvidia.com/gpu: "2"
                    memory: "256Gi"

                # No preStop checkpoint hook needed — inference is stateless
                # PodDisruptionBudget handles graceful restarts

                volumeMounts:
                  - name: model-weights
                    mountPath: /mnt/models
                    readOnly: true

            volumes:
              - name: model-weights
                persistentVolumeClaim:
                  claimName: model-weights-pvc

            nodeSelector:
              workload-tier: inference-training
            tolerations:
              - key: nvidia.com/gpu
                operator: Exists
                effect: NoSchedule

---
# PDB: ensure at least 1 inference replica is always running during disruptions
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: llama-inference-pdb
  namespace: inference-team
spec:
  minAvailable: 1
  selector:
    matchLabels:
      ray.io/node-type: worker
      ray.io/cluster: llama-production-serving

---
# NLB + Ingress for external inference access
apiVersion: v1
kind: Service
metadata:
  name: llama-inference-nlb
  namespace: inference-team
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
spec:
  type: LoadBalancer
  selector:
    ray.io/serve: "true"
  ports:
    - name: serve
      port: 443
      targetPort: 8000

Part 6: KEDA Autoscaling — Slurm Queue-Driven Node Scaling

While Karpenter provisions nodes on demand, KEDA connects Slurm's job queue depth to Kubernetes HPA, ensuring slurmd pod replicas scale proportionally to pending workload:

# Install KEDA
helm repo add kedacore https://kedacore.github.io/charts
helm install keda kedacore/keda \
  --namespace keda \
  --create-namespace \
  --version 2.16.0

# keda-slurm-scaler.yaml
---
# External scaler: queries Slurm REST API for pending jobs
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: slurm-gpu-heavy-scaler
  namespace: slurm-system
spec:
  scaleTargetRef:
    name: hpc-cluster-gpu-heavy-nodeset    # Slinky NodeSet resource
    apiVersion: slinky.slurm.net/v1alpha1
    kind: NodeSet

  pollingInterval: 30    # Check Slurm queue every 30 seconds
  cooldownPeriod: 600    # 10 minutes before scaling down (jobs may requeue)
  minReplicaCount: 0     # Scale to zero when queue is empty
  maxReplicaCount: 32    # Max 32 p5 nodes

  triggers:
    # Scale based on pending jobs in gpu-heavy partition
    - type: external
      metadata:
        scalerAddress: "slurm-keda-scaler:9090"   # Custom KEDA external scaler
        partition: gpu-heavy
        # Target: 1 node per 2 queued multi-node jobs
        # (most jobs use 4 nodes; 2 queued = need 8 nodes)
        targetPendingJobs: "2"

---
# For Ray training — scale on KAI queue depth
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: ray-training-gpu-scaler
  namespace: ml-team
spec:
  scaleTargetRef:
    name: llama-finetune-run-001-workers
    apiVersion: ray.io/v1
    kind: RayCluster

  pollingInterval: 15
  cooldownPeriod: 300
  minReplicaCount: 4
  maxReplicaCount: 16

  triggers:
    # Scale on KAI queue pending GPU count
    - type: prometheus
      metadata:
        serverAddress: http://prometheus:9090
        metricName: kai_queue_pending_gpus
        query: |
          kai_queue_pending_gpus{queue="ml-team-nlp"}
        threshold: "4"   # Add 1 worker per 4 pending GPU-requests in queue

Part 7: Unified Observability

With three workload types on one cluster, you need one dashboard that answers: "Where are my GPUs, who has them, and what's waiting?"

NVIDIA DCGM Exporter

GPU Operator installs DCGM Exporter as a DaemonSet on every GPU node. It exports per-GPU metrics regardless of whether the workload is Slurm, Ray, or Kubernetes—the single source of truth for GPU utilization:

# dcgm-servicemonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: dcgm-exporter
  namespace: gpu-operator
spec:
  selector:
    matchLabels:
      app: nvidia-dcgm-exporter
  endpoints:
    - port: metrics
      interval: 15s
      path: /metrics

Key DCGM metrics for the unified dashboard:

# GPU utilization by node (regardless of workload type)
DCGM_FI_DEV_GPU_UTIL{gpu="0", modelName="H100-SXM4-80GB", Hostname="p5-node-01"}

# GPU memory used — critical for Slurm gang scheduler making placement decisions
DCGM_FI_DEV_FB_USED{...}

# NVLink bandwidth — shows whether distributed jobs are using topology correctly
DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL{...}

# Temperature and power — early warning for hardware issues
DCGM_FI_DEV_GPU_TEMP{...}
DCGM_FI_DEV_POWER_USAGE{...}

Slurm Metrics Exporter

Slinky includes a Prometheus exporter that scrapes slurmrestd for job and partition metrics:

# slurm-metrics-servicemonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: slurm-metrics
  namespace: slurm-system
spec:
  selector:
    matchLabels:
      app: slurm-cluster-metrics
  endpoints:
    - port: metrics
      interval: 30s

# Key metrics exposed:
# slurm_partition_cpus_total{partition="gpu-heavy", state="allocated"}
# slurm_partition_gpus_total{partition="gpu-heavy", state="idle"}
# slurm_jobs_total{state="pending", partition="gpu-heavy"}
# slurm_jobs_total{state="running", partition="gpu-heavy"}
# slurm_job_wait_time_seconds{partition="gpu-heavy", account="hpc-team-a"}

Unified Grafana Dashboard

Deploy AWS Managed Grafana and configure four dashboard panels:

┌─────────────────────────────────────────────────────────────────────┐ │ UNIFIED GPU PLATFORM DASHBOARD │ ├─────────────────────────────────────────────────────────────────────┤ │ FLEET OVERVIEW │ │ Total GPUs: 64 Active: 51 Idle: 8 Pending allocation: 5 │ │ ████████████████████████████████████████████████░░░░░░░░░░ │ │ │ ├──────────────────────┬──────────────────────┬───────────────────────┤ │ SLURM PARTITION │ KAI QUEUES │ INFERENCE SLA │ │ │ │ │ │ gpu-heavy │ ml-team-nlp: 2 jobs │ TTFT p95: 380ms ✓ │ │ Running: 32 GPUs │ hpc-team-a: 1 job │ ITL p50: 22ms ✓ │ │ Pending: 2 jobs │ inference: STABLE │ Active conns: 847 │ │ Next avail: 2h14m │ │ GPUs: 4/4 (healthy) │ │ │ Burst available: │ │ │ gpu-medium │ 16 GPUs idle │ Last preemption: │ │ Running: 8 GPUs │ → Available for │ NEVER ✓ │ │ Pending: 0 jobs │ over-quota burst │ │ │ │ │ │ ├──────────────────────┴──────────────────────┴───────────────────────┤ │ GPU UTILIZATION BY NODE (DCGM) │ │ │ │ p5-node-01 ████████████████████████████████████████████████ 96% │ │ p5-node-02 ████████████████████████████████████████████████ 94% │ │ p5-node-03 ████████████████████████████████████████████████ 95% │ │ p5-node-04 ████████████████████████████████████████████████ 93% │ │ g6e-node-01 ██████████████████████████████████░░░░░░░░░░░░░░ 68% │ │ g6e-node-02 ████████████████████████████████████████████████ 97% │ │ g6e-node-03 ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 0% │ │ ↑ Karpenter will reclaim │ ├─────────────────────────────────────────────────────────────────────┤ │ COST ATTRIBUTION (this week) │ │ │ │ hpc-team-a: \(12,480 (42%) ██████████████████████ │ │ ml-team-nlp: \)8,320 (28%) ██████████████ │ │ ml-team-cv: \(4,160 (14%) ███████ │ │ inference: \)4,160 (14%) ███████ │ │ hpc-team-b: $590 (2%) █ │ └─────────────────────────────────────────────────────────────────────┘

Part 8: The Preemption Matrix — How Priority Conflicts Resolve

The most important behavior to understand, test, and communicate to your users is exactly what happens when workloads compete. This table is the document your users need on day one:

Configure Slinky to automatically requeue Slurm jobs on preemption:

# In slurm.conf (via SlurmCluster spec slurmConfig):
PreemptMode=REQUEUE       # Requeue preempted jobs (not cancel)
PreemptType=preempt/qos   # Use QOS-based preemption
PreemptParameters=send_user_signal=SIGTERM,gracetime=120  # 2 min checkpoint window
KillWait=120              # Wait 2 min for job to checkpoint before SIGKILL

Part 9: Real-World Numbers — What the Unified Architecture Delivers

Running a representative mixed workload (4 Slurm MPI jobs, 3 Ray training runs, 1 inference service) across the two architectures:

Cluster composition: 8× p5.48xlarge (64 H100s), us-east-1a

A few notes on these numbers:

• Slurm p50 wait time increases slightly in the unified cluster—this is expected. You're sharing a pool with other workload types, so the median job waits a bit longer. But the p95 improves dramatically because Karpenter provisioning is significantly faster than adding capacity to a traditional Slurm cluster.

• Inference TTFT is essentially unchanged—KAI's inference-critical priority class ensures inference pods are never resource-starved or preempted.

• The monthly cost reduction comes primarily from GPU utilization: 78% vs 41% on the same hardware means you serve the same workloads with ~47% fewer nodes.

The Karpenter consolidation setting (WhenUnderutilized with consolidateAfter: 300s) is the key driver of the idle time improvement. The g6e-node-03 node visible in the Grafana dashboard above at 0% utilization would be terminated within 5 minutes and its workloads bin-packed onto other nodes—a $3.68/hour saving per idle node, automatically.

Part 10: Day-2 Operations

Draining a Node for Maintenance Without Breaking Jobs

# Gracefully drain a GPU node — Slinky marks slurmd as DRAINING,
# Slurm lets current job finish, Kubernetes drains remaining pods

# Step 1: Tell Slinky/Slurm to drain the node (no new jobs)
kubectl exec -n slurm-system deploy/hpc-cluster-controller -- \
  scontrol update NodeName=p5-node-02 State=DRAIN Reason="Hardware maintenance"

# Step 2: Wait for running Slurm jobs to finish (watch with squeue)
kubectl exec -n slurm-system deploy/hpc-cluster-controller -- \
  squeue --nodelist=p5-node-02 --iterate=30

# Step 3: Once Slurm shows node idle, cordon it in Kubernetes
kubectl cordon p5-node-02

# Step 4: Evict non-Slurm pods (KAI respects PDBs)
kubectl drain p5-node-02 \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --grace-period=120

# Step 5: After maintenance, uncordon and re-enable Slurm
kubectl uncordon p5-node-02
kubectl exec -n slurm-system deploy/hpc-cluster-controller -- \
  scontrol update NodeName=p5-node-02 State=RESUME

Updating the Slurm Cluster Version

Slinky handles rolling updates to Slurm components without cluster downtime:

# Update SlurmCluster spec with new image version
kubectl patch slurmcluster hpc-cluster -n slurm-system \
  --type merge \
  --patch '{"spec":{"controller":{"image":"registry.schedmd.com/slurm/slurm:24.11"}}}'

# Slinky operator performs rolling update:
# 1. Drains active slurmd pods
# 2. Updates slurmctld
# 3. Rolls slurmd pods with new image
# 4. Validates cluster health before marking complete
kubectl rollout status deployment/hpc-cluster-controller -n slurm-system

Adding a New GPU Partition for a New Team

# A new team arrives with Trainium2 nodes for training budget optimization
# Add a new partition to the existing SlurmCluster
kubectl patch slurmcluster hpc-cluster -n slurm-system --type merge \
  --patch='
  {
    "spec": {
      "partitions": [{
        "name": "trainium",
        "maxReplicas": 8,
        "defaultReplicas": 0,
        "nodeSelector": {"node.kubernetes.io/instance-type": "trn2.48xlarge"},
        "resources": {
          "requests": {
            "aws.amazon.com/neurondevice": "16",
            "memory": "768Gi",
            "cpu": "192"
          }
        },
        "gres": "neuron:16",
        "features": ["trainium2", "neuron"]
      }]
    }
  }'

# No cluster restart required. Slinky adds the partition to slurmctld config live.
# Users can immediately submit: sbatch --partition=trainium ...

When This Architecture Is and Isn't Right for You

This is not a "always better than separate clusters" answer. The unified architecture adds genuine operational complexity and some performance trade-offs that are worth being honest about.

This architecture is clearly right when:

• You have distinct teams with genuinely incompatible scheduling preferences (HPC + ML + serving)

• Your GPU fleet average utilization is below 55% across all clusters combined

• Your GPU budget is under active cost pressure

• You're willing to invest in Slinky + KAI expertise (it's newer technology with less operational history than separate Slurm + K8s clusters)

• Your inference workloads have well-defined resource requirements (KAI's inference-critical PriorityClass requires you to know in advance how many GPUs to reserve)

Stick with separate clusters when:

• HPC jobs are highly latency-sensitive to queue wait time (unified pool means higher variance in wait times at peak)

• Your compliance posture requires compute isolation between teams (MIG provides hardware isolation but not network isolation)

• Your Slurm workloads use advanced features like MPMD, heterogeneous jobs, or topology-based binding that the Slinky operator doesn't yet fully support

• You're a pure cloud-native shop with no existing Slurm users—Kueue + KubeRay alone is simpler and sufficient

Wrapping Up

Project Slinky is the architectural move that makes the unified GPU platform possible—not because it solves all the scheduling problems, but because it makes Slurm slurmd pods first-class Kubernetes citizens. Once everything is a pod, NVIDIA KAI Scheduler can apply consistent gang scheduling and priority arbitration across all three workload types. GPU Operator handles the hardware lifecycle transparently. Karpenter ensures you're never paying for idle GPU capacity between jobs.aws.amazon+3

The three-team problem at the top of this post doesn't go away—but it transforms. Instead of "three teams, three clusters, three times the GPU waste," it becomes "three teams, one pool, one set of priority policies, one cost allocation report." The HPC engineer keeps sbatch. The ML engineer keeps ray job submit. The platform engineer keeps kubectl apply. None of them see the scheduling arbitration happening underneath—they just see their jobs running faster and their GPU requests queuing fairly against each other.

That's the platform. Now go build it.

What the book actually is

A field guide to the inference stack — from CUDA kernels to Kubernetes autoscaling — that treats model serving as a living system you operate, not an architecture diagram you draw once and hand off.​

A plain account of tradeoffs: when to quantize, cache, speculate, parallelize, or disaggregate — and what each decision costs in latency, throughput, quality, and engineering attention.​

A reminder that the bottleneck is always specific: prefill and decode fail differently, image and LLM inference fail differently, and fixing the wrong one wastes everything.​

How it explains the hard parts in simple words

Prefill and decode are different problems. Prefill is compute-bound — it processes your entire input in parallel and determines time to first token. Decode is memory-bound — it generates one token at a time and determines tokens per second. Optimizing one does nothing for the other.​

The KV cache is your most valuable asset. It stores attention results so decode runs in linear time instead of quadratic. Prefix caching extends that across requests — but only if your novel tokens arrive as late in the sequence as possible, or you break the cache hit at the first token.​

Quantization is a quality contract. Lowering precision from FP16 to INT8 or INT4 frees compute and memory, but you must measure quality impact before shipping it — not after a user complaint.​

Speculative decoding trades latency for throughput. EAGLE drafts tokens using hidden states from the target model. N-gram speculation matches prefixes from the input to predict outputs. Both reduce forward passes, but both require smaller batch sizes — meaning higher cost per request.​

Disaggregation is powerful and expensive. Separating prefill and decode onto independently scaling workers unlocks enormous efficiency gains — but only makes sense past 100M–1B tokens per day. Before that, the complexity costs more than it saves.​

Model parallelism is for when one GPU isn't enough. Tensor parallelism splits the model across GPUs for lower latency. Expert parallelism routes MoE model experts across GPUs for higher throughput. Multi-node inference adds network latency — only worth it when the model doesn't fit any other way.​

Production is a systems problem, not a CUDA problem. Autoscaling, cold starts, routing, observability, zero-downtime deployments, and cost estimation all live above the runtime layer — and they determine whether your optimized model actually serves users reliably.​

The infrastructure stories that teach

The book doesn't hide the real tensions: a decode phase throttled by memory bandwidth while compute sits idle, a KV cache that fills and evicts constantly because prefix ordering was never considered, a speculative decoding setup that improves p50 latency while quietly destroying throughput at scale. Each scenario ends with a concrete decision, a changed configuration, or a new constraint — and those stack into a system that degrades gracefully instead of collapsing suddenly.​

Two ideas that quietly run through the book

The bottleneck always moves. You don't optimize inference once. You find the constraint — compute, memory bandwidth, cache hit rate, cold start time — fix it, and immediately find the next one. The work is iterative and never finished.​

Know your use case before you touch the stack. Online or offline. Consumer or B2B. Latency-sensitive or throughput-sensitive. Every optimization decision flows downstream from those answers. Getting them wrong means tuning the wrong thing confidently.​

What this looks like in practice

• Define your latency budget before touching runtime — end-to-end, not just GPU time, so you know what you're actually optimizing for

• Set prefix caching up early and protect it — keep novel tokens as late in your context as possible, or you pay full prefill cost every time

• Measure quality before quantizing — establish a baseline with evals before lowering precision, so you know what you accepted and why

• Watch batch size when enabling speculation — latency improves, throughput drops, cost rises; make that tradeoff consciously

• Fix health probes and autoscaling before adding GPUs — cold start behavior and concurrency limits determine real availability more than raw compute

• Only disaggregate at scale — separate prefill and decode workers when the traffic justifies the operational complexity, not before​

Who will get the most out of it

Engineers who've deployed models via API and now need to understand what's happening underneath when they own the serving layer.

ML engineers and SREs who want explanations that match production reality — not benchmark presentations from a conference slide.

Builders working with open models on AWS, GCP, or self-hosted clusters who need to close the gap between "it works" and "it works under load."​

What felt different to read

The writing is direct: here's the bottleneck, here's why it exists, here's what you can do about it, here's what that costs. It never pretends you can win on latency, throughput, and cost simultaneously — it shows you how to pick which one matters most for your product right now. It also doesn't romanticize any single tool. vLLM, SGLang, and TensorRT-LLM each get honest assessments of where they fit and where they don't.​

What this book pushed me to change

• Think about prefill and decode separately when something feels slow — they break in completely different ways

• Set KV cache allocation explicitly instead of leaving it at defaults, and understand where overflow goes

• Treat quantization as a quality contract — measure first, ship second

• Watch batch size whenever speculative decoding is enabled, because the throughput cost is real

• Stop reaching for disaggregation until the traffic numbers actually justify it

• Review the constraint weekly, because it moved while I was busy with something else​

Verdict

If you want universal rules and clean architecture blueprints, look elsewhere. If you want a clear, honest view of how inference actually works — the hardware, the software, the techniques, and the production systems — and how real teams make real tradeoffs under pressure, this is worth your full day.

It's free. It treats the inference layer seriously. And the inference layer is where AI products are actually won or lost.

📖 Free PDF at baseten.com/inference-engineering

This post is the end-to-end treatment of multi-account EKS administration: why the problem is hard, what AWS has built, and the actual implementation patterns with real configuration code.

The Problem Space

The issue itself articulated two distinct problem areas that platform teams run into at scale:

Problem 1: Visibility — You're responsible for 40 EKS clusters spread across 12 AWS accounts and 4 regions. Someone asks: "Which clusters are running deprecated Kubernetes versions?" You have no single place to check. You're either opening 12 browser tabs, writing ad-hoc CLI scripts, or paying for a third-party tool.

Problem 2: Lifecycle Management — A new product team needs a cluster in their isolated AWS account. Today: someone writes Terraform, manually executes a CI pipeline, manually configures IAM, manually installs add-ons, manually configures kubectl access. Each cluster is a snowflake because each engineer did it slightly differently. There's no standardized golden path

These aren't hypothetical problems. AWS encourages multi-account architectures precisely because they reduce blast radius, isolate permissions, and help organizations enforce compliance boundaries. But the more accounts you have, the harder operational control becomes. The issue originally asked: can we have our account isolation and operability too?

The answer, it turns out, is yes—but it requires composing multiple AWS services together deliberately.

The Two Architecture Models

Before getting into tooling, understand the two fundamental cluster models. Your choice between them drives everything else.​

Centralized Cluster Model

All EKS clusters live in a single Cluster Account. Dev teams in their own AWS accounts access the Kubernetes API via cross-account IAM role assumption.

┌──────────────────────────────────────────┐ │ Cluster Account │ │ │ │ ┌──────────────────────────────────┐ │ │ │ EKS Cluster │ │ │ │ - VPC, subnets, worker nodes │ │ │ │ - Namespaces per team │ │ │ └────────────────┬─────────────────┘ │ │ │ EKS Access Entries │ │ │ │ └───────────────────┼──────────────────────┘ │ Cross-account IAM role assumption │ ┌───────────┼───────────────┐ │ │ │ ┌───────▼────┐ ┌────▼───────┐ ┌────▼───────┐ │ Team A │ │ Team B │ │ Team C │ │ Account │ │ Account │ │ Account │ │ │ │ │ │ │ │ IAM Role │ │ IAM Role │ │ IAM Role │ └────────────┘ └────────────┘ └────────────┘

Works best when:

• Platform team owns all clusters

• Teams need Kubernetes isolation (namespaces) but not AWS account isolation

• Simpler networking (single VPC per cluster)

• Fewer clusters to manage

Pain points:

• RBAC complexity scales with team count

• Teams need VPN/PrivateLink to reach cluster API endpoint

• Blast radius: a misconfigured workload in Team A's namespace can affect Team B's

• Difficult compliance enforcement (data sovereignty, regulatory separation)

Decentralized Cluster Model

Each team or product domain gets their own EKS cluster in their own AWS account.

┌────────────────────┐ │ Management Account│ │ (Control Plane) │ │ - Visibility │ │ - Policy │ │ - Lifecycle Mgmt │ └──────────┬─────────┘ │ AWS Organizations ┌─────┼──────┐ │ │ │ ┌────▼───┐ │ ┌────▼────┐ │ Team A │ │ │ Team B │ │ Account│ │ │ Account │ │ EKS │ │ │ EKS │ │ Cluster│ │ │ Cluster │ └────────┘ │ └─────────┘ ┌─▼──────┐ │ Team C │ │ Account│ │ EKS │ │ Cluster│ └────────┘

Works best when:

• Strong tenant isolation required (compliance, data residency)

• Teams own full stack including infrastructure

• Need per-account cost attribution

• Regulated industries (financial services, healthcare)

Pain points:

• Routing must be established between cluster VPCs (Transit Gateway, VPC peering)

• Operational overhead multiplies with cluster count

• Add-on management, version upgrades, and compliance become hard to coordinate

• This is exactly the problem GitHub issue #1708 was created to solve

Most large organizations run a hybrid: centralized clusters for shared services (monitoring, CI/CD, service mesh control plane), decentralized clusters for product domains.

Part 1: Visibility — EKS Dashboard

AWS shipped EKS Dashboard on May 21, 2025. It's the answer to the visibility problem: a native, zero-cost dashboard inside the EKS console that aggregates cluster data across all accounts in your AWS Organization and all AWS regions.

What It Shows

EKS Dashboard provides three layers of insight:

1. Cluster inventory:

• All EKS clusters across all member accounts and regions in one view

• Version distribution (how many clusters on 1.29, 1.30, 1.31, 1.32?)

• Support status (Standard, Extended Support, End of Life)

• Forecasted Extended Support cost: Before a cluster version falls off standard support, you can see the dollar impact of not upgrading

2. Managed node groups:

• Instance type distribution across your fleet

• Launch template versions

• AMI versions (how many node groups are behind on AMI updates?)

• Node group health status

3. EKS add-ons:

• Add-on versions across all clusters

• Add-on health status

• Which clusters are running outdated add-on versions

It also includes a global map view showing cluster geographic distribution, and can show connected clusters (EKS Anywhere, EKS Hybrid, or even non-EKS Kubernetes clusters registered via EKS Connector).

Enabling EKS Dashboard

The setup is two steps and takes under 5 minutes:aws.amazon+1

Step 1: Enable Trusted Access via AWS Organizations

This allows EKS to read data from member accounts using the AWSServiceRoleForAmazonEKSDashboard service-linked role:

# From the management or delegated administrator account
aws organizations enable-aws-service-access \
  --service-principal eks-dashboard.amazonaws.com

# Verify trusted access is enabled
aws organizations list-aws-service-access-for-organization \
  --query 'EnabledServicePrincipals[?ServicePrincipal=eks-dashboard.amazonaws.com]'

Or via CloudFormation (recommended for auditability):

# enable-eks-dashboard.yaml
AWSTemplateFormatVersion: '2010-09-09'
Description: Enable EKS Dashboard trusted access via AWS Organizations

Resources:
  EKSDashboardTrustedAccess:
    Type: AWS::Organizations::ServicePrincipal
    Properties:
      ServicePrincipal: eks-dashboard.amazonaws.com

  # Optional: Designate a delegated administrator
  # (so you don't need management account login for dashboard)
  EKSDashboardDelegatedAdmin:
    Type: AWS::Organizations::DelegatedAdministrator
    Properties:
      AccountId: !Sub "{{resolve:ssm:/org/platform-account-id}}"
      ServicePrincipal: eks-dashboard.amazonaws.com

Step 2: Access the Dashboard

Navigate to the EKS console in the US East (N. Virginia) region from either:

• The AWS Organizations management account

• A delegated administrator account (recommended: your platform team account)

Click Dashboard in the left navigation. You'll immediately see all clusters across all accounts your organization manages.

Delegated Administrator Pattern (recommended): Rather than having everyone log into the management account (which should have very restricted access), designate your platform/ops account as a delegated administrator for EKS Dashboard. Platform engineers can then access the dashboard from the familiar platform account without touching the management account.

Practical Dashboard Workflows

The dashboard transforms operational questions that previously required scripting into single-click answers:

"Which clusters are at risk?"

1. Filter by Version Support Status = End of Life

2. See all clusters that need immediate attention

3. Export to CSV for stakeholder reporting

"What will Extended Support cost us if we don't upgrade?"

• Dashboard shows forecasted Extended Support charges per cluster

• Gives platform team hard dollar numbers to justify upgrade prioritization

"Which node groups are on outdated AMIs?"

• Filter Managed Node Groups by AMI version

• See exactly which accounts/clusters need AMI refresh before next security scan

"What add-on versions are we running fleet-wide?"

• The add-ons view aggregates versions across all clusters

• Filter by add-on name, compare against current recommended version

• Identify clusters that need aws-ebs-csi-driver or vpc-cni updated

Dashboard Limitations

EKS Dashboard solves visibility, not action. You can see that 12 clusters need upgrading, but you still need to go execute those upgrades cluster by cluster. That's what lifecycle management (Part 2) addresses.​

The dashboard is also currently read-only and based in us-east-1 (aggregates data from all regions, but the console is only in N. Virginia).

Part 2: Lifecycle Management — ACK + kro + Argo CD

For lifecycle management, AWS's recommended pattern as of February 2026 is the combination of:

• ACK (AWS Controllers for Kubernetes): Manage AWS resources (VPCs, IAM roles, EKS clusters) as Kubernetes CRDs

• kro (Kube Resource Orchestrator): Compose ACK resources into higher-level abstractions, handle dependency ordering

• Argo CD: GitOps controller for applying cluster manifests and bootstrapping add-ons

This is a hub-and-spoke model: a management cluster in a central account acts as the control plane. It runs ACK controllers and Argo CD. When you want a new workload cluster in a team's account, you submit a YAML file to a Git repository—the system handles the rest.

Why Not Just Use Terraform?

A legitimate question. Terraform handles this well for infrastructure. The problem is the seam between infrastructure and Kubernetes:

• Terraform creates the EKS cluster

• Then you need a separate process to install add-ons (Helm charts)

• Then another process to configure RBAC, access entries, namespaces

• Then another to deploy application workloads

You end up with pipelines calling Terraform calling kubectl calling Helm, with state scattered across Terraform backends and Helm release records.

The ACK+kro+ArgoCD pattern unifies this under the Kubernetes API. Everything—VPC, IAM roles, EKS cluster, add-ons, application workloads—is expressed as Kubernetes manifests and managed through a single GitOps pipeline.

Important caveat: kro is still in active development and not yet recommended for production use cases where API stability is required. Use it in non-critical environments while it stabilizes. ACK itself is production-ready.

Architecture Deep Dive

┌──────────────────────────────────────────────────────────────┐ │ Management Account │ │ │ │ ┌────────────────────────────────────────────────────────┐ │ │ │ Management Cluster (EKS) │ │ │ │ │ │ │ │ ┌─────────────────┐ ┌────────────────────────────┐ │ │ │ │ │ ACK Controllers│ │ Argo CD │ │ │ │ │ │ - eks │ │ ┌──────────────────────┐ │ │ │ │ │ │ - ec2 │ │ │ ApplicationSets │ │ │ │ │ │ │ - iam │ │ │ (add-ons per cluster)│ │ │ │ │ │ └────────┬─────────┘ │ └──────────┬───────────┘ │ │ │ │ │ │ └────────────┼───────────────┘ │ │ │ │ ┌────────▼────────────────────────── │ │ │ │ │ │ kro Controller │ │ │ │ │ │ ResourceGraphDefinitions │ │ │ │ │ │ - EksCluster (high-level CRD) │ │ │ │ │ │ - Vpc │ │ │ │ │ └─────────────────────────────────────│─────────────────┘ │ │ │ │ │ │ └────────────────────────────────────────┼───────────────────┘ │ │ │ │ └───────────────────────────────────────────┼───────────────────┘ Git Repository ──────────────────►│ │ IAM Role Assumption (cross-account) │ │ │ add-on sync ┌─────▼─────────────────────────▼────────┐ │ Workload Account │ │ │ │ ┌──────────────────────────────────┐ │ │ │ Workload EKS Cluster │ │ │ │ (provisioned by ACK) │ │ │ │ (add-ons by Argo CD) │ │ │ └──────────────────────────────────┘ │ │ │ └─────────────────────────────────────────┘

When a developer raises a PR with a new EksCluster manifest:

1. Argo CD detects the PR merge and applies the EksCluster CR to the management cluster

2. kro decomposes the EksCluster into individual ACK CRs (VPC, Subnets, IAM roles, EKS cluster)—in dependency order

3. ACK controllers assume a role in the target workload account and create the AWS resources via API

4. Once the cluster exists, Argo CD registers it as a remote target

5. ApplicationSets in Argo CD automatically generate Applications to install add-ons on the new cluster

The developer submitted one YAML file. The entire cluster, networking, IAM, and add-on bootstrap happened automatically.

Setting Up ACK with Cross-Account IAM

Step 1: Install EKS Capabilities on management cluster

EKS Capabilities provides managed ACK controllers, kro, and Argo CD directly via EKS—no manual Helm chart management:

# Enable EKS Capabilities (ACK + kro + Argo CD as managed add-ons)
eksctl create addon --name ack-controller-eks \
  --cluster management-cluster \
  --service-account-role-arn arn:aws:iam::111122223333:role/ack-eks-controller

eksctl create addon --name ack-controller-ec2 \
  --cluster management-cluster \
  --service-account-role-arn arn:aws:iam::111122223333:role/ack-ec2-controller

eksctl create addon --name ack-controller-iam \
  --cluster management-cluster \
  --service-account-role-arn arn:aws:iam::111122223333:role/ack-iam-controller

eksctl create addon --name kro \
  --cluster management-cluster

eksctl create addon --name argocd \
  --cluster management-cluster

Or via Terraform:

# ack-addons.tf
resource "aws_eks_addon" "ack_eks" {
  cluster_name             = module.eks.cluster_name
  addon_name               = "ack-eks-controller"
  service_account_role_arn = aws_iam_role.ack_eks.arn
}

resource "aws_eks_addon" "ack_ec2" {
  cluster_name             = module.eks.cluster_name
  addon_name               = "ack-ec2-controller"
  service_account_role_arn = aws_iam_role.ack_ec2.arn
}

resource "aws_eks_addon" "ack_iam" {
  cluster_name             = module.eks.cluster_name
  addon_name               = "ack-iam-controller"
  service_account_role_arn = aws_iam_role.ack_iam.arn
}

resource "aws_eks_addon" "kro" {
  cluster_name = module.eks.cluster_name
  addon_name   = "kro"
}

resource "aws_eks_addon" "argocd" {
  cluster_name = module.eks.cluster_name
  addon_name   = "argocd"
}

Step 2: IAM roles for ACK cross-account access

In the management account, the ACK controller IAM role needs sts:AssumeRole permissions:

# Management account: ACK controller IAM role
resource "aws_iam_role" "ack_eks_controller" {
  name = "ack-eks-controller"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "eks.amazonaws.com" }
      Action    = "sts:AssumeRole"
      Condition = {
        StringEquals = {
          "aws:SourceAccount" = data.aws_caller_identity.current.account_id
        }
      }
    }]
  })
}

# Policy to assume roles in workload accounts
resource "aws_iam_role_policy" "ack_cross_account" {
  name = "ack-cross-account-assume"
  role = aws_iam_role.ack_eks_controller.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect   = "Allow"
      Action   = "sts:AssumeRole"
      Resource = "arn:aws:iam::*:role/ack"  # The ack role in all workload accounts
    }]
  })
}

In each workload account, create the IAM role that ACK will assume:

# Workload account: role that ACK assumes (deploy via StackSets to all member accounts)
resource "aws_iam_role" "ack_workload_role" {
  name = "ack"  # Must match the resource in management account policy above

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = {
        AWS = "arn:aws:iam::${var.management_account_id}:role/ack-eks-controller"
      }
      Action = "sts:AssumeRole"
    }]
  })
}

# Policies for what ACK can create in workload accounts
resource "aws_iam_role_policy_attachment" "ack_eks" {
  role       = aws_iam_role.ack_workload_role.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
}

resource "aws_iam_role_policy_attachment" "ack_ec2" {
  role       = aws_iam_role.ack_workload_role.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2FullAccess"
}

resource "aws_iam_role_policy_attachment" "ack_iam" {
  role       = aws_iam_role.ack_workload_role.name
  policy_arn = "arn:aws:iam::aws:policy/IAMFullAccess"
}

Deploy the workload account IAM role using AWS CloudFormation StackSets (apply to entire OU):

aws cloudformation create-stack-set \
  --stack-set-name ack-workload-role \
  --template-url s3://my-templates/ack-workload-role.yaml \
  --parameters ParameterKey=ManagementAccountId,ParameterValue=111122223333 \
  --permission-model SERVICE_MANAGED \
  --auto-deployment "Enabled=true,RetainStacksOnAccountRemoval=false"

# Deploy to all accounts in your Workload OU
aws cloudformation create-stack-instances \
  --stack-set-name ack-workload-role \
  --deployment-targets "OrganizationalUnitIds=ou-xxxx-yyyyyy" \
  --regions us-east-1

Step 3: Configure IAMRoleSelector in management cluster

This maps namespaces to workload-account IAM roles:

# iamroleselector-teamA.yaml
apiVersion: services.k8s.aws/v1alpha1
kind: IAMRoleSelector
metadata:
  name: team-a-account-config
spec:
  # IAM role in Team A's workload account
  arn: arn:aws:iam::222233334444:role/ack
  # All ACK CRs in this namespace will use this role
  namespaceSelector:
    names:
    - cluster-team-a
---
# iamroleselector-teamB.yaml
apiVersion: services.k8s.aws/v1alpha1
kind: IAMRoleSelector
metadata:
  name: team-b-account-config
spec:
  arn: arn:aws:iam::444455556666:role/ack
  namespaceSelector:
    names:
    - cluster-team-b

With this in place, any ACK resource created in the cluster-team-a namespace will automatically be provisioned in Team A's AWS account (222233334444). No credentials in manifests, no manual role switching—just namespaced intent.

Defining the kro ResourceGraphDefinition

This is the key abstraction layer. Instead of submitting 15 separate ACK CRs for every new cluster, you submit one EksCluster CR that kro decomposes automatically:

# rgd-eks-cluster.yaml
apiVersion: kro.run/v1alpha1
kind: ResourceGraphDefinition
metadata:
  name: ekscluster.kro.run
spec:
  schema:
    apiVersion: v1alpha1
    kind: EksCluster
    spec:
      name: string
      region: string | default="us-east-1"
      k8sVersion: string | default="1.32"
      managementAccountId: string  # For Argo CD access entry
      vpc:
        create: boolean | default=true
        vpcId: string | default=""     # Only if vpc.create=false
        privateSubnet1Id: string | default=""
        privateSubnet2Id: string | default=""
      nodeGroup:
        instanceType: string | default="m5.xlarge"
        minSize: integer | default=2
        maxSize: integer | default=10
      addons:
        ebsCsiDriver: boolean | default=true
        efsCsiDriver: boolean | default=false
        karpenter: boolean | default=false

  resources:
  # 1. Cluster IAM Role
  - id: clusterRole
    template:
      apiVersion: iam.services.k8s.aws/v1alpha1
      kind: Role
      metadata:
        namespace: "${schema.spec.name}"
        name: "${schema.spec.name}-cluster-role"
        annotations:
          services.k8s.aws/region: "${schema.spec.region}"
      spec:
        name: "${schema.spec.name}-cluster-role"
        assumeRolePolicyDocument: |
          {
            "Version": "2012-10-17",
            "Statement": [{
              "Effect": "Allow",
              "Principal": {"Service": "eks.amazonaws.com"},
              "Action": "sts:AssumeRole"
            }]
          }
        policies:
        - arn:aws:iam::aws:policy/AmazonEKSClusterPolicy

  # 2. Node IAM Role
  - id: nodeRole
    template:
      apiVersion: iam.services.k8s.aws/v1alpha1
      kind: Role
      metadata:
        namespace: "${schema.spec.name}"
        name: "${schema.spec.name}-node-role"
        annotations:
          services.k8s.aws/region: "${schema.spec.region}"
      spec:
        name: "${schema.spec.name}-node-role"
        assumeRolePolicyDocument: |
          {
            "Version": "2012-10-17",
            "Statement": [{
              "Effect": "Allow",
              "Principal": {"Service": "ec2.amazonaws.com"},
              "Action": "sts:AssumeRole"
            }]
          }
        policies:
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
        - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
        - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly

  # 3. VPC (conditional - only if vpc.create=true)
  - id: vpc
    includeWhen:
    - "${schema.spec.vpc.create}"
    template:
      apiVersion: ec2.services.k8s.aws/v1alpha1
      kind: VPC
      metadata:
        namespace: "${schema.spec.name}"
        name: "${schema.spec.name}-vpc"
        annotations:
          services.k8s.aws/region: "${schema.spec.region}"
      spec:
        cidrBlock: "10.0.0.0/16"
        enableDNSHostnames: true
        enableDNSSupport: true
        tags:
        - key: Name
          value: "${schema.spec.name}-vpc"

  # 4. EKS Cluster (depends on IAM roles - kro handles ordering via CEL references)
  - id: ekscluster
    template:
      apiVersion: eks.services.k8s.aws/v1alpha1
      kind: Cluster
      metadata:
        namespace: "${schema.spec.name}"
        name: "${schema.spec.name}"
        annotations:
          services.k8s.aws/region: "${schema.spec.region}"
      spec:
        name: "${schema.spec.name}"
        # CEL expression - kro resolves this after clusterRole is created
        roleARN: "${clusterRole.status.ackResourceMetadata.arn}"
        version: "${schema.spec.k8sVersion}"
        accessConfig:
          authenticationMode: "API_AND_CONFIG_MAP"
        resourcesVPCConfig:
          # CEL expressions extract generated field (VPC ID) from vpc CR
          subnetIDs:
          - "${vpc.status.subnets[0]}"
          - "${vpc.status.subnets[1]}"
          endpointPublicAccess: false
          endpointPrivateAccess: true
        computeConfig:
          enabled: true
          nodeRoleARN: "${nodeRole.status.ackResourceMetadata.arn}"

  # 5. Access entry for Argo CD (management cluster)
  - id: accessEntry
    template:
      apiVersion: eks.services.k8s.aws/v1alpha1
      kind: AccessEntry
      metadata:
        namespace: "${schema.spec.name}"
        name: "${schema.spec.name}-argocd-access"
        annotations:
          services.k8s.aws/region: "${schema.spec.region}"
      spec:
        clusterName: "${schema.spec.name}"
        # Allow Argo CD in management cluster to sync into workload cluster
        principalARN: "arn:aws:iam::${schema.spec.managementAccountId}:role/hub-cluster-argocd-controller"
        type: STANDARD
        accessPolicies:
        - accessScope:
            type: "cluster"
          policyARN: "arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy"

  # 6. Argo CD Secret (registers workload cluster with Argo CD)
  - id: argocdSecret
    template:
      apiVersion: v1
      kind: Secret
      metadata:
        name: "${schema.spec.name}"
        namespace: argocd
        labels:
          argocd.argoproj.io/secret-type: cluster
        annotations:
          managed-by: kro
      type: Opaque
      stringData:
        name: "${schema.spec.name}"
        # ARN-based server URL (EKS Capabilities feature)
        server: "${ekscluster.status.ackResourceMetadata.arn}"
        project: "default"
        config: |
          {
            "awsAuthConfig": {
              "clusterName": "${schema.spec.name}",
              "roleARN": "arn:aws:iam::${schema.spec.managementAccountId}:role/hub-cluster-argocd-controller"
            }
          }

Apply the ResourceGraphDefinition once to the management cluster. Now anyone with access can provision a new cluster.

Creating a New Workload Cluster

With the RGD in place, provisioning a cluster in Team A's account is a single YAML file submitted via PR:

# clusters/team-a/production.yaml
apiVersion: kro.run/v1alpha1
kind: EksCluster
metadata:
  name: team-a-production
  namespace: cluster-team-a  # Maps to IAMRoleSelector → Team A's account
spec:
  name: team-a-production
  region: us-east-1
  k8sVersion: "1.32"
  managementAccountId: "111122223333"
  vpc:
    create: true
  nodeGroup:
    instanceType: m5.xlarge
    minSize: 2
    maxSize: 20
  addons:
    ebsCsiDriver: true
    efsCsiDriver: false
    karpenter: true

PR merged → Argo CD applies this CR → kro decomposes it → ACK creates AWS resources in Team A's account → Argo CD bootstraps add-ons → Cluster is live.

Bootstrapping Add-ons with Argo CD ApplicationSets

Once Argo CD knows about the workload cluster (via the Secret created by kro), ApplicationSets with the Cluster Generator auto-install add-ons:

# applicationset-addons.yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: eks-addons-aws-ebs-csi
  namespace: argocd
spec:
  generators:
  - clusters:
      selector:
        matchLabels:
          # Only clusters that have EBS CSI enabled
          addon/ebs-csi-driver: "true"

  template:
    metadata:
      name: "{{name}}-ebs-csi-driver"
    spec:
      project: default
      source:
        repoURL: https://kubernetes-sigs.github.io/aws-ebs-csi-driver
        chart: aws-ebs-csi-driver
        targetRevision: 2.26.0
        helm:
          values: |
            controller:
              serviceAccount:
                annotations:
                  eks.amazonaws.com/role-arn: {{metadata.annotations.ebs-csi-role-arn}}
      destination:
        server: "{{server}}"  # Populated from cluster Secret
        namespace: kube-system
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
---
# Karpenter ApplicationSet
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: eks-addons-karpenter
  namespace: argocd
spec:
  generators:
  - clusters:
      selector:
        matchLabels:
          addon/karpenter: "true"

  template:
    metadata:
      name: "{{name}}-karpenter"
    spec:
      project: default
      source:
        repoURL: public.ecr.aws/karpenter
        chart: karpenter
        targetRevision: 1.1.0
        helm:
          values: |
            settings:
              clusterName: "{{metadata.annotations.cluster-name}}"
              clusterEndpoint: "{{metadata.annotations.cluster-endpoint}}"
            serviceAccount:
              annotations:
                eks.amazonaws.com/role-arn: "{{metadata.annotations.karpenter-role-arn}}"
      destination:
        server: "{{server}}"
        namespace: kube-system
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Any new cluster with the addon/karpenter: "true" label on its Argo CD Secret automatically gets Karpenter installed. No manual steps.

Cross-Account kubectl Access Without Static Credentials

Once clusters are provisioned, engineers need kubectl access. The right pattern here combines EKS access entries with IAM role assumption—no static credentials anywhere:

# In each workload account: developer access role
resource "aws_iam_role" "eks_developer_access" {
  name = "eks-developer-access"

  # Trust policy: any principal from management account with developer tag can assume
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = {
        AWS = "arn:aws:iam::${var.management_account_id}:root"
      }
      Action = "sts:AssumeRole"
      Condition = {
        StringEquals = {
          "aws:PrincipalTag/eks-role" = "developer"
        }
      }
    }]
  })
}

Add access entry to the workload cluster (via ACK, so it's in Git):

# Access entry for developer role (include in cluster RGD or separate ApplicationSet)
apiVersion: eks.services.k8s.aws/v1alpha1
kind: AccessEntry
metadata:
  namespace: cluster-team-a
  name: team-a-developer-access
  annotations:
    services.k8s.aws/region: us-east-1
spec:
  clusterName: team-a-production
  principalARN: arn:aws:iam::222233334444:role/eks-developer-access
  type: STANDARD
  accessPolicies:
  - accessScope:
      type: namespace
      namespaces:
      - team-a-app
    policyARN: arn:aws:eks::aws:cluster-access-policy/AmazonEKSEditPolicy

Configure kubectl:

# Developer's local setup (management account SSO session)
# Step 1: Assume role in workload account
aws sts assume-role \
  --role-arn arn:aws:iam::222233334444:role/eks-developer-access \
  --role-session-name dev-session \
  --profile management-account

# Step 2: Update kubeconfig using assumed role
aws eks update-kubeconfig \
  --name team-a-production \
  --region us-east-1 \
  --role-arn arn:aws:iam::222233334444:role/eks-developer-access \
  --profile management-account

# Now kubectl works with Team A's cluster
kubectl get pods -n team-a-app

For daily use, configure this in ~/.aws/config for transparent role chaining:

# ~/.aws/config
[profile team-a-dev]
role_arn = arn:aws:iam::222233334444:role/eks-developer-access
source_profile = management-sso
role_session_name = dev-session

[profile team-b-dev]
role_arn = arn:aws:iam::444455556666:role/eks-developer-access
source_profile = management-sso
role_session_name = dev-session

# Switch clusters cleanly
AWS_PROFILE=team-a-dev kubectl get pods -n team-a-app
AWS_PROFILE=team-b-dev kubectl get pods -n team-b-app

The End-to-End Platform Workflow

Putting this all together, here's the complete lifecycle of a cluster in your multi-account platform:

Day 0: Platform setup (once)

1. Enable EKS Dashboard trusted access in AWS Organizations

2. Deploy management cluster

3. Install ACK controllers, kro, Argo CD via EKS Capabilities

4. Deploy workload account IAM role via CloudFormation StackSets

5. Define EksCluster ResourceGraphDefinition in kro

6. Configure Argo CD ApplicationSets for each add-on

Day 1: New team onboarding

1. Create AWS account for new team (Control Tower or Account Factory)

2. CloudFormation StackSets automatically deploys ack IAM role in new account

3. Developer submits PR: one EksCluster YAML file

4. PR merged → cluster provisioned in team's account → add-ons installed

5. IAM access entry + developer role = kubectl access without manual setup

Day N: Ongoing operations

• EKS Dashboard shows fleet-wide version status → platform team prioritizes upgrades

• Version upgrade: change k8sVersion in cluster YAML → PR → automated update

• New add-on rollout: update ApplicationSet → propagates to all clusters with label

• Cost allocation: each cluster in isolated account → AWS Cost Explorer per team

Incident response:

• EKS Dashboard identifies affected cluster accounts from central view

• kubectl access via pre-configured role assumption (no emergency credential creation)

• Audit trail: all cluster changes are Git commits with author, timestamp, reviewer

GitOps for Multi-Account Configuration Management

When you have 40 clusters across 12 accounts, configuration drift is inevitable without discipline. The GitOps pattern solves this:

git-ops-repo/
├── clusters/
│   ├── team-a/
│   │   ├── production.yaml     # EksCluster CR
│   │   └── staging.yaml
│   ├── team-b/
│   │   └── production.yaml
│   └── platform/
│       └── shared-services.yaml
├── add-ons/
│   ├── karpenter/
│   │   └── applicationset.yaml
│   └── aws-load-balancer-controller/
│       └── applicationset.yaml
└── policies/
    ├── network-policies/
    │   └── applicationset.yaml     # Applied to all clusters
    └── pod-security/
        └── applicationset.yaml

Every change to any cluster across any account is a PR. Every PR has a reviewer. Every merge has a timestamp. When your security team asks "when was network policy X applied to Team B's cluster?"—it's a git log command.

Best Practices and Operational Recommendations

1. Use StackSets for account bootstrapping, not manual IAM setup

The ack IAM role and any other cross-account prerequisites should be deployed via StackSets targeted at your Workload OU. New accounts created via Control Tower/Account Factory automatically get these resources.

2. Use access entries, not aws-auth ConfigMap

Access entries are managed via EKS API (and therefore via ACK, in Git). The aws-auth ConfigMap is a manual file that can lock you out if misconfigured. For all new clusters, use authenticationMode: API.

3. Tag clusters consistently from day one

EKS Dashboard filtering is only as good as your tags. Establish a tagging standard before you have 40 clusters:

metadata:
  labels:
    team: "team-a"
    environment: "production"
    cost-center: "eng-platform"
    tier: "critical"
    addon/ebs-csi-driver: "true"
    addon/karpenter: "true"

4. Don't use kro for production clusters yet

kro's APIs are not stable as of early 2026. Use it for development/staging environments where you can tolerate API changes. For production clusters, use ACK directly with explicit dependency ordering or continue with Terraform for the infrastructure layer.​

5. Test restore of cluster configurations quarterly

Multi-account architecture means more blast radius for misconfiguration. Test restoring cluster state from Git manifests in DR environments. Combine with AWS Backup (EKS integration) for persistent volume coverage.

Wrapping Up

The three years between GitHub issue #1708 opening and today represent a genuine maturation of AWS's multi-account EKS story. The two problem areas that issue identified—visibility and lifecycle management—now have production-ready answers.kubernetes+2

EKS Dashboard solves visibility cleanly: five minutes of setup, no extra cost, immediate fleet-wide insight into version compliance, add-on health, and upgrade cost forecasting. If you're running more than three EKS clusters across multiple accounts, enable it today—there's no reason not to.aws.amazon+1

ACK + kro + Argo CD solves lifecycle management with a powerful but more operationally complex pattern. The value—cluster provisioning as a YAML PR, automatic add-on bootstrapping, GitOps audit trail—is substantial. The investment in understanding ACK cross-account IAM, kro ResourceGraphDefinitions, and Argo CD ApplicationSets is real. Start with the pattern in non-production accounts, stabilize your RGDs, then graduate to production.

For most platform teams, the journey looks like: Enable EKS Dashboard immediately → Standardize cross-account kubectl access via access entries → Implement ACK for cluster provisioning → Adopt kro once it reaches API stability → End up with a hub-and-spoke platform where new teams go from "we need a cluster" to "cluster is live with all add-ons" in the time it takes to review a PR.

That's the platform the AWS containers roadmap team was building toward when they opened that issue in 2022. It's largely here now.

Your clean "lift everything to AWS" plan now has asterisks. You need to run workloads on-premises and in the cloud, manage them consistently, and ensure that a regional cloud outage doesn't take down your entire application.

This is the problem Amazon ECS Anywhere with cross-region load balancing solves. Not in a hand-wavy "hybrid cloud" marketing sense, but concretely: the same ECS API, the same task definitions, the same IAM roles, the same deployment pipelines—whether your container is running on an EC2 instance in us-east-1 or on a bare metal server in your Frankfurt data center. And when you front the whole thing with AWS Global Accelerator, you get intelligent, health-aware traffic distribution that automatically routes users away from failures, whether those failures are in AWS or on-premises.

This post is an end-to-end implementation guide. We'll build the full architecture: on-premises servers registered as ECS external instances, tasks deployed via the EXTERNAL launch type, Traefik running as a sidecar load balancer for inbound traffic (working around ECS Anywhere's native load balancing limitation), Network Load Balancers in AWS regions, and Global Accelerator tying it all together.

Understanding the Problem Space

Before we touch a terminal, let's understand exactly why this architecture exists and what limitations it's working around.

What ECS Anywhere Does

ECS Anywhere extends the ECS control plane beyond AWS infrastructure. Every ECS cluster has two types of capacity:

• EC2/Fargate capacity: Instances AWS manages or serverless compute

• External instances: Your own servers, VMs, or bare metal machines registered to the cluster via AWS Systems Manager

When you register an external instance, two agents get installed on it:oneuptime+1

• SSM Agent: Handles authentication to AWS APIs. Maintains IAM credentials rotated every 30 minutes using a hardware fingerprint—even if connectivity is lost temporarily, credentials are refreshed after reconnection

• ECS Agent: Manages container lifecycle, reports task status back to the ECS control plane, pulls from ECR (or any accessible registry)

From ECS's perspective, an external instance in your Frankfurt data center is just another container instance in the cluster. You deploy tasks to it using --launch-type EXTERNAL. The control plane doesn't care where the instance physically is.

The Critical Limitation You Must Understand

ECS Anywhere has one significant networking constraint that shapes this entire architecture: Elastic Load Balancing is not supported for external instances.

This means you cannot attach an AWS-managed ALB or NLB directly to tasks running on external instances. The awsvpc network mode is also unsupported—external instances must use bridge, host, or none network modes.

This isn't a bug—it's a fundamental constraint. AWS load balancers are VPC-native constructs. They register targets by IP or instance ID within a VPC. Your on-premises server has a private RFC1918 address in your own network, not a VPC. There's no way for an AWS load balancer to register it as a target without it being VPC-connected.

This is why the architecture uses Traefik as an on-premises load balancer—a container running on an external instance that load balances traffic across other containers on the same or nearby external instances, using the ECS API to discover task endpoints dynamically.

The Cross-Region Load Balancing Layer

The on-premises Traefik proxy handles inbound distribution within a location. The cross-region layer handles traffic distribution between locations (on-premises, us-east-1, eu-west-1).

AWS Global Accelerator provides two static Anycast IP addresses that serve as the global entry point for your application. Traffic arrives at the nearest AWS edge location and is forwarded over the AWS global backbone to the healthiest, closest endpoint—whether that's an NLB in us-east-1 or your on-premises servers behind a Network Load Balancer in a different region.aws.amazon+1

The full traffic path for a user in Germany accessing your application:

User in Frankfurt
    → AWS Edge (Frankfurt PoP)        [Anycast routing]
    → AWS Global Accelerator          [Health check routing]
    → NLB in eu-west-1  OR            [If AWS tasks healthy]
    → On-premises Frankfurt servers   [If direct routing configured]
        → Traefik proxy               [On-prem load balancing]
            → ECS task containers     [Actual workload]

Architecture Components

Component Overview

The full architecture has six layers:

1. On-Premises External Instances

Physical or virtual servers in your data center, registered as ECS external instances via SSM. Runs the ECS agent, SSM agent, Docker, and your workload containers.

2. Traefik Reverse Proxy (On-Premises)

A containerized reverse proxy running as an ECS service on an external instance dedicated to load balancing. Discovers ECS task endpoints via the ECS API and routes inbound HTTP/HTTPS traffic across containers.

3. ECS Cluster (Unified)

A single ECS cluster containing both external instances (on-premises) and EC2/Fargate capacity (AWS). The same cluster runs EXTERNAL launch type tasks and EC2/FARGATE launch type tasks.

4. Network Load Balancers (Per Region)

NLBs in each AWS region front the cloud-based ECS tasks. Unlike the external instance limitation, AWS ECS services on EC2/Fargate integrate natively with NLBs.

5. AWS Global Accelerator

Two static Anycast IPs serve as the global front door. Global Accelerator routes traffic to NLBs (and optionally, to registered on-premises endpoints) based on health, latency, and traffic dial configuration.aws.amazon+1

6. AWS Systems Manager (Control Channel)

All communication between the ECS control plane and external instances flows through SSM. This means your on-premises servers only need outbound HTTPS access to AWS endpoints—no inbound firewall rules, no VPN required for the control plane.oneuptime+1

Network Connectivity Requirements

Your on-premises instances need outbound HTTPS (TCP 443) access to these AWS endpoints:

# ECS control plane
ecs-a-*.{region}.amazonaws.com     # Task management
ecs-t-*.{region}.amazonaws.com     # Task and container metrics
ecs.{region}.amazonaws.com         # ECS service endpoint

# SSM (control channel)
ssm.{region}.amazonaws.com
ec2messages.{region}.amazonaws.com
ssmmessages.{region}.amazonaws.com

# For pulling container images
ecr.{region}.amazonaws.com
*.dkr.ecr.{region}.amazonaws.com

This is important: no VPN, no AWS Direct Connect, no site-to-site connectivity is required for the ECS control plane. Direct Connect or VPN is only needed if your on-premises containers need to reach VPC-native resources like RDS databases or private APIs.

Step 1: IAM Roles and Permissions

Everything starts with IAM. We need roles for external instances and for Traefik to call ECS APIs.

ECS Anywhere Instance Role

# Create the trust policy (SSM assumes this role)
cat > ecs-anywhere-trust-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ssm.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

# Create the IAM role
aws iam create-role \
  --role-name ecsAnywhereInstanceRole \
  --assume-role-policy-document file://ecs-anywhere-trust-policy.json \
  --description "Role assumed by SSM on ECS Anywhere external instances"

# Attach the two required managed policies
aws iam attach-role-policy \
  --role-name ecsAnywhereInstanceRole \
  --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore

aws iam attach-role-policy \
  --role-name ecsAnywhereInstanceRole \
  --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerServiceforEC2Role

The AmazonSSMManagedInstanceCore policy allows the instance to communicate with Systems Manager. AmazonEC2ContainerServiceforEC2Role allows the ECS agent to register instances, pull task metadata, and report container status.

Traefik Task Role

Traefik needs read access to ECS APIs to discover container endpoints:[

# Create the Traefik task policy
cat > traefik-ecs-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "TraefikECSReadAccess",
      "Effect": "Allow",
      "Action": [
        "ecs:ListClusters",
        "ecs:DescribeClusters",
        "ecs:ListTasks",
        "ecs:DescribeTasks",
        "ecs:DescribeContainerInstances",
        "ecs:DescribeTaskDefinition",
        "ec2:DescribeInstances",
        "ssm:DescribeInstanceInformation"
      ],
      "Resource": ["*"]
    }
  ]
}
EOF

aws iam create-policy \
  --policy-name TraefikECSReadAccess \
  --policy-document file://traefik-ecs-policy.json

# Create the task execution role trust policy
cat > task-trust-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ecs-tasks.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

# Create Traefik task role
aws iam create-role \
  --role-name traefik-task-role \
  --assume-role-policy-document file://task-trust-policy.json

# Attach the policy
ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
aws iam attach-role-policy \
  --role-name traefik-task-role \
  --policy-arn arn:aws:iam::${ACCOUNT_ID}:policy/TraefikECSReadAccess

# Task execution role (for pulling images, CloudWatch Logs)
aws iam create-role \
  --role-name ecsTaskExecutionRole \
  --assume-role-policy-document file://task-trust-policy.json

aws iam attach-role-policy \
  --role-name ecsTaskExecutionRole \
  --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy

Note that ssm:DescribeInstanceInformation is required by Traefik to map SSM instance IDs to IP addresses for ECS Anywhere external instances. Without this, Traefik can discover tasks but cannot resolve their actual network addresses.

Step 2: Create the ECS Cluster

Create a single cluster that will contain both external instances and cloud-based capacity:

# Create the cluster with CloudWatch Container Insights enabled
aws ecs create-cluster \
  --cluster-name hybrid-production \
  --settings name=containerInsights,value=enabled \
  --tags key=Environment,value=production key=Architecture,value=hybrid

# Verify cluster creation
aws ecs describe-clusters \
  --clusters hybrid-production \
  --query 'clusters[0].{Name:clusterName,Status:status,ActiveServices:activeServicesCount}'

This single cluster will show all capacity—external instances from your data center and EC2/Fargate from AWS—in one unified view.

Step 3: Register External Instances

Each on-premises server that will run ECS tasks must be registered. This uses the SSM activation mechanism.aws.amazon+1

Generate the Activation

# Create SSM activation
# This generates an activation code and ID that you run on the on-premises server
aws ssm create-activation \
  --iam-role ecsAnywhereInstanceRole \
  --registration-limit 20 \
  --default-instance-name "ecs-external-onprem" \
  --description "ECS Anywhere external instances for hybrid cluster" \
  --tags "Key=Location,Value=frankfurt-dc" "Key=Environment,Value=production" \
  --expiration-date "2026-12-31T00:00:00Z" \
  --region us-east-1

# Output:
# {
#   "ActivationId": "abc12345-1234-1234-1234-abc1234567890",
#   "ActivationCode": "codeXXXXXXXXXXXX"
# }

Store the ActivationId and ActivationCode—you'll need them on each on-premises server.

Install Agents on On-Premises Servers

SSH into each on-premises server and run the ECS Anywhere installation script:aws.amazon+1

# On each on-premises server (Ubuntu 22/24, RHEL 9, Amazon Linux 2023)

# Download and run the ECS Anywhere installation script
curl -o "/tmp/ecs-anywhere-install.sh" \
  "https://amazon-ecs-agent.s3.amazonaws.com/ecs-anywhere-install-latest.sh"

chmod +x /tmp/ecs-anywhere-install.sh

# Install with activation credentials
sudo bash /tmp/ecs-anywhere-install.sh \
  --region us-east-1 \
  --cluster hybrid-production \
  --activation-id abc12345-1234-1234-1234-abc1234567890 \
  --activation-code codeXXXXXXXXXXXX

The script performs the following automatically:

1. Installs Docker if not present

2. Installs and configures the SSM agent

3. Registers the instance with Systems Manager using the activation credentials

4. Installs the ECS container agent

5. Configures the ECS agent to communicate with the cluster

6. Starts both agents as system services

After the script completes, verify the instance is registered:

# On the AWS side - check SSM managed instances
aws ssm describe-instance-information \
  --filters "Key=tag:Location,Values=frankfurt-dc" \
  --query 'InstanceInformationList[*].{ID:InstanceId,Ping:PingStatus,IP:IPAddress}'

# Check ECS cluster capacity
aws ecs list-container-instances \
  --cluster hybrid-production \
  --filter "attribute:ecs.capability.external exists"

# Describe external instances
aws ecs describe-container-instances \
  --cluster hybrid-production \
  --container-instances $(aws ecs list-container-instances \
    --cluster hybrid-production \
    --filter "attribute:ecs.capability.external exists" \
    --query 'containerInstanceArns' \
    --output text) \
  --query 'containerInstances[*].{ID:containerInstanceArn,Status:status,CPU:registeredResources[?name=CPU].integerValue|[0],Mem:registeredResources[?name=MEMORY].integerValue|[0]}'

Configure Custom Attributes for Task Placement

Custom attributes allow you to target specific tasks to specific external instances—the key mechanism for putting Traefik on dedicated "load balancer" instances:​

# Assign load balancer role to dedicated instance
aws ecs put-attributes \
  --cluster hybrid-production \
  --attributes name=role,value=loadbalancer,targetType=container-instance,targetId=<INSTANCE_ARN_1>

# Assign webserver role to application instances
aws ecs put-attributes \
  --cluster hybrid-production \
  --attributes name=role,value=webserver,targetType=container-instance,targetId=<INSTANCE_ARN_2>

aws ecs put-attributes \
  --cluster hybrid-production \
  --attributes name=role,value=webserver,targetType=container-instance,targetId=<INSTANCE_ARN_3>

# Add location attribute for topology-aware placement
aws ecs put-attributes \
  --cluster hybrid-production \
  --attributes name=location,value=frankfurt-dc,targetType=container-instance,targetId=<INSTANCE_ARN_1>

aws ecs put-attributes \
  --cluster hybrid-production \
  --attributes name=location,value=frankfurt-dc,targetType=container-instance,targetId=<INSTANCE_ARN_2>

aws ecs put-attributes \
  --cluster hybrid-production \
  --attributes name=location,value=frankfurt-dc,targetType=container-instance,targetId=<INSTANCE_ARN_3>

You can also configure the necessary routing rules on the external instances to support IAM roles for tasks when not using an ECS-optimized AMI:

# Required on external instances for IAM roles for tasks to work
sudo sysctl -w net.ipv4.conf.all.route_localnet=1
sudo iptables -t nat -A PREROUTING -p tcp -d 169.254.170.2 --dport 80 \
  -j DNAT --to-destination 127.0.0.1:51679
sudo iptables -t nat -A OUTPUT -d 169.254.170.2 -p tcp -m tcp --dport 80 \
  -j REDIRECT --to-ports 51679

# Persist across reboots
sudo sh -c "echo 'net.ipv4.conf.all.route_localnet=1' >> /etc/sysctl.conf"
sudo iptables-save > /etc/iptables/rules.v4

Step 4: Deploy Traefik as the On-Premises Load Balancer

Since ECS Anywhere doesn't support Elastic Load Balancing, we deploy Traefik as a containerized reverse proxy on a dedicated external instance. Traefik uses its built-in ECS provider to automatically discover containers, making it a natural fit for this pattern.

Understanding Traefik's ECS Provider

Traefik's ECS provider:

• Calls the ECS API (ListTasks, DescribeTasks, DescribeContainerInstances) every 15 seconds

• For external instances, also calls ssm:DescribeInstanceInformation to resolve instance IPs

• Builds its routing table from the discovered task IPs and ports

• Uses Docker labels on task containers to determine routing rules

• Updates routes dynamically as tasks start and stop—no Traefik restart needed

This means when ECS scales your application tasks up or down, Traefik's routing table updates within 15 seconds.

Traefik Task Definition

// task-definition-traefik.json
{
  "family": "TraefikLoadBalancer",
  "cpu": "512",
  "memory": "256",
  "networkMode": "host",
  "requiresCompatibilities": ["EXTERNAL"],
  "taskRoleArn": "arn:aws:iam::123456789012:role/traefik-task-role",
  "executionRoleArn": "arn:aws:iam::123456789012:role/ecsTaskExecutionRole",
  "containerDefinitions": [
    {
      "name": "traefik",
      "image": "traefik:v3.2",
      "essential": true,
      "portMappings": [
        {
          "hostPort": 80,
          "protocol": "tcp",
          "containerPort": 80
        },
        {
          "hostPort": 443,
          "protocol": "tcp",
          "containerPort": 443
        },
        {
          "hostPort": 8080,
          "protocol": "tcp",
          "containerPort": 8080
        }
      ],
      "command": [
        "--log.level=INFO",
        "--accesslog=true",
        "--api.dashboard=true",
        "--api.insecure=true",
        "--providers.ecs.ecsAnywhere=true",
        "--providers.ecs.region=us-east-1",
        "--providers.ecs.autoDiscoverClusters=true",
        "--providers.ecs.clusters=hybrid-production",
        "--providers.ecs.exposedByDefault=false",
        "--entrypoints.web.address=:80",
        "--entrypoints.websecure.address=:443",
        "--entrypoints.web.http.redirections.entrypoint.to=websecure",
        "--entrypoints.web.http.redirections.entrypoint.scheme=https",
        "--certificatesresolvers.letsencrypt.acme.email=ops@example.com",
        "--certificatesresolvers.letsencrypt.acme.storage=/data/acme.json",
        "--certificatesresolvers.letsencrypt.acme.tlschallenge=true",
        "--metrics.prometheus=true",
        "--ping=true"
      ],
      "mountPoints": [
        {
          "sourceVolume": "traefik-data",
          "containerPath": "/data"
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/traefik-loadbalancer",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "traefik"
        }
      },
      "healthCheck": {
        "command": ["CMD", "traefik", "healthcheck", "--ping"],
        "interval": 30,
        "timeout": 5,
        "retries": 3,
        "startPeriod": 10
      }
    }
  ],
  "volumes": [
    {
      "name": "traefik-data",
      "host": {
        "sourcePath": "/opt/traefik/data"
      }
    }
  ],
  "placementConstraints": [
    {
      "type": "memberOf",
      "expression": "attribute:role == loadbalancer"
    },
    {
      "type": "memberOf",
      "expression": "attribute:location == frankfurt-dc"
    }
  ]
}

Key configuration choices:

• networkMode: host: Required for Traefik to bind ports 80/443 directly on the external instance's network interface. In bridge mode, external traffic wouldn't reach the container

• ecsAnywhere=true: Tells Traefik's ECS provider to include external instances in discovery, not just EC2 instances

• exposedByDefault=false: Only route to containers that explicitly opt-in via Docker labels—prevents accidentally routing to the ECS agent or SSM agent containers

• --ping=true: Enables the /ping health endpoint, used later for Global Accelerator health checks

• TLS with Let's Encrypt: Handles certificate issuance and renewal automatically

Create the CloudWatch log group first:

aws logs create-log-group --log-group-name /ecs/traefik-loadbalancer
aws logs put-retention-policy \
  --log-group-name /ecs/traefik-loadbalancer \
  --retention-in-days 30

Register and deploy:

aws ecs register-task-definition \
  --cli-input-json file://task-definition-traefik.json

aws ecs create-service \
  --cluster hybrid-production \
  --service-name TraefikLoadBalancer \
  --task-definition TraefikLoadBalancer:1 \
  --desired-count 1 \
  --launch-type EXTERNAL \
  --deployment-configuration "minimumHealthyPercent=100,maximumPercent=200" \
  --deployment-controller type=ECS

Verify Traefik is running:

# Get the external instance IP
INSTANCE_IP=$(aws ssm describe-instance-information \
  --filters "Key=tag:Location,Values=frankfurt-dc" \
  --query 'InstanceInformationList[?contains(InstanceId, mi-)].IPAddress|[0]' \
  --output text)

# Check Traefik dashboard
curl -s http://${INSTANCE_IP}:8080/api/overview | jq '{version:.version, providers:.features.providerPlugins}'

# Check health endpoint
curl -s http://${INSTANCE_IP}:8080/ping
# Expected: OK

Step 5: Deploy the Application Tasks on External Instances

Now deploy your actual application containers. We'll use a sample web API to illustrate the routing configuration:

// task-definition-api.json
{
  "family": "WebAPI",
  "cpu": "512",
  "memory": "512",
  "networkMode": "bridge",
  "requiresCompatibilities": ["EXTERNAL"],
  "executionRoleArn": "arn:aws:iam::123456789012:role/ecsTaskExecutionRole",
  "containerDefinitions": [
    {
      "name": "api",
      "image": "123456789012.dkr.ecr.us-east-1.amazonaws.com/web-api:latest",
      "essential": true,
      "portMappings": [
        {
          "hostPort": 0,
          "protocol": "tcp",
          "containerPort": 8080
        }
      ],
      "dockerLabels": {
        "traefik.enable": "true",
        "traefik.http.routers.api.rule": "Hostapi.example.com) || PathPrefix/api)",
        "traefik.http.routers.api.entrypoints": "websecure",
        "traefik.http.routers.api.tls": "true",
        "traefik.http.routers.api.tls.certresolver": "letsencrypt",
        "traefik.http.services.api.loadbalancer.server.port": "8080",
        "traefik.http.services.api.loadbalancer.healthcheck.path": "/health",
        "traefik.http.services.api.loadbalancer.healthcheck.interval": "10s",
        "traefik.http.services.api.loadbalancer.sticky.cookie": "true",
        "traefik.http.services.api.loadbalancer.sticky.cookie.name": "lb_session"
      },
      "environment": [
        {
          "name": "ENVIRONMENT",
          "value": "production"
        },
        {
          "name": "LOCATION",
          "value": "on-premises"
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/web-api-external",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "web-api"
        }
      },
      "healthCheck": {
        "command": ["CMD-SHELL", "curl -f http://localhost:8080/health || exit 1"],
        "interval": 15,
        "timeout": 5,
        "retries": 3,
        "startPeriod": 30
      }
    }
  ],
  "placementConstraints": [
    {
      "type": "memberOf",
      "expression": "attribute:role == webserver"
    },
    {
      "type": "memberOf",
      "expression": "attribute:location == frankfurt-dc"
    }
  ]
}

The Docker labels are the critical Traefik configuration:

• traefik.enable=true: Opts this container into Traefik routing (required since we set exposedByDefault=false)

• traefik.http.routers.api.rule: Host and/or path matching rule for routing inbound requests

• traefik.http.services.api.loadbalancer.server.port: Which container port Traefik proxies to

• traefik.http.services.api.loadbalancer.healthcheck.path: Traefik health checks each container backend before routing to it

• sticky.cookie: Session affinity via cookie—useful for stateful applications

Register and deploy:

# Create log group
aws logs create-log-group --log-group-name /ecs/web-api-external

aws ecs register-task-definition \
  --cli-input-json file://task-definition-api.json

# Deploy 4 tasks spread across the 2 webserver instances
aws ecs create-service \
  --cluster hybrid-production \
  --service-name WebAPI-OnPrem \
  --task-definition WebAPI:1 \
  --desired-count 4 \
  --launch-type EXTERNAL \
  --placement-strategy type=spread,field=instanceId \
  --deployment-configuration "minimumHealthyPercent=50,maximumPercent=200"

The spread placement strategy distributes tasks evenly across all external instances with the webserver attribute, avoiding concentration on a single host.

Verify Traefik Discovery

After the tasks start, verify Traefik has discovered them:

# Check Traefik's view of the backend services
curl -s http://${INSTANCE_IP}:8080/api/http/services | \
  jq '.[] | {name:.name, servers:[.serverStatus|to_entries[]|{url:.key, status:.value}]}'

# Expected output shows 4 task containers as backends:
# {
#   "name": "api@ecs",
#   "servers": [
#     {"url": "http://10.0.1.15:32768", "status": "UP"},
#     {"url": "http://10.0.1.15:32769", "status": "UP"},
#     {"url": "http://10.0.1.16:32768", "status": "UP"},
#     {"url": "http://10.0.1.16:32769", "status": "UP"}
#   ]
# }

# Test round-robin load balancing
for i in 1 2 3 4; do
  curl -s -H "Host: api.example.com" \
    http://${INSTANCE_IP}/api/health | jq -r '.container_id'
done

# Should see 4 different container IDs cycling

Step 6: Deploy ECS Tasks in AWS Regions

For the cloud side, deploy the same application on EC2 or Fargate with native load balancing support.

Create the AWS-Side Infrastructure with Terraform

# main.tf
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

# Primary region (us-east-1)
provider "aws" {
  alias  = "primary"
  region = "us-east-1"
}

# Secondary region (eu-west-1)
provider "aws" {
  alias  = "secondary"
  region = "eu-west-1"
}

# VPC for primary region
module "vpc_primary" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"
  providers = { aws = aws.primary }

  name = "hybrid-vpc-primary"
  cidr = "10.10.0.0/16"

  azs             = ["us-east-1a", "us-east-1b", "us-east-1c"]
  private_subnets = ["10.10.1.0/24", "10.10.2.0/24", "10.10.3.0/24"]
  public_subnets  = ["10.10.101.0/24", "10.10.102.0/24", "10.10.103.0/24"]

  enable_nat_gateway = true
  single_nat_gateway = false  # One per AZ for HA
}

# Network Load Balancer - Primary Region
resource "aws_lb" "primary" {
  provider = aws.primary
  name               = "hybrid-nlb-primary"
  internal           = false
  load_balancer_type = "network"
  subnets            = module.vpc_primary.public_subnets

  enable_deletion_protection       = true
  enable_cross_zone_load_balancing = true

  tags = {
    Environment = "production"
    Region      = "us-east-1"
  }
}

resource "aws_lb_target_group" "primary_api" {
  provider = aws.primary
  name        = "hybrid-api-primary-tg"
  port        = 443
  protocol    = "TLS"
  target_type = "ip"
  vpc_id      = module.vpc_primary.vpc_id

  health_check {
    enabled             = true
    healthy_threshold   = 2
    unhealthy_threshold = 2
    timeout             = 10
    interval            = 30
    protocol            = "HTTPS"
    path                = "/health"
  }

  tags = { Environment = "production" }
}

resource "aws_lb_listener" "primary_https" {
  provider          = aws.primary
  load_balancer_arn = aws_lb.primary.arn
  port              = 443
  protocol          = "TLS"
  ssl_policy        = "ELBSecurityPolicy-TLS13-1-2-2021-06"
  certificate_arn   = aws_acm_certificate.primary.arn

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.primary_api.arn
  }
}

# ECS Cluster - Primary Region
resource "aws_ecs_cluster" "primary" {
  provider = aws.primary
  name     = "hybrid-production"

  setting {
    name  = "containerInsights"
    value = "enabled"
  }
}

# ECS Service for Web API - Primary Region (Fargate)
resource "aws_ecs_service" "primary_api" {
  provider        = aws.primary
  name            = "WebAPI-Cloud"
  cluster         = aws_ecs_cluster.primary.id
  task_definition = aws_ecs_task_definition.api_fargate.arn
  desired_count   = 3
  launch_type     = "FARGATE"

  network_configuration {
    subnets          = module.vpc_primary.private_subnets
    security_groups  = [aws_security_group.ecs_tasks.id]
    assign_public_ip = false
  }

  load_balancer {
    target_group_arn = aws_lb_target_group.primary_api.arn
    container_name   = "api"
    container_port   = 8080
  }

  deployment_configuration {
    minimum_healthy_percent = 100
    maximum_percent         = 200

    deployment_circuit_breaker {
      enable   = true
      rollback = true
    }
  }

  lifecycle {
    ignore_changes = [desired_count]
  }
}

The Fargate task definition for the cloud-based deployment uses awsvpc networking mode (supported in AWS, unlike ECS Anywhere external instances):

// task-definition-api-fargate.json
{
  "family": "WebAPI-Fargate",
  "cpu": "512",
  "memory": "1024",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "executionRoleArn": "arn:aws:iam::123456789012:role/ecsTaskExecutionRole",
  "taskRoleArn": "arn:aws:iam::123456789012:role/web-api-task-role",
  "containerDefinitions": [
    {
      "name": "api",
      "image": "123456789012.dkr.ecr.us-east-1.amazonaws.com/web-api:latest",
      "essential": true,
      "portMappings": [
        {
          "containerPort": 8080,
          "protocol": "tcp"
        }
      ],
      "environment": [
        {
          "name": "ENVIRONMENT",
          "value": "production"
        },
        {
          "name": "LOCATION",
          "value": "aws-cloud"
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/web-api-cloud",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "web-api"
        }
      },
      "healthCheck": {
        "command": ["CMD-SHELL", "curl -f http://localhost:8080/health || exit 1"],
        "interval": 15,
        "timeout": 5,
        "retries": 3
      }
    }
  ]
}

Step 7: Configure AWS Global Accelerator

Global Accelerator is where the cross-region load balancing comes together. It provides two static Anycast IPs and routes traffic based on health and latency.

Create the Accelerator

# Create Global Accelerator
aws globalaccelerator create-accelerator \
  --name hybrid-ecs-accelerator \
  --ip-address-type IPV4 \
  --enabled \
  --tags Key=Application,Value=HybridECS Key=Environment,Value=Production \
  --region us-west-2  # Global Accelerator is global but managed from us-west-2

# Get the static IP addresses (use these in your DNS)
ACCELERATOR_ARN=$(aws globalaccelerator list-accelerators \
  --query 'Accelerators[?Name=hybrid-ecs-accelerator].AcceleratorArn' \
  --output text \
  --region us-west-2)

aws globalaccelerator describe-accelerator \
  --accelerator-arn ${ACCELERATOR_ARN} \
  --query 'Accelerator.IpSets[0].IpAddresses' \
  --region us-west-2
# Returns two static IP addresses - put these in your DNS A records

Create the Listener

# Create listener on port 443
aws globalaccelerator create-listener \
  --accelerator-arn ${ACCELERATOR_ARN} \
  --protocol TCP \
  --port-ranges FromPort=443,ToPort=443 \
  --client-affinity SOURCE_IP \
  --region us-west-2

LISTENER_ARN=$(aws globalaccelerator list-listeners \
  --accelerator-arn ${ACCELERATOR_ARN} \
  --query 'Listeners[0].ListenerArn' \
  --output text \
  --region us-west-2)

CLIENT_AFFINITY=SOURCE_IP ensures the same client IP always routes to the same endpoint region during a session—important for stateful applications.

Create Endpoint Groups

Each endpoint group corresponds to a region and contains endpoints (NLBs):

# Endpoint group for primary region (us-east-1)
aws globalaccelerator create-endpoint-group \
  --listener-arn ${LISTENER_ARN} \
  --endpoint-group-region us-east-1 \
  --endpoint-configurations \
    EndpointId=${NLB_PRIMARY_ARN},Weight=100,ClientIPPreservationEnabled=true \
  --traffic-dial-percentage 100 \
  --health-check-protocol HTTPS \
  --health-check-path /health \
  --health-check-port 443 \
  --health-check-interval-seconds 10 \
  --threshold-count 2 \
  --region us-west-2

# Endpoint group for secondary region (eu-west-1)
aws globalaccelerator create-endpoint-group \
  --listener-arn ${LISTENER_ARN} \
  --endpoint-group-region eu-west-1 \
  --endpoint-configurations \
    EndpointId=${NLB_SECONDARY_ARN},Weight=100,ClientIPPreservationEnabled=true \
  --traffic-dial-percentage 100 \
  --health-check-protocol HTTPS \
  --health-check-path /health \
  --health-check-port 443 \
  --health-check-interval-seconds 10 \
  --threshold-count 2 \
  --region us-west-2

TrafficDialPercentage: Controls what percentage of traffic goes to each endpoint group. During a deployment or failover test, you can dial us-east-1 down to 0% to drain traffic without removing the endpoint group entirely.

ClientIPPreservationEnabled: Forwards the original client IP to the NLB target—required if your application needs real client IPs for logging or rate limiting.

Health Check Behavior

Global Accelerator health checks work at the endpoint group level:aws.amazon+1

• If an NLB's health check fails consistently (threshold-count times), Global Accelerator stops sending traffic to that endpoint group

• Traffic automatically shifts to the next healthiest endpoint group based on routing policy

• When the failing endpoint recovers, traffic can shift back automatically

This is your cross-region failover mechanism. A regional outage that takes down the us-east-1 NLB will cause Global Accelerator to shift 100% of traffic to eu-west-1 within ~30 seconds (10s health check × 2 failures + propagation time).

Step 8: Connecting On-Premises to Global Accelerator

Here's the subtlety that the AWS Solutions Guidance solves in a specific way: how does on-premises traffic participate in the Global Accelerator routing?

There are two patterns:

Pattern A: On-Premises as an NLB Target (Recommended for Direct Connect customers)

If you have AWS Direct Connect or Site-to-Site VPN, you can register your on-premises Traefik instance as a target in an NLB target group using its private IP. The NLB's target group must be configured with target_type=ip and the on-premises IPs must be reachable from the VPC:

# Register on-premises Traefik instance as NLB target
# Requires Direct Connect or VPN for VPC connectivity
aws elbv2 register-targets \
  --target-group-arn ${TG_ARN} \
  --targets \
    Id=10.100.1.10,Port=443 \   # On-premises Traefik instance 1
    Id=10.100.1.11,Port=443     # On-premises Traefik instance 2 (HA)

The NLB health-checks these IPs. If they're healthy, NLB distributes traffic across both cloud Fargate tasks and on-premises Traefik (which further distributes to ECS external instance tasks).

Pattern B: Separate On-Premises Endpoint Group

Register your on-premises public IPs or Elastic IPs directly as endpoints in Global Accelerator (not going through an NLB):

# Register on-premises public IP directly in Global Accelerator
aws globalaccelerator create-endpoint-group \
  --listener-arn ${LISTENER_ARN} \
  --endpoint-group-region us-east-1 \
  --endpoint-configurations \
    EndpointId=${ON_PREM_ELASTIC_IP_ALLOCATION_ID},Weight=50,ClientIPPreservationEnabled=false \
  --traffic-dial-percentage 30 \
  --health-check-protocol HTTPS \
  --health-check-path /ping \
  --health-check-port 8080 \
  --region us-west-2

The /ping endpoint is served by Traefik's built-in health check (--ping=true). Global Accelerator probes this to determine if the on-premises location is reachable and healthy.

The traffic-dial-percentage=30 routes 30% of global traffic to this on-premises endpoint group, 70% to the AWS regions. This is your lever for gradual traffic shifting and testing.

Traffic Weighting Within Endpoint Groups

Within an endpoint group you can weight individual endpoints:

# During canary deployment of new on-premises instance
aws globalaccelerator update-endpoint-group \
  --endpoint-group-arn ${ENDPOINT_GROUP_ARN} \
  --endpoint-configurations \
    EndpointId=${NLB_ARN},Weight=80 \        # 80% to existing NLB
    EndpointId=${NEW_NLB_ARN},Weight=20 \    # 20% to new deployment
  --region us-west-2

Step 9: Automated Failover and Disaster Recovery

The architecture's DR capabilities come from combining ECS service auto-recovery, NLB health checks, and Global Accelerator routing.aws.amazon+1

Failure Scenario 1: Single ECS Task Failure

Trigger: An application container crashes on an external instance.

Response:

1. ECS health check (from task definition healthCheck) detects failure

2. ECS stops the unhealthy task and starts a replacement on a healthy external instance

3. Traefik detects the container is no longer in DescribeTasks results at next polling interval (~15s)

4. Traefik removes the unhealthy backend from the routing pool

5. User impact: Requests that hit Traefik during the ~15s detection window may fail; subsequent requests route to healthy containers

Recovery time: 15–30 seconds

Failure Scenario 2: External Instance Failure

Trigger: An on-premises server hosting ECS tasks becomes unreachable.

Response:

1. ECS agent stops heartbeating → instance marked DRAINING after timeout

2. ECS reschedules tasks from the failed instance onto remaining external instances

3. Traefik stops routing to containers on the failed instance when they're no longer in DescribeTasks

Important constraint: ECS can only reschedule to external instances in the same cluster with the webserver attribute. If you only have 2 external instance webservers and both fail simultaneously, ECS has nowhere to reschedule. This is why the multi-region cloud capacity matters—cloud Fargate tasks continue serving via Global Accelerator even if on-premises capacity is completely lost.

Failure Scenario 3: On-Premises Location Failure

Trigger: Complete data center outage. All external instances unreachable.

Response:

1. Global Accelerator health checks fail for the on-premises endpoint group

2. After threshold-count consecutive failures (2 × 10s = ~20s), traffic dial shifts to 0%

3. 100% of traffic routes to AWS NLBs in us-east-1 and eu-west-1

4. Cloud ECS services (Fargate) continue serving normally

User impact: ~20–30 seconds for Global Accelerator to detect and reroute. During this window, ~30% of requests (those dialed to on-premises) fail.

Recovery: When the data center comes back, SSM agent reconnects (credential rotation resumes), ECS agent re-registers external instances, tasks restart, Traefik discovers them, Global Accelerator health checks succeed, traffic dials back up automatically.

Failure Scenario 4: AWS Region Failure

Trigger: us-east-1 regional outage.

Response:

1. Global Accelerator health checks fail for us-east-1 endpoint group

2. Traffic shifts to eu-west-1 NLB and/or on-premises endpoints (if healthy)

3. ECS control plane impact: if the ECS service is in us-east-1, you cannot deploy new tasks to external instances registered to that cluster from the ECS console

Recovery consideration: Your ECS cluster and external instance registration are tied to a specific region. If us-east-1 is your cluster region and it's unavailable, you lose the ability to manage external instances through ECS—even though those instances are physically on-premises and running. This is why for maximum DR, you should register your on-premises instances to clusters in multiple regions.

DR Architecture for Maximum Resilience

# Register the same on-premises server to clusters in multiple regions
# (An external instance can only be in ONE cluster at a time)
# So you need separate physical servers registered to each regional cluster

# Germany DC Server Pool A → Registered to ECS cluster in eu-west-1
# Germany DC Server Pool B → Registered to ECS cluster in us-east-1
# Traefik on each pool routes to its local containers

# Both Traefik endpoints registered in Global Accelerator
# If eu-west-1 ECS control plane fails, Pool B (controlled by us-east-1) still serves

Step 10: Monitoring and Observability

With workloads spanning on-premises and multiple AWS regions, centralized observability is essential.

CloudWatch Container Insights

ECS external instances send metrics to CloudWatch like any other ECS instance:

# View Container Insights metrics for the hybrid cluster
aws cloudwatch get-metric-statistics \
  --namespace "ECS/ContainerInsights" \
  --metric-name "CpuUtilized" \
  --dimensions Name=ClusterName,Value=hybrid-production \
  --start-time "$(date -u -d '1 hour ago' '+%Y-%m-%dT%H:%M:%S')" \
  --end-time "$(date -u '+%Y-%m-%dT%H:%M:%S')" \
  --period 300 \
  --statistics Average

# Separate metrics for external vs cloud instances
aws cloudwatch get-metric-statistics \
  --namespace "ECS/ContainerInsights" \
  --metric-name "CpuUtilized" \
  --dimensions \
    Name=ClusterName,Value=hybrid-production \
    Name=LaunchType,Value=EXTERNAL \
  --start-time "$(date -u -d '1 hour ago' '+%Y-%m-%dT%H:%M:%S')" \
  --end-time "$(date -u '+%Y-%m-%dT%H:%M:%S')" \
  --period 300 \
  --statistics Average

CloudWatch Alarms for Cross-Location Health

# Global Accelerator health alarm
resource "aws_cloudwatch_metric_alarm" "onprem_endpoint_health" {
  alarm_name          = "onprem-global-accelerator-health"
  comparison_operator = "LessThanThreshold"
  evaluation_periods  = 2
  metric_name         = "EndpointsHealthy"
  namespace           = "AWS/GlobalAccelerator"
  period              = 60
  statistic           = "Average"
  threshold           = 1
  alarm_description   = "On-premises endpoint group has no healthy endpoints"
  treat_missing_data  = "breaching"

  dimensions = {
    Accelerator    = aws_globalaccelerator_accelerator.main.id
    EndpointGroup  = aws_globalaccelerator_endpoint_group.onprem.id
    Listener       = aws_globalaccelerator_listener.main.id
  }

  alarm_actions = [aws_sns_topic.operations.arn]
}

# ECS external instance connectivity
resource "aws_cloudwatch_metric_alarm" "external_instance_count" {
  alarm_name          = "ecs-external-instance-count-low"
  comparison_operator = "LessThanThreshold"
  evaluation_periods  = 3
  metric_name         = "RegisteredContainerInstancesCount"
  namespace           = "ECS/ContainerInsights"
  period              = 60
  statistic           = "Average"
  threshold           = 2  # Alert if fewer than 2 external instances are healthy
  alarm_description   = "ECS cluster has fewer than 2 external instances registered"
  treat_missing_data  = "breaching"

  dimensions = {
    ClusterName = "hybrid-production"
  }

  alarm_actions = [aws_sns_topic.operations.arn]
}

# Traefik backend health
resource "aws_cloudwatch_log_metric_filter" "traefik_backend_down" {
  name           = "traefik-backend-down"
  pattern        = "[timestamp, level=\"error\", msg=\"No servers available yet\"]"
  log_group_name = "/ecs/traefik-loadbalancer"

  metric_transformation {
    name      = "TraefikBackendDown"
    namespace = "HybridECS/Traefik"
    value     = "1"
  }
}

resource "aws_cloudwatch_metric_alarm" "traefik_backend_errors" {
  alarm_name          = "traefik-no-backends"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 1
  metric_name         = "TraefikBackendDown"
  namespace           = "HybridECS/Traefik"
  period              = 60
  statistic           = "Sum"
  threshold           = 0

  alarm_actions = [aws_sns_topic.operations.arn]
}

Traefik Prometheus Metrics

Traefik exposes Prometheus metrics when --metrics.prometheus=true is set. Scrape them from your on-premises Prometheus or AWS Managed Prometheus:

# prometheus-scrape-config.yaml (for Prometheus running on-premises or via AMP)
scrape_configs:
  - job_name: 'traefik-onprem'
    static_configs:
      - targets:
          - '10.100.1.10:8080'
          - '10.100.1.11:8080'
    metrics_path: /metrics
    scheme: http

    # Key metrics to alert on:
    # traefik_backend_requests_total
    # traefik_backend_request_duration_seconds_histogram_bucket
    # traefik_backend_server_up (0 = down, 1 = up)
    # traefik_service_requests_total{code="5xx"}

Deployment Pipeline Considerations

When you have a unified ECS cluster with external and cloud capacity, your CI/CD pipeline can deploy to both locations from a single update-service call.

# Deploy new image to both on-premises and cloud tasks simultaneously
NEW_IMAGE="123456789012.dkr.ecr.us-east-1.amazonaws.com/web-api:v2.0.0"

# Update the task definition
NEW_TD=$(aws ecs register-task-definition \
  --cli-input-json "$(aws ecs describe-task-definition \
    --task-definition WebAPI \
    --query 'taskDefinition' | \
    jq --arg image "$NEW_IMAGE" \
    '.containerDefinitions[0].image = $image | del(.taskDefinitionArn,.revision,.status,.requiresAttributes,.compatibilities,.registeredAt,.registeredBy)'
  )" \
  --query 'taskDefinition.taskDefinitionArn' \
  --output text)

# Deploy on-premises tasks
aws ecs update-service \
  --cluster hybrid-production \
  --service WebAPI-OnPrem \
  --task-definition $NEW_TD \
  --force-new-deployment

# Deploy cloud tasks (primary region)
aws ecs update-service \
  --cluster hybrid-production \
  --service WebAPI-Cloud \
  --task-definition $NEW_TD \
  --force-new-deployment

For zero-downtime deployments on external instances:

• ECS rolling update: minimumHealthyPercent=50 ensures at least half the tasks remain running during deployment

• Traefik health checks (healthcheck.path=/health) prevent traffic from routing to tasks that are starting or not yet healthy

• Combined: new tasks start → Traefik waits for health check → old tasks drain → zero downtime

Cost Model

Understanding the cost structure helps justify this architecture versus alternatives:

Included in ECS Anywhere pricing:

• ECS Anywhere charges $0.01025 per hour per registered external instance

• This covers ECS control plane management—the SSM connectivity, scheduling, deployment management

Not included (your existing costs):

• The on-premises hardware itself (leveraging existing investment)

• Your data center's power and networking

AWS cost additions for this architecture:

• Global Accelerator: ~\(0.025/GB for data processed + \)0.025/hour per accelerator

• NLBs per region: ~$0.008/hour + LCU charges

• CloudWatch Container Insights: ~$0.50 per instance/month

• ECR image storage and data transfer (external instances pull images from ECR)

The cost value proposition: You use existing on-premises hardware investment for compute-heavy workloads that must remain on-premises (compliance), while paying AWS only for the control plane management and cross-region routing—not for the underlying compute.

Operational Runbook Snippets

A few operational tasks that come up frequently:

Drain an external instance for maintenance:

# Gracefully drain tasks before taking the instance offline
aws ecs update-container-instances-state \
  --cluster hybrid-production \
  --container-instances <INSTANCE_ARN> \
  --status DRAINING

# Wait for all tasks to move off
watch -n 10 "aws ecs describe-container-instances \
  --cluster hybrid-production \
  --container-instances <INSTANCE_ARN> \
  --query 'containerInstances[0].{Status:status, RunningTasks:runningTasksCount, PendingTasks:pendingTasksCount}'"

Deregister an external instance:

# Deregister from ECS
aws ecs deregister-container-instance \
  --cluster hybrid-production \
  --container-instance <INSTANCE_ARN> \
  --force

# Deregister from SSM
aws ssm deregister-managed-instance \
  --instance-id <SSM_INSTANCE_ID>

Emergency traffic failover to cloud-only:

# Zero out traffic dial for on-premises endpoint group
aws globalaccelerator update-endpoint-group \
  --endpoint-group-arn ${ONPREM_ENDPOINT_GROUP_ARN} \
  --traffic-dial-percentage 0 \
  --region us-west-2

# Verify all traffic going to cloud regions
aws cloudwatch get-metric-statistics \
  --namespace AWS/GlobalAccelerator \
  --metric-name NewFlowCount \
  --dimensions Name=Accelerator,Value=${ACCELERATOR_ID} \
  --start-time "$(date -u -d '5 minutes ago' '+%Y-%m-%dT%H:%M:%S')" \
  --end-time "$(date -u '+%Y-%m-%dT%H:%M:%S')" \
  --period 60 \
  --statistics Sum

When to Use This Architecture

This pattern is right for you when you have a genuine requirement for both locations—not as a stepping stone to full cloud migration:

Strong fit:

• Data residency/sovereignty: Regulations prohibit certain data leaving a geography, but you still want unified orchestration

• Latency-critical edge workloads: Manufacturing, retail POS, industrial IoT where cloud round-trip adds unacceptable latency

• Brownfield hybrid: Existing on-premises infrastructure with multi-year depreciation that isn't economically justified to retire yet

• Compliance-driven split: Some tiers (web frontend) in cloud, others (database, PII processing) on-premises

Poor fit:

• If you're planning to be fully in AWS within 18 months—the operational overhead of hybrid isn't worth it for a transitional state; use Direct Connect + standard ECS instead

• If your on-premises servers don't have reliable internet connectivity (SSM requires stable outbound HTTPS)

• If your workloads need awsvpc networking mode (not supported on external instances)

Wrapping Up

The architecture this guidance implements is a serious, production-grade answer to an old problem: you have infrastructure in multiple places, some of it non-negotiably on-premises, and you need consistent container management and global traffic distribution across all of it.

Amazon ECS Anywhere removes the "different tools for different places" problem. You write one task definition, one service definition, one deployment pipeline. Traefik fills the gap that native ECS load balancing doesn't cover for external instances. Global Accelerator provides the routing intelligence to stitch it all together with automatic failover.aws.amazon+2

The moving parts—SSM activation, Traefik's ECS provider, Global Accelerator endpoint groups, traffic dials—each require understanding before you trust them in production. But once the architecture is in place and you've run your first DR drill (route 100% traffic to cloud, verify application health, restore), you'll have something genuinely rare: a hybrid infrastructure that you can actually sleep through an on-premises outage with.

All implementation code and infrastructure patterns in this post are based on the AWS Solutions Library Guidance for Building a Hybrid Architecture Using Amazon ECS Anywhere with Cross-Region Load Balancing and the accompanying GitHub reference implementation. The Traefik ECS Anywhere load balancing pattern references the AWS Containers blog post by Cameron Senese. The ECS Anywhere networking constraints and agent behavior are documented in the Amazon ECS Developer Guide.

I've been running distributed GPU training jobs on EKS for a while now. At some point every team hits the same wall: you submit two training jobs, both need 8 GPUs, your cluster has 16 GPUs free, and nothing starts. No error. No timeout. No alert. Just pods sitting in Pending forever while your GPU nodes report healthy.

For a long time I assumed it was a misconfiguration on my end — wrong node selector, wrong resource request, some RBAC I missed. Then I started digging into what the default Kubernetes scheduler actually does when it sees a distributed training job, and the answer changed how I think about ML infrastructure on EKS entirely.

This post covers the problem, why it happens, and the specific Volcano + Kueue + Karpenter configuration that solves it — including the interaction between all three that nobody documents end-to-end.

The Problem: The Default Scheduler Has No Concept of "All or Nothing"

The Kubernetes default scheduler was designed for services. Schedule 3 replicas out of 5? Fine — the service degrades gracefully but keeps running. Schedule 6 workers out of 8 for a PyTorch DDP job? The job is dead. Every worker sits idle waiting for the rendezvous that never comes because the other 2 workers are stuck.

The default scheduler processes pods one at a time, greedily, with no concept that a group of pods lives or dies together. Here is exactly what happens when you submit two 8-GPU jobs to a 16-GPU cluster simultaneously:

Job A needs: pods A1–A8 (8 GPUs total)
Job B needs: pods B1–B8 (8 GPUs total)
Cluster has: 16 GPUs available

What the scheduler does:
  A1 scheduled → claims GPU node 1
  A2 scheduled → claims GPU node 2
  B1 scheduled → claims GPU node 3   ← interleaved
  B2 scheduled → claims GPU node 4
  A3, A4 scheduled → nodes 5, 6
  B3, B4 scheduled → nodes 7, 8
  ...
  A5, A6, A7, A8 → Pending (no GPU nodes left)
  B5, B6, B7, B8 → Pending (no GPU nodes left)

Final state:
  16 GPUs allocated → 0 work done
  Both jobs waiting for each other to release
  No error. No timeout. No alert.

This is not an edge case. It is the default behaviour of every Kubernetes cluster the moment you run more than one concurrent distributed training job. And it is completely silent — your GPU nodes show as healthy, your pods show as Pending with Insufficient nvidia.com/gpu, and nothing in your monitoring stack tells you why.

What Gang Scheduling Is, and Why Volcano Implements It Correctly

Gang scheduling solves this with one rule: a group of pods either all get scheduled simultaneously, or none of them do. No partial allocation. No deadlock.

Volcano is a CNCF-graduated batch scheduling system built specifically for this pattern. Its core primitive is the PodGroup — a Kubernetes object that defines an atomic scheduling unit.

apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
  name: pytorch-job-a
spec:
  minMember: 8          # schedule nothing until 8 pods can run simultaneously
  minResources:
    nvidia.com/gpu: "8"
  queue: training-queue
  priorityClassName: high-priority

When minMember: 8 is set, Volcano holds all 8 pods in a virtual queue, waits until 8 GPU slots are simultaneously available, and only then schedules all 8 atomically. Job A gets all 8. Job B waits in queue. Job B's pods never enter a Pending state that blocks Job A. The deadlock is structurally impossible.

Volcano also ships a fair-share queue system — Queue objects with weights and capacities that implement multi-tenant GPU time-sharing, priority preemption, and hierarchical queuing. In theory, you install Volcano, add schedulerName: volcano to your jobs, and the problem is solved.

In practice, you've just created a new class of failures. Because you almost certainly also have Karpenter.

The Volcano + Karpenter Conflict

Karpenter and Volcano have fundamentally different philosophies about when a pod should exist in Kubernetes, and that conflict surfaces in production in three distinct ways.

Failure Mode 1 — Karpenter Sees Nothing, Nodes Are Never Provisioned

Karpenter's entire operating model is reactive: it watches for Pending pods and provisions nodes to satisfy them. Volcano's entire operating model is the opposite: hold pods in a virtual queue before they become Pending, until the gang can be fully satisfied.

The result: when your cluster doesn't have enough existing GPU capacity for a full gang, Volcano holds the pods internally. They never enter Kubernetes Pending state. Karpenter, watching for Pending pods, sees nothing. No nodes are provisioned. The job waits forever — not because of a deadlock, but because the two systems are completely invisible to each other.

The fix requires setting minAvailable in the PodGroup at the Kueue admission layer (covered below) to force pods into Pending state after Volcano determines the gang can be satisfied with newly provisioned nodes — rather than requiring existing capacity only.

Failure Mode 2 — Karpenter Consolidation Kills a Running Training Job

This one is more expensive. Your training job is 3 hours in. Karpenter's consolidation logic notices that 2 of your GPU nodes are running at 55% GPU utilisation — natural for distributed training during checkpoint saves, data loading, and NCCL communication overhead. From Karpenter's perspective, those look like underutilised nodes that can be consolidated.

Karpenter cordons and drains one node. The worker pod on it is evicted. The training job's PodGroup drops below minMember. Volcano marks the entire PodGroup as Unknown. The PyTorch DDP process group fails its heartbeat. Every worker pod crashes.

No alert fires. Karpenter logs consolidated 1 node, saved $X/hr. Your training logs show a NCCL communication timeout. 3 GPU-hours are gone.

The fix is a dedicated Karpenter NodePool for GPU training with a hard consolidation policy:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu-training
spec:
  disruption:
    consolidationPolicy: WhenEmpty     # only consolidate truly empty nodes
    consolidateAfter: Never            # never consolidate occupied training nodes
    budgets:
    - nodes: "0"                       # zero disruptions during business hours
      schedule: "0 6 * * 1-5"         # Mon–Fri 6am IST
      duration: 16h
    - nodes: "10%"                     # allow limited cleanup overnight
      schedule: "0 22 * * *"
      duration: 8h
  template:
    metadata:
      labels:
        workload-type: gpu-training
    spec:
      requirements:
      - key: karpenter.k8s.aws/instance-family
        operator: In
        values: ["p4d", "p5", "g5"]
      - key: karpenter.sh/capacity-type
        operator: In
        values: ["on-demand"]          # never Spot for multi-hour training runs
      - key: topology.kubernetes.io/zone
        operator: In
        values: ["ap-south-1a"]        # single AZ for EFA/NVLink locality

consolidationPolicy: WhenEmpty is the critical field. It tells Karpenter: only reclaim a GPU node if it has zero running pods. Since training workers are always running during a job, Karpenter will not touch them. Your 3-hour run completes.

Failure Mode 3 — Topology Constraints Break Instance Diversity

Volcano's networkTopology can request that all pods in a PodGroup land within the same rack or Availability Zone — mandatory for NVLink bandwidth on p4d clusters and EFA-based collective communication on p5. Karpenter's instance diversity heuristics do the opposite: spread provisioning across AZs for resilience. These requirements are directly contradictory.

The resolution is constraining the training NodePool to a single AZ (shown above) alongside a matching PodGroup topology hint:

apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
  name: llama-finetune-v2
spec:
  minMember: 8
  networkTopology:
    mode: HardConstraint
    highBandwidthMultiNodeAllocation: 2   # require high-bandwidth interconnect for all pods

For inference NodePools, keep multi-AZ diversity. Only the training NodePool needs single-AZ pinning.

Kueue: The Admission Layer That Makes It All Manageable

If Volcano is the gang scheduling engine, Kueue is the traffic controller sitting above it that prevents the failure modes above from occurring in the first place.

Kueue operates at admission: jobs don't become pods until Kueue admits them, and Kueue only admits them when the cluster has confirmed quota. The key distinction:

• Kueue decides whether to admit a job right now

• Volcano decides how to schedule the admitted job atomically

• Karpenter decides what nodes to provision for it

Used together: Kueue holds a PyTorchJob in its queue until 8 GPU nodes are confirmed available (existing or provisionable). Kueue admits it → Volcano schedules atomically → Karpenter provisions any needed nodes → all 8 pods start simultaneously. No partial allocation. No deadlock. No mid-job eviction.

apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: training-cluster-queue
spec:
  namespaceSelector: {}
  cohort: ml-platform
  resourceGroups:
  - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
    flavors:
    - name: gpu-p5
      resources:
      - name: nvidia.com/gpu
        nominalQuota: 64        # 8 × p5.48xlarge, 8 GPUs each
        borrowingLimit: 16      # can borrow from inference queue when idle
  preemption:
    reclaimWithinCohort: LowerPriority
    withinClusterQueue: LowerPriority

The cohort field is the most powerful and least understood Kueue feature. Multiple ClusterQueues in the same cohort can lend unused quota to each other — your training queue borrows GPU capacity from an idle inference queue during off-hours, and the inference queue reclaims it with priority preemption when a new model deployment arrives. Multi-tenant GPU scheduling, without manual intervention.

Wiring It All Together: The Full PyTorchJob Spec

With Kueue, Volcano, and Karpenter configured, this is what a production training job looks like on EKS:

apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
  name: llama-finetune-v2
  namespace: ml-training
  labels:
    kueue.x-k8s.io/queue-name: training-local-queue    # Kueue admission
spec:
  pytorchReplicaSpecs:
    Master:
      replicas: 1
      restartPolicy: OnFailure
      template:
        spec:
          schedulerName: volcano                  # Volcano gang scheduling
          priorityClassName: training-high
          tolerations:
          - key: nvidia.com/gpu
            operator: Exists
            effect: NoSchedule
          containers:
          - name: pytorch
            image: 763104351884.dkr.ecr.ap-south-1.amazonaws.com/pytorch-training:2.3.0-gpu-py311-cu121-ubuntu20.04-ec2
            resources:
              limits:
                nvidia.com/gpu: "8"
                vpc.amazonaws.com/efa: "4"          # EFA devices for NCCL
            env:
            - name: NCCL_DEBUG
              value: "INFO"
            - name: NCCL_SOCKET_IFNAME
              value: "eth0"
            - name: FI_EFA_USE_DEVICE_RDMA
              value: "1"
    Worker:
      replicas: 7                          # 8 total (1 Master + 7 Workers)
      restartPolicy: OnFailure
      template:
        spec:
          schedulerName: volcano
          tolerations:
          - key: nvidia.com/gpu
            operator: Exists
            effect: NoSchedule
          containers:
          - name: pytorch
            image: 763104351884.dkr.ecr.ap-south-1.amazonaws.com/pytorch-training:2.3.0-gpu-py311-cu121-ubuntu20.04-ec2
            resources:
              limits:
                nvidia.com/gpu: "8"
                vpc.amazonaws.com/efa: "4" 

Three things worth calling out in this spec:

schedulerName: volcano — without this, the default scheduler handles your pods and you're back to the deadlock problem. This field is the switch.

vpc.amazonaws.com/efa — EFA (Elastic Fabric Adapter) devices for NCCL collective operations. On p4d and p5 instances, NCCL over EFA is 4–8x faster than NCCL over standard Ethernet. The EFA device plugin must be installed separately; it is not part of the NVIDIA device plugin.[aws.amazon]​

restartPolicy: OnFailure — with Kueue managing admission, a failed job re-enters the Kueue queue automatically rather than running with whatever partial capacity is available. The job waits for a full gang again.

Installation Order and Namespace Scoping

Installation order matters and getting it wrong produces errors that are difficult to trace.

# 1. Karpenter — nodes must exist before schedulers use them
helm install karpenter oci://public.ecr.aws/karpenter/karpenter \
  --namespace karpenter --create-namespace \
  --version "1.3.0"

# 2. Volcano — batch scheduler, replaces default for annotated pods only
helm install volcano volcano-sh/volcano \
  --namespace volcano-system --create-namespace \
  --version "1.10.0"

# 3. Kueue — admission webhook, runs upstream of everything
helm install kueue oci://registry.k8s.io/kueue/charts/kueue \
  --namespace kueue-system --create-namespace \
  --version "0.10.0"

# 4. KubeFlow Training Operator — PyTorchJob CRD
kubectl apply -k "github.com/kubeflow/training-operator/manifests/overlays/standalone"

Critical: scope Volcano to GPU namespaces only. Running Volcano as the cluster-wide default scheduler breaks standard Deployments and Services that expect default scheduler behaviour — specifically, Volcano's bin-packing policy removes the spread-across-nodes behaviour that replicas rely on.

# Label your ML namespaces — jobs in these namespaces use schedulerName: volcano
kubectl label namespace ml-training volcano.sh/queue-name=training-queue
kubectl label namespace ml-inference volcano.sh/queue-name=inference-queue

# Everything else (kube-system, monitoring, etc.) uses the default scheduler
# No label = no Volcano involvement

Volcano vs Kueue vs Native Kubernetes — Where Things Stand in 2026

Kubernetes v1.35 (December 2025) shipped Workload-Aware Scheduling — native gang scheduling upstream. Worth knowing where everything stands:

My take: Kueue + Volcano together is the right production stack for 2026. Kueue handles the multi-tenant admission and quota story. Volcano handles the scheduling atomicity. Karpenter handles the compute. Native Kubernetes gang scheduling is worth watching but is alpha, untested under real multi-tenant load, and lacks the queue management that Kueue provides.

The Three Rules That Prevent All Three Failure Modes

After everything above, the configuration changes come down to three rules:

Rule 1: Always consolidationPolicy: WhenEmpty on GPU training NodePools. Never let Karpenter make utilisation-based eviction decisions on training nodes. GPU workloads have natural utilisation valleys that look idle but aren't.

Rule 2: Always set minMember equal to total replicas, never a subset. Setting minMember: 4 when your job needs 8 workers means Volcano schedules 4 pods and the job hangs waiting for the rest. The whole point of gang scheduling is atomicity — partial gangs defeat the purpose.

Rule 3: Scope Volcano to ML namespaces only. One schedulerName: volcano annotation in a non-ML deployment can break replica spreading across nodes cluster-wide. Namespace labeling keeps blast radius contained.

Before and After

The deadlock was never your code. It was never a Kubernetes bug. It was three systems — each working correctly in isolation — that had never been configured to work together.

What the book actually is

A production blueprint that treats GenAI workloads on Kubernetes as living systems you operate and pay for — not demos you deploy and forget.​

A plain account of tradeoffs: when to use HPA, VPA, KEDA, or Karpenter — and what each scaling decision costs in GPU spend, cold start time, and operational attention.​

A reminder that observability isn't a finishing touch — it's how you see what your AI workload is doing in a real cluster when users are waiting and GPUs are expensive.​

How it explains the hard parts in simple words

Containers are the foundation, not the destination. The book earns the right to talk about GenAI by first explaining why containers matter for AI workloads — reproducibility, portability, and the ability to ship a model and its dependencies together without surprises.​

Kubernetes is a fit for GenAI for specific reasons. Scheduling GPU nodes, managing heterogeneous workloads, autoscaling based on custom metrics, isolating model serving from training — these aren't generic Kubernetes features, they're exactly what GenAI infrastructure needs.​

Scaling means different things at different layers. HPA scales on CPU and memory. VPA right-sizes requests over time. KEDA scales on custom signals like queue depth. Karpenter provisions the right node for the right workload. The book explains when each one matters and why using the wrong one at the wrong layer costs you quietly.​

GPU optimization is where cost lives. MIG partitions a single GPU into isolated slices for smaller models. MPS allows multiple processes to share GPU compute time. Fractional allocation and Spot Instances cut spend dramatically — but only if health checks, resource limits, and interruption handling are boring and reliable first.​

Observability must be tied to the AI workload, not just the node. Prometheus and Grafana tell you the cluster is healthy. NVIDIA DCGM tells you the GPU is healthy. Neither tells you if your model is drifting, your RAG pipeline is degrading, or your latency is crossing a threshold users actually feel. The book covers all three layers.​

Security is a depth problem, not a checkbox. Supply chain, host, network, runtime, secrets management, RBAC, service meshes, IAM — each layer adds protection, and the book covers them in order from the outside in rather than as a list of features to enable.​

Delivery should be calm and reversible. GitOps and Argo CD make GenAI model deployments pull-based and auditable so rollbacks are quick, drift is visible, and the team's attention stays on users rather than deployment state.​

The production stories that teach

The book doesn't skip the uncomfortable parts — GPU nodes that sit idle because autoscaling was tuned for CPU workloads, RAG pipelines that degrade silently because nobody was watching embedding latency, and cost reports that arrive as surprises because Kubecost wasn't set up until after the first bill. Each problem leads to a decision, a configuration change, or a new guardrail — and those stack into a system that fails more gently and costs less over time.​

Two ideas that quietly run through the book

Cost is a first-class concern, not an afterthought. GPUs are expensive. Spot Instances help. Karpenter helps. Kubecost makes the bill readable. But none of that works if workloads aren't sized, scheduled, and monitored with cost in mind from the beginning.​

Foundation first, then production. The book insists on understanding what the AI workload needs before touching Kubernetes — model behavior, resource patterns, latency requirements, scaling signals. Without that foundation, you're copying YAML and hoping for the best.​

What this looks like in practice

• Set up observability before scaling — instrument GPU metrics, model latency, and RAG pipeline health before adding nodes, or you're scaling blindly

• Define scaling signals per workload — CPU-based HPA makes no sense for GPU-bound inference; use KEDA with the right custom metric from the start

• Enable MIG or MPS before buying more GPUs — fractional allocation often doubles effective capacity before any new hardware is needed

• Treat Spot Instances as default with proper interruption handling — the savings are real, but only if your workloads are stateless and your health probes are tight

• GitOps your model deployments — version-controlled, pull-based, auditable deploys make rollbacks safe and fast when a new model version behaves unexpectedly

• Run Kubecost from day one — cost surprises in GPU infrastructure arrive fast and large; visibility early prevents decisions you can't undo​

Who will get the most out of it

Engineers who know Kubernetes and want to understand what running real GenAI workloads on it actually demands — beyond the YAML.​

DevOps and platform teams who are being asked to support LLM serving, fine-tuning pipelines, and RAG systems and need a production-grade mental model fast.​

Solutions architects and engineering leads who want to understand how infrastructure choices show up as cost, reliability, and team attention — not just technical specs.​

What felt different to read

The writing is straight: here's the workload, here's what it needs, here's what Kubernetes gives you, here's what it costs if you get it wrong. It never pretends that GPU infrastructure is simple or that "just use managed Kubernetes" answers the hard questions. It also doesn't pick a side in the EKS vs. self-managed debate — it explains when the cloud bill becomes a steering wheel and when owning more of the stack makes sense.​

What this book pushed me to change

• Define GPU resource requests and limits explicitly before deploying any model — defaults will bankrupt you quietly

• Set up NVIDIA DCGM monitoring alongside Prometheus from day one, not after the first GPU incident

• Think about MIG partitioning early for multi-tenant model serving instead of over-provisioning nodes

• Treat Karpenter node provisioning configuration as a first-class engineering artifact, not an afterthought

• Move model deployment pipelines to GitOps so rollbacks are a command, not a conversation

• Schedule a weekly GPU utilization review — idle capacity is invisible until the invoice arrives​

Verdict

If you want a clean Kubernetes tutorial or a standalone GenAI primer, look elsewhere. If you want an honest, production-grade guide to running GenAI workloads on Kubernetes — covering the full journey from first container to cost-optimized, observable, secure, multi-region production — this is worth your full day.

It treats the Kubernetes + GenAI stack with the seriousness it deserves: expensive, operationally complex, and worth getting right before you scale.​

📖 Available at packtpub.com

I've been running GPU inference workloads on EKS for a while now. You get to a point where you've tuned everything you can see — Karpenter provisions nodes fast, the control plane is healthy, CUDA initialises cleanly. And yet, there is this wall you keep hitting during scale-out events: a gap of six to eight minutes between "node is Ready" and "pod is serving requests."

For a long time I accepted it. The container image is 40–60 GB. Of course it takes time. Then I started looking at exactly where those minutes were going, and the answer was less obvious than I expected.

The Real Anatomy of a GPU Cold Start

When Karpenter provisions a fresh GPU node and your pod gets scheduled onto it, there are five distinct phases:

Everything except the image pull phase is largely fixed. Karpenter + EC2 Fleet provisioning takes ~90 seconds. Node bootstrapping and joining the cluster takes ~45 seconds. CUDA initialisation and model loading are workload-dependent but relatively stable. The only phase with significant variance — and significant opportunity — is the image pull.

For a 10 GB vLLM AWS Deep Learning Container, the standard containerd pull takes 1 minute 52 seconds. For a 40–60 GB production ML image with bundled model weights, that number climbs to 6–8 minutes. AWS's own data puts image pull time at more than 75% of total container startup time for large images. You are paying for a p4d.24xlarge — roughly ₹900/hour — while it downloads layers.​

This is the problem SOCI Snapshotter solves. But the way most people first encounter it leads them to the wrong mode for GPU workloads, and makes things worse. Let me explain.

SOCI: Two Modes, One Right Answer for GPU

SOCI (Seekable OCI) is a containerd snapshotter plugin open-sourced by AWS. It has two distinct operating modes that are easy to conflate:​

Why Lazy Loading is the wrong mode for GPU containers: The premise of lazy loading is that your container only needs a fraction of its image at startup, so why download everything? This works beautifully for a Node.js API server. It works terribly for a vLLM or Triton container. The moment your container process starts, it initialises the CUDA runtime, loads cuDNN, loads NCCL, loads your model weights — all at once, all immediately. With lazy loading enabled, the container process starts, then stalls as the SOCI snapshotter frantically fetches every layer on-demand in response to CUDA's filesystem reads. You end up downloading everything anyway, but through a slow on-demand fetch path with no parallelism, instead of the regular pull. In practice it is often slower than standard containerd.​

Parallel Pull takes the opposite approach: download everything upfront, but transform how the download happens. This is what you want for GPU workloads.​

How Parallel Pull Actually Works

Standard containerd has two sequential bottlenecks:​

Phase 1 — Layer Fetch: Uses a single TCP connection per layer, downloading layers one at a time. Even with a 25 Gbps network interface (standard on g5, p4d, p5 instances), a single connection cannot saturate the bandwidth for a large layer. The rest of your network capacity sits idle.

Phase 2 — Layer Unpack: Takes each downloaded layer and decompresses + extracts it to disk. Serial, one layer at a time. On a 64-core p4d.24xlarge, 63 cores wait while one core unpacks a 5 GB CUDA layer.

SOCI Parallel Pull attacks both phases simultaneously:

• Download: Splits each layer into configurable chunks (concurrent_download_chunk_size: 16mb) and downloads them via concurrent HTTP range requests. Multiple layers download in parallel. Available bandwidth is actually utilised.

• Unpack: Multiple layers are decompressed and extracted concurrently, scaled to the number of available CPU cores. The I/O-bound serial unpack becomes a CPU-optimised parallel operation.​

The result on a real production benchmark — 10 GB vLLM AWS DLC on m6i.8xlarge with EBS optimised at 16K IOPS and 1000 MiB/s throughput:

112.9 seconds → 45.1 seconds. A 60% reduction in pull time with correctly tuned configuration. Grab Engineering reported a similar result in production: 60 seconds to 24 seconds after tuning, a 60% improvement. The untuned default config delivers less — roughly 42% improvement — which is why configuration matters.

The EKS-Specific Setup Nobody Documents End-to-End

SOCI Parallel Pull is not enabled by default on EKS AMIs as of early 2026. Recent versions of Amazon Linux 2023 and Bottlerocket EKS-optimised AMIs have it built in, but you must configure it via userData. Here is the complete path for each node provisioning model.​

Option A: Karpenter EC2NodeClass (Recommended)

This is the cleanest approach. Configure SOCI once in your EC2NodeClass userData, and every Karpenter-provisioned node in that class gets it automatically.​

apiVersion: karpenter.k8s.aws/v1 kind: EC2NodeClass metadata: name: gpu-soci spec: role: KarpenterNodeRole-my-cluster instanceStorePolicy: RAID0 # bind NVMe if available amiSelectorTerms: - alias: al2023@latest instanceSelectorTerms: - name: g5.2xlarge - name: g5.4xlarge - name: p4d.24xlarge userData: | MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="//"

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: gpu-soci
spec:
  role: KarpenterNodeRole-my-cluster
  instanceStorePolicy: RAID0          # bind NVMe if available
  amiSelectorTerms:
    - alias: al2023@latest
  instanceSelectorTerms:
    - name: g5.2xlarge
    - name: g5.4xlarge
    - name: p4d.24xlarge
  userData: |
    MIME-Version: 1.0
    Content-Type: multipart/mixed; boundary="//"

    --//
    Content-Type: text/x-shellscript; charset="us-ascii"

    #!/bin/bash
    # Download SOCI snapshotter v0.11.0+
    ARCH=$(uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/')
    SOCI_VERSION="0.11.0"
    curl -Lo /tmp/soci-snapshotter.tar.gz \
      "https://github.com/awslabs/soci-snapshotter/releases/download/v\({SOCI_VERSION}/soci-snapshotter-\){SOCI_VERSION}-linux-${ARCH}.tar.gz"
    tar -xzf /tmp/soci-snapshotter.tar.gz -C /usr/local/bin soci soci-snapshotter-grpc

    # Configure SOCI
    mkdir -p /etc/soci-snapshotter-grpc
    cat > /etc/soci-snapshotter-grpc/config.toml << 'EOF'
    [snapshotter]
    allow_invalid_mounpoints = false

    [pull]
    max_concurrent_downloads_per_image = 10
    concurrent_download_chunk_size = "16mb"
    max_concurrent_unpacks_per_image = 10
    discard_unpacked_layers = true
    EOF

    # Systemd service
    cat > /etc/systemd/system/soci-snapshotter.service << 'EOF'
    [Unit]
    Description=SOCI Snapshotter
    After=network.target
    [Service]
    ExecStart=/usr/local/bin/soci-snapshotter-grpc --config /etc/soci-snapshotter-grpc/config.toml
    Restart=always
    [Install]
    WantedBy=multi-user.target
    EOF

    systemctl daemon-reload
    systemctl enable --now soci-snapshotter

    --//
    Content-Type: application/node.eks.aws

    apiVersion: node.eks.aws/v1alpha1
    kind: NodeConfig
    spec:
      kubelet:
        config:
          imageServiceEndpoint: unix:///run/soci-snapshotter-grpc/soci-snapshotter-grpc.sock
      containerd:
        config: |
          [proxy_plugins.soci]
            type    = "snapshot"
            address = "/run/soci-snapshotter-grpc/soci-snapshotter-grpc.sock"
          [proxy_plugins.soci.exports]
            root = "/var/lib/containerd/io.containerd.snapshotter.v1.soci"
          [plugins."io.containerd.grpc.v1.cri".containerd]
            snapshotter              = "soci"
            disable_snapshot_annotations = false
            discard_unpacked_layers  = false
    --//

Option B: Bottlerocket (Simpler)

Bottlerocket has native SOCI Parallel Pull configuration baked in — no shell script, no binary download, no systemd wrangling.​

[settings.container-runtime]
snapshotter = "soci"

[settings.container-runtime-plugins.soci-snapshotter]
pull-mode = "parallel-pull-unpack"

[settings.container-runtime-plugins.soci-snapshotter.parallel-pull-unpack]
max-concurrent-downloads-per-image = 10
concurrent-download-chunk-size     = "16mb"
max-concurrent-unpacks-per-image   = 10
discard-unpacked-layers            = true

# Bind SOCI root dir to NVMe instance store on g5/p4d/p5
[settings.bootstrap-commands.k8s-ephemeral-storage]
commands = [
    ["apiclient", "ephemeral-storage", "init"],
    ["apiclient", "ephemeral-storage", "bind",
     "--dirs", "/var/lib/containerd", "/var/lib/kubelet",
               "/var/log/pods", "/var/lib/soci-snapshotter"]
]
essential = true
mode      = "always"

Bottlerocket also shows slightly better benchmark results because it uses a different decompression library (zstd vs gzip by default). If you are on AL2023 and want parity, use zstd-compressed images in ECR.​

Option C: Managed Node Groups via Launch Template

For teams not using Karpenter, you need a custom launch template. The userData content is identical to Option A — wrap it in base64 and attach it to your MNG launch template configuration.​

# Create the launch template with SOCI userData
aws ec2 create-launch-template \
  --launch-template-name soci-gpu-lt \
  --launch-template-data '{
    "ImageId": "'"$AMI_ID"'",
    "InstanceType": "g5.2xlarge",
    "UserData": "'"$(base64 -w0 soci-userdata.txt)"'"
  }'

# Reference it in your managed node group
eksctl create nodegroup \
  --cluster $CLUSTER_NAME \
  --name gpu-soci-ng \
  --launch-template-id $LAUNCH_TEMPLATE_ID

The Tuning Parameters That Actually Matter

The default SOCI configuration is deliberately conservative. Leaving defaults in place delivers roughly half the possible improvement. Here is what each parameter controls and what to set for ECR-hosted ML images:

Grab Engineering's production finding is instructive: tuning just these four parameters reduced their image pull from 60 seconds to 24 seconds — the difference between "configured" and "optimised." The default numbers exist for safety at low concurrency; they are not right for high-bandwidth, multi-core GPU instances.​

The storage constraint is often the overlooked bottleneck. SOCI Parallel Pull trades memory efficiency for disk throughput — it writes directly to disk (no in-memory buffering). If your EBS volume is a default gp3 with 3,000 IOPS and 125 MiB/s throughput, the storage layer will become your bottleneck before the network does. For SOCI to run at full speed, configure your EBS volume at:​

Throughput: 1000 MiB/s
IOPS:       16,000
Type:       gp3

For g5, p4d, and p5 instances that include NVMe instance store — bind SOCI's root directory (/var/lib/soci-snapshotter) to the NVMe volume. The read/write speed of NVMe removes the storage bottleneck entirely and is the configuration that produces the best benchmark numbers.

The Dockerfile Layer Ordering Question

This is where things get specific to ML images, and where I see teams get tripped up.

In Lazy Loading mode (which we've established is wrong for GPU), Dockerfile layer ordering matters enormously — SOCI fetches layers on-demand as files are accessed, so placing frequently-accessed layers early means they're resolved from local cache faster. For Parallel Pull mode, layer ordering has no impact on startup time — everything is pulled before the container starts.

However, layer ordering still matters for build performance and cache efficiency. For ML images, the optimal Dockerfile structure is:

# Layer 1: Base OS — rarely changes, always cached
FROM nvidia/cuda:12.4.1-cudnn-devel-ubuntu22.04

# Layer 2: System packages — changes infrequently
RUN apt-get update && apt-get install -y python3-pip git && rm -rf /var/lib/apt/lists/*

# Layer 3: Python dependencies — changes on library updates
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Layer 4: Application code — changes frequently, keep it thin
COPY src/ /app/src/

# Layer 5: Model weights — only here if you bake them in
# (Consider EFS or S3 CSI mount instead — keeps this layer manageable)
COPY model/ /app/model/

The reasoning: SOCI (and ECR's content-addressable storage) can skip re-pulling layers that haven't changed. If your model weights are in the last layer, a code change in Layer 4 doesn't invalidate the weights cache. If your weights are Layer 2, every pip install forces a full re-pull of the largest part of your image.

The stronger recommendation: Don't bake large model weights into your container image at all. Mount them via EFS CSI (shared across pods, multi-AZ) or S3 CSI (Mountpoint for S3, streaming directly from S3 at startup). Your container image drops to 10–15 GB of framework + dependencies, SOCI's parallel pull handles it in under 45 seconds, and your model weights are independently versioned and not duplicated per ECR tag.

Scoping SOCI to GPU Workloads Only

You don't want every pod on the cluster going through the SOCI snapshotter. Lightweight API pods, Argo CD workers, CoreDNS — they have tiny images and the standard overlayfs snapshotter handles them fine. Use a Kubernetes RuntimeClass to route only GPU workloads through SOCI:​

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: soci
handler: soci   # matches the proxy plugin name in containerd config
scheduling:
  nodeClassification:
    nodeSelector:
      eks.amazonaws.com/nodegroup: gpu-soci-ng  # only on SOCI-enabled nodes

Then reference it explicitly in your GPU Deployment:

spec:
  template:
    spec:
      runtimeClassName: soci          # only this pod uses SOCI snapshotter
      containers:
      - name: vllm
        image: 763104351884.dkr.ecr.us-east-1.amazonaws.com/vllm:0.9-gpu-py312-ec2
        resources:
          limits:
            nvidia.com/gpu: "1"

Standard pods on SOCI-enabled nodes continue using the default overlayfs snapshotter. There is no performance regression on non-GPU workloads, and you avoid the overhead of running every image pull through the SOCI daemon.

Validating It Is Working

After deployment, verify the snapshotter is active and pull times have changed:

# Confirm SOCI snapshotter is running on the node
kubectl exec -n kube-system ds/aws-node -- \
  crictl info | grep -i snapshotter

# Expected output:
# "snapshotter": "soci"

# Check actual pull time from pod events
kubectl describe pod vllm-<pod-id> | grep -E "Pulled|Pulling"

# Normal  Pulling   3m29s  kubelet  Pulling image "...vllm:0.9-gpu-py312-ec2"
# Normal  Pulled    2m44s  kubelet  Successfully pulled image in 45.121s

# Check SOCI daemon logs for pull activity
journalctl -u soci-snapshotter -f --since "5 minutes ago"

If your Pulled event still shows 90+ seconds after configuring SOCI, the two most common causes are: (1) the containerd config was not applied correctly — disable_snapshot_annotations = false is required for SOCI to intercept pulls; and (2) the EBS volume is the bottleneck — check iostat during a pull and confirm you are not at IOPS or throughput limits.

What This Changes Operationally

The 60% pull time reduction is the headline number, but the downstream effects matter more for production operations:

Autoscaling responsiveness: Karpenter's ability to respond to traffic spikes depends on how fast a new GPU pod can become ready. Cutting the cold start from 11 minutes to 5.5 minutes means your inference fleet responds to demand surges more than twice as fast. For an LLM API serving users, that difference is felt.

Spot Interruption recovery: On Spot instances, your g5.2xlarge gets interrupted. Karpenter provisions a replacement. With standard pull, that replacement is unavailable for 11+ minutes. With SOCI, it's back in 5.5 minutes. At scale, this is the difference between a noticeable degradation and a transparent rebalance.

Multi-model deployments: Teams running multiple model variants (7B, 13B, 70B) on the same cluster see compounding gains. A node serving a fresh model that previously required 3 separate 6-minute pulls — for three different models — now needs 3 pulls at 2.5 minutes each. Cluster time-to-ready drops by over an hour in a day of regular operations.

Cost: A g5.48xlarge costs approximately ₹2,200/hour on-demand. Eight minutes of idle time during image pull per cold start, across a 10-node fleet cycling through multiple warm-ups per day, is not a rounding error.

The One-Line Summary

SOCI Parallel Pull is already built into recent Amazon Linux 2023 and Bottlerocket EKS-optimised AMIs. It is not the default. It requires four configuration parameters, an optimised EBS volume or NVMe binding, and a RuntimeClass to scope it correctly to GPU workloads. It cuts image pull time by 60% on real production images. There is no reason not to have it on every GPU node in your cluster.

The complete Karpenter blueprint with SOCI configuration is available in the AI on EKS repository from AWS Labs.​

This fundamentally changes the conversation around GPU workloads on ECS. You now have two viable paths: the traditional self-managed EC2 approach where you control everything, and the newer managed instances approach where AWS handles the infrastructure lifecycle while you focus on your containers. Neither is universally better—it depends on your specific requirements, team capabilities, and tolerance for operational overhead.

In this post, I'll walk through both approaches, show you the actual configurations, and help you decide which makes sense for your workload. I'm not going to sugarcoat the complexity—running GPU inference in production has real operational costs regardless of which path you choose.

The Three Compute Options for ECS (and Why It Matters)

Before we dive into implementations, let's clarify the current ECS compute landscape:

Fargate: Fully serverless, no server management, but no GPU support. If your workload doesn't need GPUs, this is still the simplest option.

ECS Managed Instances: AWS provisions, patches, and replaces EC2 instances for you. You get to specify instance families (including GPU types), use Reserved Instances or Spot, but you can't SSH into hosts or use custom AMIs. This launched in September 2025 and became generally available across all commercial regions in October 2025.

Traditional ECS on EC2: You manage everything—AMIs, patching, Auto Scaling Groups, capacity providers. Maximum control, maximum operational burden.

For GPU workloads specifically, Fargate isn't an option. You're choosing between Managed Instances (less operational work, some constraints) and self-managed EC2 (full control, more operational burden).

When to Choose Which Approach

Let me be direct about this because the documentation doesn't always make it clear:

Use ECS Managed Instances when:

• You want AWS to handle instance patching and lifecycle management

• You're comfortable with instances being replaced on AWS's schedule

• You don't need custom AMIs or host-level customizations

• You want to leverage Reserved Instances or Spot pricing without managing ASGs yourself

• Your team doesn't have dedicated platform engineering resources

Use Traditional EC2 when:

• You need custom AMIs with specific kernel modules or configurations

• You require SSH access for debugging or custom tooling

• You have compliance requirements that mandate specific host configurations

• You want complete control over when instances are replaced or updated

• You have the engineering capacity to manage infrastructure lifecycle

I've used both approaches, and honestly, for most ML inference workloads, Managed Instances is now my default choice. The operational relief is real, and the constraints rarely matter for containerized GPU workloads. But if you're doing something that requires kernel-level tweaking or highly specialized host configurations, you'll need traditional EC2.

Approach 1: ECS Managed Instances (The New Way)

Let's start with the newer approach because it's simpler and likely what most teams should use going forward.

Architecture Overview

With Managed Instances, you don't create launch templates or Auto Scaling Groups yourself. Instead, you define instance requirements in a capacity provider, and AWS handles the rest. The instances come pre-configured with NVIDIA drivers and CUDA toolkit, which eliminates one of the biggest pain points of the traditional approach.

AWS automatically provisions instances that match your requirements, patches them regularly, and replaces them on a managed schedule. You get infrastructure that stays current without building patching automation.

Step 1: Create the Cluster

resource "aws_ecs_cluster" "gpu_managed" {
  name = "gpu-inference-managed"

  setting {
    name  = "containerInsights"
    value = "enabled"
  }
}

Nothing special here—it's just a standard ECS cluster. The magic happens in the capacity provider configuration.

Step 2: Define the Managed Capacity Provider

This is where you specify GPU requirements:

resource "aws_ecs_capacity_provider" "gpu_managed" {
  name = "gpu-managed-capacity-provider"

  auto_scaling_group_provider {
    auto_scaling_group_arn = aws_autoscaling_group.ecs_managed_gpu.arn
    managed_scaling {
      maximum_scaling_step_size = 2
      minimum_scaling_step_size = 1
      status                    = "ENABLED"
      target_capacity           = 100
      instance_warmup_period    = 300
    }
    managed_termination_protection = "DISABLED"
    managed_draining              = "ENABLED"
  }
}

Wait—there's still an Auto Scaling Group here? Yes, but it's configured differently. With Managed Instances, the ASG uses a special configuration that tells ECS to manage the instances:

resource "aws_launch_template" "ecs_managed_gpu" {
  name_prefix   = "ecs-managed-gpu-"
  image_id      = data.aws_ssm_parameter.ecs_managed_ami.value
  instance_type = "g4dn.xlarge"

  iam_instance_profile {
    arn = aws_iam_instance_profile.ecs_instance.arn
  }

  vpc_security_group_ids = [aws_security_group.ecs_gpu.id]

  tag_specifications {
    resource_type = "instance"
    tags = {
      Name                 = "ecs-managed-gpu"
      AmazonECSManaged     = true  # This is critical
    }
  }

  # Instance requirements for GPU selection
  instance_requirements {
    accelerator_types         = ["gpu"]
    accelerator_count {
      min = 1
      max = 1
    }
    accelerator_manufacturers = ["nvidia"]
    
    # Can also specify allowed instance types
    allowed_instance_types = ["g4dn.xlarge", "g4dn.2xlarge"]
  }

  # No user data needed! AWS handles GPU configuration
}

The key differences from traditional EC2:

• The AmazonECSManaged = true tag tells AWS to manage this instance

• instance_requirements lets you specify GPU needs declaratively

• No user data script needed—AWS pre-configures NVIDIA drivers and ECS agent GPU support

• You can specify a range of instance types and let AWS choose the most cost-effective option available

The instance_requirements block is powerful. Instead of hard-coding g4dn.xlarge, you can say "I need 1 NVIDIA GPU" and let AWS select from available instance types. This helps with Spot availability and cost optimization.

Step 3: Task Definition (Unchanged)

The task definition is identical to the traditional approach:

resource "aws_ecs_task_definition" "gpu_inference_managed" {
  family                   = "llm-inference-managed"
  requires_compatibilities = ["EC2"]
  network_mode             = "awsvpc"
  cpu                      = "4096"
  memory                   = "16384"
  execution_role_arn       = aws_iam_role.task_execution.arn
  task_role_arn            = aws_iam_role.task.arn

  container_definitions = jsonencode([
    {
      name      = "inference-container"
      image     = "nvidia/cuda:12.1.0-runtime-ubuntu22.04"
      essential = true

      command = [
        "/bin/bash",
        "-c",
        "nvidia-smi && python3 /app/serve_model.py"
      ]

      environment = [
        {
          name  = "NVIDIA_VISIBLE_DEVICES"
          value = "all"
        },
        {
          name  = "NVIDIA_DRIVER_CAPABILITIES"
          value = "compute,utility"
        }
      ]

      portMappings = [
        {
          containerPort = 8000
          protocol      = "tcp"
        }
      ]

      logConfiguration = {
        logDriver = "awslogs"
        options = {
          "awslogs-group"         = "/ecs/gpu-inference-managed"
          "awslogs-region"        = data.aws_region.current.name
          "awslogs-stream-prefix" = "gpu"
        }
      }

      resourceRequirements = [
        {
          type  = "GPU"
          value = "1"
        }
      ]
    }
  ])
}

This is exactly the same as with traditional EC2. The task definition doesn't care whether it's running on managed or self-managed instances.

Step 4: Create the Service

resource "aws_ecs_service" "gpu_inference_managed" {
  name            = "llm-inference-managed"
  cluster         = aws_ecs_cluster.gpu_managed.id
  task_definition = aws_ecs_task_definition.gpu_inference_managed.arn
  desired_count   = 2

  capacity_provider_strategy {
    capacity_provider = aws_ecs_capacity_provider.gpu_managed.name
    weight            = 1
  }

  network_configuration {
    subnets          = var.private_subnet_ids
    security_groups  = [aws_security_group.inference_tasks.id]
    assign_public_ip = false
  }

  load_balancer {
    target_group_arn = aws_lb_target_group.inference.arn
    container_name   = "inference-container"
    container_port   = 8000
  }

  deployment_configuration {
    maximum_percent         = 200
    minimum_healthy_percent = 100
  }
}

Again, no changes needed here. The beauty of ECS Managed Instances is that the task and service definitions remain unchanged—you're just swapping out how the underlying infrastructure is managed.

What AWS Actually Manages

Here's what happens behind the scenes with Managed Instances:

• AWS provisions instances that match your instance_requirements

• Instances are automatically patched and updated on a regular schedule

• When instances need replacement (for security patches or maintenance), AWS drains tasks and replaces them gracefully

• You can't SSH into these instances—if you need debugging, use CloudWatch Logs and Container Insights

• Custom AMIs aren't supported—AWS uses its own managed AMI with pre-configured GPU drivers

The trade-off is clear: you give up some control (no SSH, no custom AMIs) in exchange for AWS handling operational toil.

Approach 2: Traditional Self-Managed EC2 (The Classic Way)

Now let's look at the traditional approach. This is what we've been using before December 2025, and it's still the right choice when you need full control.

Architecture Overview

You're responsible for everything: choosing the AMI, configuring user data to enable GPU support, managing the Auto Scaling Group, handling patching and updates. ECS just provides the orchestration layer on top of infrastructure you own.

Step 1: Select and Configure the AMI

You need a GPU-optimized ECS AMI with NVIDIA drivers pre-installed:

# Get the latest GPU-optimized ECS AMI
aws ssm get-parameters \
  --names /aws/service/ecs/optimized-ami/amazon-linux-2/gpu/recommended \
  --region us-east-1 \
  --query 'Parameters[0].Value' \
  --output text | jq -r '.image_id'

This returns something like ami-0c02fb55956c7d316. The AMI includes NVIDIA drivers, CUDA toolkit, and Docker GPU runtime pre-configured.

Alternatively, you could use AWS Deep Learning AMIs, which include multiple ML frameworks, but they're larger and include tools you might not need for production inference.

Step 2: Create the Launch Template with User Data

This is the critical part—the user data script that enables GPU support in the ECS agent:

resource "aws_launch_template" "gpu_traditional" {
  name_prefix   = "gpu-ecs-traditional-"
  image_id      = "ami-0c02fb55956c7d316"  # GPU-optimized ECS AMI
  instance_type = "g4dn.xlarge"

  iam_instance_profile {
    name = aws_iam_instance_profile.ecs_instance.name
  }

  vpc_security_group_ids = [aws_security_group.ecs_gpu.id]

  user_data = base64encode(<<-EOF
    #!/bin/bash
    # Configure ECS agent for GPU support
    cat >> /etc/ecs/ecs.config <<ECS_CONFIG
    ECS_CLUSTER=${aws_ecs_cluster.gpu_traditional.name}
    ECS_ENABLE_GPU_SUPPORT=true
    ECS_ENABLE_TASK_IAM_ROLE=true
    ECS_ENABLE_CONTAINER_METADATA=true
    ECS_ENABLE_SPOT_INSTANCE_DRAINING=true
    ECS_LOGLEVEL=info
    ECS_CONFIG

    # Verify NVIDIA drivers are loaded
    nvidia-smi

    # Additional customizations you might need
    # Install CloudWatch agent
    wget https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm
    rpm -U ./amazon-cloudwatch-agent.rpm

    # Custom monitoring or logging setup
    # Your custom scripts here
  EOF
  )

  tag_specifications {
    resource_type = "instance"
    tags = {
      Name    = "ecs-gpu-traditional"
      Cluster = aws_ecs_cluster.gpu_traditional.name
    }
  }
}

The ECS_ENABLE_GPU_SUPPORT=true line is absolutely critical. Without it, even though the instance has GPUs, the ECS agent won't expose them for task scheduling. I've debugged this multiple times—tasks just sit in PENDING state because ECS doesn't see any GPU capacity.

The beauty of this approach is that you can add whatever customizations you need in the user data script. Need custom monitoring? SSH keys for debugging? Specific kernel parameters? You can configure it all here.

Step 3: Create the Auto Scaling Group

resource "aws_autoscaling_group" "gpu_traditional" {
  name                = "gpu-ecs-asg-traditional"
  desired_capacity    = 1
  max_size            = 5
  min_size            = 0
  vpc_zone_identifier = var.private_subnet_ids
  health_check_type   = "EC2"
  health_check_grace_period = 300

  launch_template {
    id      = aws_launch_template.gpu_traditional.id
    version = "$Latest"
  }

  tag {
    key                 = "Name"
    value               = "ecs-gpu-instance"
    propagate_at_launch = true
  }

  lifecycle {
    ignore_changes = [desired_capacity]
  }
}

I typically start with min_size = 0 for cost optimization—if there are no GPU tasks running, the cluster scales down to zero instances. For production workloads with consistent traffic, set min_size = 1 or higher to avoid cold start delays.

Step 4: Create the Capacity Provider

resource "aws_ecs_capacity_provider" "gpu_traditional" {
  name = "gpu-traditional-capacity-provider"

  auto_scaling_group_provider {
    auto_scaling_group_arn         = aws_autoscaling_group.gpu_traditional.arn
    managed_termination_protection = "ENABLED"

    managed_scaling {
      maximum_scaling_step_size = 2
      minimum_scaling_step_size = 1
      status                    = "ENABLED"
      target_capacity           = 100
    }
  }
}

resource "aws_ecs_cluster" "gpu_traditional" {
  name = "gpu-inference-traditional"
}

resource "aws_ecs_cluster_capacity_providers" "gpu_traditional" {
  cluster_name = aws_ecs_cluster.gpu_traditional.name

  capacity_providers = [aws_ecs_capacity_provider.gpu_traditional.name]

  default_capacity_provider_strategy {
    capacity_provider = aws_ecs_capacity_provider.gpu_traditional.name
    weight            = 1
  }
}

The target_capacity = 100 means ECS tries to maintain 100% available capacity. When GPU tasks are pending, the capacity provider triggers the ASG to scale up.

Task Definition and Service

These are identical to the Managed Instances approach—I won't repeat the code. The task definition specifies resourceRequirements with type = "GPU" and value = "1", and the service uses the capacity provider.

What You're Responsible For

With traditional EC2, you own:

• AMI updates: When new GPU drivers or security patches are released, you need to update the AMI and roll instances

• Instance lifecycle: Replacing unhealthy instances, handling retirement notifications

• Capacity management: Tuning the ASG min/max/desired settings and capacity provider parameters

• Debugging: You can SSH into instances (if configured), run nvidia-smi, check ECS agent logs at /var/log/ecs/ecs-agent.log

• Custom configuration: Complete freedom to install monitoring agents, configure kernel parameters, mount volumes, etc.

The operational burden is real, but so is the control. For teams with platform engineering resources, this flexibility is valuable.

Side-by-Side Comparison

Let me break down the practical differences you'll experience day-to-day:

GPU Instance Types and Selection

Both approaches support the same GPU instance families:

• g4dn (NVIDIA T4): Best for inference. A g4dn.xlarge gives you 1 GPU with 16GB memory, costs around $0.526/hour. My go-to for serving models like Stable Diffusion or smaller LLMs.

• g5 (NVIDIA A10G): Newer generation, better performance. A g5.xlarge provides 24GB GPU memory. Great for production inference at scale.

• p3 (NVIDIA V100): Training workhorse. P3.2xlarge has 16GB GPU memory, ideal for fine-tuning.

• p4d (NVIDIA A100): Heavy hitter for large-scale training. P4d.24xlarge gives you 8 A100 GPUs with 320GB total GPU memory.

With Managed Instances, you can use instance_requirements to specify GPU needs flexibly:

instance_requirements {
  accelerator_types = ["gpu"]
  accelerator_count {
    min = 1
    max = 1
  }
  accelerator_manufacturers = ["nvidia"]
  allowed_instance_types    = ["g4dn.xlarge", "g4dn.2xlarge", "g5.xlarge"]
}

This lets AWS pick from multiple instance types based on availability and Spot pricing, which improves your chances of getting capacity.

With traditional EC2, you hard-code the instance type in the launch template, though you can create multiple launch templates and capacity providers if you want flexibility.

Real-World Operational Scenarios

Here are issues I've hit that apply to both approaches:

Cold Start Times: Pulling large ML model images (10GB+) takes time. First task placement on a new instance can take 5-10 minutes. Pre-pull images or load models from S3 at runtime.

GPU Memory Leaks: If your inference code leaks memory, GPUs gradually fill until tasks fail. GPU memory isn't automatically reclaimed like CPU memory. Implement proper cleanup in your application code.

IPv6 Issues: The ECS GPU AMIs have IPv6 enabled, which can cause package manager failures. Add echo "ip_resolve=4" >> /etc/yum.conf to user data if installing additional packages (only relevant for traditional EC2 since Managed Instances don't allow custom configuration).

Task Draining During Instance Replacement: With Managed Instances, AWS replaces instances on its schedule. Your application must handle graceful shutdown. Set appropriate deregistration_delay on target groups and handle SIGTERM in your containers.

Spot Interruption Handling: Both approaches support Spot instances, but handling interruptions requires application-level resilience. Use the 2-minute warning to checkpoint work and drain tasks gracefully.

Cost Optimization Strategies

GPU instances are expensive. Here's how to manage costs:

Scale to Zero: Set ASG min_size = 0 so instances terminate when idle. Accept cold start penalty for sporadic workloads.

Use Spot Instances: Can save 70%+ for non-critical workloads. Both approaches support Spot. With Managed Instances, you still get AWS-managed patching and lifecycle on Spot instances, which is nice.

Right-Size Instances: Don't use p3.8xlarge when g4dn.xlarge suffices. Profile your workload's actual GPU memory and compute needs.

Reserved Instances: For predictable steady-state workloads, Reserved Instances or Savings Plans significantly reduce costs. Both approaches support this, though with Managed Instances you also pay the management layer fee.

Batch Processing: If doing inference on queued jobs rather than real-time serving, batch multiple requests to maximize GPU utilization.

Model Optimization: Use quantization (INT8/FP16) to reduce memory footprint. Smaller models can run on cheaper instance types.

Monitoring GPU Usage

For Managed Instances, you don't have SSH access. This creates a monitoring challenge that's worth addressing upfront.​

The Current Limitation: As of Jan 2026, there's no native CloudWatch GPU metrics support for ECS Managed Instances. There's an open feature request on the AWS containers roadmap (issue #2734) that's been marked as "Proposed," and based on AWS's typical response patterns, this will likely be addressed soon. CloudWatch Container Insights already supports sub-minute GPU metrics for EKS as of November 2025, so the infrastructure exists—it just hasn't been extended to ECS Managed Instances yet.

Until AWS ships native GPU metrics support, monitoring for Managed Instances relies on:

• CloudWatch Container Insights for container-level metrics (CPU, memory, network—but not GPU)

• Custom CloudWatch metrics published from your application code using the AWS SDK

• Application-level GPU metrics exposed via your inference server (e.g., Prometheus metrics endpoint with a sidecar to push to CloudWatch)

This means you need to instrument your application to emit GPU utilization metrics. Here's a practical example using the pynvml library to publish metrics from your inference container:

import pynvml
import boto3
from datetime import datetime

cloudwatch = boto3.client('cloudwatch')

def publish_gpu_metrics():
    pynvml.nvmlInit()
    device_count = pynvml.nvmlDeviceGetCount()
    
    for i in range(device_count):
        handle = pynvml.nvmlDeviceGetHandleByIndex(i)
        
        # Get GPU utilization
        util = pynvml.nvmlDeviceGetUtilizationRates(handle)
        
        # Get memory info
        mem_info = pynvml.nvmlDeviceGetMemoryInfo(handle)
        
        # Publish to CloudWatch
        cloudwatch.put_metric_data(
            Namespace='ECS/GPU',
            MetricData=[
                {
                    'MetricName': 'GPUUtilization',
                    'Value': util.gpu,
                    'Unit': 'Percent',
                    'Timestamp': datetime.utcnow(),
                    'Dimensions': [
                        {'Name': 'GPUIndex', 'Value': str(i)},
                        {'Name': 'TaskID', 'Value': os.environ.get('TASK_ID')}
                    ]
                },
                {
                    'MetricName': 'GPUMemoryUsed',
                    'Value': mem_info.used / (1024**2),  # Convert to MB
                    'Unit': 'Megabytes',
                    'Timestamp': datetime.utcnow(),
                    'Dimensions': [
                        {'Name': 'GPUIndex', 'Value': str(i)},
                        {'Name': 'TaskID', 'Value': os.environ.get('TASK_ID')}
                    ]
                }
            ]
        )
    
    pynvml.nvmlShutdown()

Run this periodically (every 30-60 seconds) in a background thread within your inference container.

For traditional EC2, you have significantly more monitoring options since you control the instances:

# SSH into instance via Session Manager
aws ssm start-session --target <instance-id>

# Real-time GPU monitoring
watch -n 1 nvidia-smi

# Check ECS agent logs for GPU-related issues
cat /var/log/ecs/ecs-agent.log | grep GPU

# Detailed GPU info
nvidia-smi -q

You can also install the CloudWatch agent with GPU support, which automatically collects NVIDIA GPU metrics and sends them to CloudWatch without requiring application-level instrumentation:

# CloudWatch agent config for GPU metrics (add to user data)
cat > /opt/aws/amazon-cloudwatch-agent/etc/nvidia_gpu.json <<'EOF'
{
  "metrics": {
    "namespace": "CWAgent",
    "metrics_collected": {
      "nvidia_gpu": {
        "measurement": [
          {
            "name": "utilization_gpu",
            "rename": "nvidia_smi_utilization_gpu",
            "unit": "Percent"
          },
          {
            "name": "utilization_memory",
            "rename": "nvidia_smi_utilization_memory",
            "unit": "Percent"
          },
          {
            "name": "memory_total",
            "rename": "nvidia_smi_memory_total",
            "unit": "Megabytes"
          },
          {
            "name": "memory_used",
            "rename": "nvidia_smi_memory_used",
            "unit": "Megabytes"
          },
          {
            "name": "memory_free",
            "rename": "nvidia_smi_memory_free",
            "unit": "Megabytes"
          },
          {
            "name": "temperature_gpu",
            "rename": "nvidia_smi_temperature_gpu",
            "unit": "None"
          }
        ],
        "metrics_collection_interval": 60
      }
    }
  }
}
EOF

# Start CloudWatch agent with GPU monitoring
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
  -a fetch-config \
  -m ec2 \
  -s \
  -c file:/opt/aws/amazon-cloudwatch-agent/etc/nvidia_gpu.json

This publishes 17 GPU metrics per GPU to CloudWatch automatically. For production workloads on traditional EC2, this is the recommended approach—it provides comprehensive GPU monitoring without requiring application changes.​

My Recommendation: If GPU metrics visibility is critical for your workload right now and you can't wait for AWS to ship native support for Managed Instances, that's a legitimate reason to choose traditional EC2. The CloudWatch agent GPU support for traditional EC2 instances is mature and battle-tested. However, if you can tolerate application-level instrumentation or are willing to wait a few months for native support (given that issue #2734 was opened in December 2025 and AWS typically moves fast on high-visibility feature requests), Managed Instances still offers significant operational advantages.

Once AWS adds native GPU metrics to Managed Instances—which I expect will happen in 2026 based on the roadmap activity—the monitoring gap will close, making Managed Instances even more compelling for GPU workloads.

Making the Decision

After deploying GPU workloads on ECS using both approaches, here's my practical guidance:

Start with ECS Managed Instances if:

• Your team is small or lacks dedicated platform engineering

• You're running standard ML inference workloads without exotic requirements

• You value operational simplicity over maximum control

• You're comfortable with AWS handling instance replacement timing

• You don't need to debug at the host level frequently

Use traditional EC2 if:

• You need custom kernel modules or host-level configuration

• You require SSH access for debugging or operational tooling

• You have compliance requirements for specific AMI configurations

• Your team has capacity to manage infrastructure lifecycle

• You want to avoid the managed layer cost (though factor in your engineering time cost)

For most teams deploying LLM inference, Stable Diffusion, or similar GPU workloads, I'd recommend starting with Managed Instances. The operational relief is significant, and you can always migrate to traditional EC2 later if you hit limitations.

Example: Deploying a Llama 2 Inference Service

Let me show a concrete example using Managed Instances to deploy a Llama 2 7B inference service.

Dockerfile:

FROM nvidia/cuda:12.1.0-runtime-ubuntu22.04

RUN apt-get update && apt-get install -y python3 python3-pip
RUN pip3 install torch transformers accelerate

COPY serve.py /app/serve.py
WORKDIR /app

CMD ["python3", "serve.py"]

Task Definition (abbreviated):

resource "aws_ecs_task_definition" "llama2_inference" {
  family = "llama2-7b-inference"
  # ... standard settings ...
  
  container_definitions = jsonencode([{
    name  = "llama2"
    image = "your-registry/llama2-inference:latest"
    
    resourceRequirements = [{
      type  = "GPU"
      value = "1"
    }]
    
    environment = [
      { name = "MODEL_ID", value = "meta-llama/Llama-2-7b-hf" },
      { name = "MAX_BATCH_SIZE", value = "8" }
    ]
  }])
}

Capacity Provider with Managed Instances:

instance_requirements {
  accelerator_types         = ["gpu"]
  accelerator_count { min = 1, max = 1 }
  accelerator_manufacturers = ["nvidia"]
  allowed_instance_types    = ["g5.xlarge", "g5.2xlarge"]
}

With this setup, AWS provisions g5 instances with A10G GPUs, pre-configured with NVIDIA drivers. Your container starts, downloads the Llama 2 model weights (ideally from S3, not during startup), and serves inference requests. When traffic drops, instances scale down. When instances need patching, AWS drains tasks and replaces them.

Total infrastructure code? About 150 lines of Terraform. No user data scripts, no GPU driver debugging, no AMI management.

Wrapping Up

The introduction of ECS Managed Instances fundamentally changes the GPU workload conversation on ECS. The old advice of "Fargate doesn't support GPUs, so you're managing EC2 instances" is outdated. You now have a middle ground that handles most of the operational burden while still supporting GPU workloads.

Start with Managed Instances unless you have specific reasons not to. You can always migrate to traditional EC2 later if you hit limitations. The configuration is nearly identical—mostly just swapping out how the capacity provider and launch template are defined.

The hardest parts of running GPU workloads on ECS aren't about infrastructure anymore—they're about application concerns like model loading times, memory management, request batching, and handling inference failures gracefully. Focus your engineering effort there, and let AWS handle the infrastructure lifecycle.

And remember: whichever approach you choose, ECS_ENABLE_GPU_SUPPORT=true in the ECS agent config is non-negotiable for traditional EC2. With Managed Instances, AWS handles that for you. That alone might be worth the managed layer cost.

Now go deploy some models.

References

Key AWS documentation and industry sources referenced in this post:
- ECS Managed Instances: A practical comparison with Fargate and EC2
- Amazon ECS Managed Instances now available in all commercial AWS Regions
- How to Run GPU Workloads on ECS: Complete Implementation Guide
- Amazon ECS task definitions for GPU workloads