Building a Production EKS Platform: GitOps, Pod Identity, and the Golden Path

TL;DR

I built and operate a production EKS platform running Kubernetes 1.34 in eu-west-1 for the Tucaken SaaS backend. The architecture splits into two ownership domains. CDK provisions the EKS substrate: control plane, system node group, Karpenter, and Pod Identity bindings. A dedicated kubernetes-bootstrap repository owns everything ArgoCD reconciles inside the cluster. An app-of-apps root Application with sync-wave: -10 seeds every platform service and workload. Pod Identity replaces static credentials entirely. Node capacity is two-tier: a small always-on system node group for platform components, plus Karpenter for workload autoscaling. That keeps infrastructure costs predictable. Together these form the platform's golden path: the paved default route a new workload follows from repository to running pod.

The Problem: Defaults Don't Scale

Managed EKS solves the control plane problem. It does not solve the platform problem.

Out of the box, EKS gives you a running cluster and a blank canvas. Node provisioning, secret management, ingress, observability, database access patterns, deployment strategy: all of that is yours to define. Teams that skip defining it end up with in-cluster state that nobody fully owns. Credentials rotated manually. Node scaling done by hand. No version control over the platform configuration itself.

For the Tucaken SaaS backend I needed answers to three specific questions before writing a single manifest:

Who controls the cluster substrate? VPC layout, IAM roles, KMS keys, node groups. These are infrastructure-team concerns: they change infrequently, they require careful review, and a mistake here can take down the entire cluster. They belong in CDK rather than GitOps.

Who controls what runs inside the cluster? ArgoCD applications, Helm chart versions, secret sync, database migrations. These change on every feature branch merge. They belong in Git, reconciled by ArgoCD, reviewed as pull requests.

How do pods get AWS credentials? The wrong answer: static docker-registry secrets, long-lived IAM access keys stored in Kubernetes Secrets, CronJobs to refresh ECR tokens. I had all three in an earlier iteration. ECR tokens expire after 12 hours, and the refresh CronJob I had planned never got built. ArgoCD Image Updater was silently failing on every ECR poll, causing the deploy pipeline to hang for ten minutes on every image push before I noticed.

The right answer is EKS Pod Identity. One binding per ServiceAccount, IAM credentials rotated automatically, no token management whatsoever.

Those three questions drove every architectural decision documented here.

Architecture: Split Ownership and Two-Tier Nodes

The split is deliberate, and it is the platform's golden path. A new workload follows one paved route: a Helm chart in kubernetes-bootstrap, an ArgoCD Application in the app-of-apps tree, and a Pod Identity binding in CDK if it needs AWS access. Everything else, nodes, secrets, ingress, observability, is already paved.

CDK in cdk-monitoring provisions the EKS control plane, system managed node group, and Karpenter. The kubernetes-bootstrap repository owns everything ArgoCD reconciles onto the cluster. In-cluster changes are pure-Git pull requests, reviewed alongside the manifests they affect. The two repos never share state at runtime: CDK writes SSM parameters, kubernetes-bootstrap reads cluster endpoints at ArgoCD bootstrap time, and the two concerns stay independent.

Node Provisioning: System MNG + Karpenter

Node capacity is two-tier by design, not by accident.

A system managed node group of 2x t3.medium instances runs AL2023 with a dedicated=system:NoSchedule taint. CoreDNS, Karpenter itself, and ArgoCD land here. These nodes are always on; they are the cluster's nervous system. Losing them means losing the ability to schedule anything else.

Karpenter provisions workload nodes on demand. No cluster-autoscaler. Karpenter's EC2NodeClass uses tag-based security group discovery (kubernetes.io/cluster/<name>: owned) rather than a pre-provisioned SG ID, a deliberate choice explained in the Challenge Log. Karpenter also requires an SQS interruption queue to handle Spot interruption notices and node termination events gracefully; that queue is provisioned in EksKarpenterStack alongside the NodePool.

Pod Identity Bindings

Pod Identity (the modern replacement for IRSA, the older role-per-ServiceAccount mechanism built on OIDC federation) binds an IAM role to a specific Kubernetes ServiceAccount in a specific namespace. No static tokens. No 12-hour expiry. No refresh CronJobs. The EksPodIdentityStack in CDK defines every binding the platform needs:

• argocd/argocd-image-updater — ecr:GetAuthorizationToken (account-scoped) + ecr:DescribeImages and ecr:ListImages (repository-scoped)
• ingestion-sa — bedrock:InvokeModel
• headlamp/token-pusher — ssm:PutParameter on the Headlamp viewer-token parameter
• grafana — CloudWatch metric and log read actions for the Grafana CloudWatch datasource
• waf-annotator — lifecycle managed outside CloudFormation (explained in the Challenge Log)

The EKS Pod Identity documentation recommends Pod Identity over IRSA for new clusters; the credential rotation is automatic and the binding model is simpler to reason about at scale.

App-of-Apps: The GitOps Spine

ArgoCD reconciles everything from argocd-apps/eks/development/. A root Application (sync-wave: -10) seeds every other Application in the tree. Platform services (PgBouncer, External Secrets Operator, the LGTM observability stack, cert-manager, Crossplane, ARC self-hosted runners) land at sync-waves 0 through 2. Tucaken workloads follow at sync-wave 3 and above.

The ordering matters. Workload pods reference ExternalSecrets by name. If ESO has not synced the Secret yet, the pod fails to start. Sync waves are the contract that prevents this: wave 2 (ESO + ExternalSecrets) completes before wave 3 (workloads) begins. ArgoCD's app-of-apps documentation covers the pattern, but the production detail, which services need to be Ready before workloads start, is something you only learn by watching a deploy sequence fail.

ArgoCD Image Updater polls ECR for new image tags and git-writes the updated tag back to argocd-apps/eks/development/. CI pushes an image; Image Updater detects it; ArgoCD reconciles the updated tag. The deploy pipeline requires no manual kubectl commands.

Data Tier: PgBouncer as the Single RDS Ingress

Every pod connects to RDS PostgreSQL through PgBouncer in transaction mode. Direct RDS connections don't scale: each idle connection holds a backend slot. PgBouncer multiplexes hundreds of pod connections onto a small pool of real RDS connections.

Two exceptions. Bootstrap Jobs and RDS schema migration Jobs connect directly to RDS, bypassing PgBouncer. The reason: PgBouncer may not be Ready on first deploy, and a migration Job that waits on PgBouncer creates a circular dependency. One documented debt remains: the read-only grafana_ro role fails SASL authentication through PgBouncer, so the Grafana datasource stays on the postgres master user until PgBouncer's SASL configuration is resolved.

Implementation: CDK Stacks, Helm Charts, Sync Waves

The CDK app decomposes the cluster into single-purpose stacks with explicit dependency ordering. Each stack is one failure domain. Rolling back EksKarpenterStack doesn't touch the control plane. Redeploying EksPodIdentityStack doesn't affect node groups. This is not organisational tidiness. It is operationally essential when you need to patch a single concern at 2am without risk of cascading CloudFormation changes.

EksClusterStack creates the managed control plane, a KMS envelope key for Kubernetes Secrets encryption, and the CloudWatch log group. No node group. The first node doesn't exist until EksSystemNodeGroupStack runs.

PlatformRdsStack runs in parallel with EksClusterStack. Both perform their own Vpc.fromLookup() using the shared VPC name tag, so neither has a CDK dependency on the other. In the GitHub Actions pipeline, deploy-platform-rds and deploy-cluster both needs: [setup] and run concurrently, shaving real minutes off the full deployment.

The reusable _deploy-eks.yml workflow synthesises stack names via just ci-synth kubernetes development, then fans out to six dependent jobs:

1# .github/workflows/_deploy-eks.yml (simplified)
2jobs:
3  setup:       # CDK synth → outputs stack names
4  deploy-cluster:      needs: [setup]
5  deploy-platform-rds: needs: [setup]          # parallel with cluster
6  deploy-system-ng:    needs: [setup, deploy-cluster]
7  deploy-pod-identity: needs: [setup, deploy-system-ng]
8  deploy-addons:       needs: [setup, deploy-pod-identity]
9  deploy-karpenter:    needs: [setup, deploy-addons]
10  deploy-access:       needs: [setup, deploy-cluster]  # independent chain

No secrets: inherit. Every job declares exactly the secrets it needs: deploy-access gets the OIDC and admin role ARNs for Access Entry synth, and every other job gets only the deploy role. Removing implicit secret inheritance reduces the blast radius of a compromised job from "all repo secrets" to one OIDC role.

The 21 Helm charts in charts/ are ArgoCD-managed. Local rendering for validation:

1# Render a chart locally for inspection
2helm template platform-rds charts/platform-rds/chart \
3  -f charts/platform-rds/chart/values-development.yaml

Post-Sync Jobs handle RDS schema migrations. Every ArgoCD sync triggers an idempotent DDL migration Job that connects directly to RDS (bypassing PgBouncer), applies pending migrations, and exits. Running it twice on the same schema is safe. This gives you schema-as-code with the same Git review cycle as everything else.

Challenge Log

ECR Credentials That Never Refreshed

ArgoCD Image Updater was configured with a static docker-registry secret for ECR authentication. The token behind it goes stale half a day after issue. A comment in the config referenced a refresh CronJob. That CronJob never existed.

The symptom: every image push to ECR caused the Image Updater to fail with "authorization token has expired", log a silent error, and retry for ten minutes before giving up. From the outside, the deploy pipeline looked stuck. The fix was not to build the CronJob. It was to eliminate the token entirely.

I added a PodIdentityBinding for argocd/argocd-image-updater with ecr:GetAuthorizationToken (account-scoped, required for Docker login) and ecr:DescribeImages + ecr:ListImages (repository-scoped, for the newest-build comparison strategy). Image Updater's Helm config switches to the awsecr: credential type. IAM credentials are now rotated automatically by Pod Identity. No token, no expiry, no CronJob.

Transferable value: any time you find yourself writing a token refresh CronJob, stop and ask whether Pod Identity or IRSA makes the CronJob unnecessary. It almost always does.

IMDS Hop Limit and the Invisible Network Bridge

After adding a Pod Identity binding for ingestion-sa, pods were still failing with CredentialsProviderError. The EC2 instance metadata service (IMDS, the on-instance HTTP endpoint that serves credentials and instance details) is how Pod Identity delivers temporary credentials to pods. By default, EC2 instances have httpPutResponseHopLimit=1. That limit works for processes running directly on the instance, but pods sit behind an extra network hop inside the node, so the IMDS response never arrives.

The fix is one line in EC2NodeClass:

1# charts/argocd-eks/values.yaml (EC2NodeClass fragment)
2apiVersion: karpenter.k8s.aws/v1
3kind: EC2NodeClass
4spec:
5  metadataOptions:
6    httpPutResponseHopLimit: 2   # allows IMDS reach from inside Linux network bridge
7  securityGroupSelectorTerms:
8    - tags:
9        kubernetes.io/cluster/k8s-eks-development: owned

Setting the hop limit to 2 lets the IMDS PUT response traverse the bridge hop. Karpenter's EC2NodeClass documentation covers EC2NodeClass metadataOptions, but the hop limit default is one of those settings that only becomes visible when a pod can't acquire credentials and the error message points nowhere obvious.

CloudFormation Ownership Is Fragile for Pod Identity Associations

All four properties on AWS::EKS::PodIdentityAssociation are replace triggers in CloudFormation: cluster name, namespace, service account name, and role ARN. Change any of them and CloudFormation tries to delete the old association and create a new one.

When I moved the WAF annotator ServiceAccount to a different namespace, CloudFormation attempted to create a new association before the old one had been deleted. Result: 409 AlreadyExists, because the existing association still owned the (cluster, namespace, SA) tuple.

The fix was to remove CfnPodIdentityAssociation from the CDK lifecycle entirely for this binding. The association is created once via the AWS CLI and its ARN is stored in an SSM parameter:

1# Create the association outside CloudFormation
2aws eks create-pod-identity-association \
3  --cluster-name k8s-eks-development \
4  --namespace admin-api \
5  --service-account waf-annotator \
6  --role-arn <roleArn>
7
8# ARN stored in SSM under the platform's eks parameter prefix

The association continues working. CloudFormation no longer owns its lifecycle.

Transferable value: when every identity field on a resource is a replace trigger, CloudFormation can only ever delete-and-recreate it, so take that resource out of CloudFormation's hands. The general pattern is in the Lessons section.

Karpenter Race Conditions During Dev Shutdown

A dev-shutdown Lambda (triggered at 4am) scaled all Deployments to zero to save cost. It scaled everything, including karpenter and argocd. Two cascading failures followed:

1. Karpenter scaled to zero before NodeClaims were drained. Karpenter's termination finalizer never cleared. Nodes were stuck in etcd indefinitely as zombies.
2. ArgoCD scaled to zero during the drain window. On cluster restart, ArgoCD wasn't running to reconcile workloads. HPA demand went unmet. The cluster came back up with no workloads running.

The fix required careful ordering:

• ScaleDownFn: scale ArgoCD and Karpenter to zero first (preventing ArgoCD from syncing during the drain window and Karpenter from provisioning replacement nodes). Drain NodeClaims completely. Then set the system MNG to zero.
• ScaleUpFn: restore system nodes, wait for Ready, restore Karpenter (2 replicas), wait for rollout, restore ArgoCD (1 replica), wait for rollout, then allow WAF sync.

The EksSchedulerStack 4am Lambda patches both Karpenter and ArgoCD Deployments via the k8s API directly (EKS bearer token + SigV4) so the same restore logic applies whether the cluster is started manually or via EventBridge.

ArgoCD Secret Bootstrap: The Helm Keep Annotation Trap

argocd-secret carries helm.sh/resource-policy: keep. That annotation tells Helm: do not delete this resource on helm upgrade --install if the release already exists. The intent is to preserve the ArgoCD server secret key across upgrades.

After a cluster rebuild, the Helm release exists (it's in the cluster state) but argocd-secret doesn't (it was deleted with the old cluster). Helm sees the existing release, skips creating resources annotated with keep, and leaves argocd-secret absent, and the ArgoCD server and dex pods crash-loop on startup.

The fix: an idempotent ensure_argocd_secret() function runs between helm upgrade --install and the ArgoCD readiness wait. It creates argocd-secret with a random server secret key if the Secret is missing, and is a no-op if it exists. The same step runs in both the ScaleUpFn Lambda and the ArgoCD install workflow in GitHub Actions.

Junior Corner

Why split ownership between CDK and GitOps instead of putting everything in one repo?

Think of it like a building. The structural engineer designs the foundation and load-bearing walls: that's CDK, provisioning VPC, IAM, EKS. The interior designer specifies what goes inside: that's GitOps, defining what ArgoCD reconciles. You'd never ask the interior designer to move a load-bearing wall. You'd never ask the structural engineer to pick the curtain colours.

In practice this means: infrastructure changes (VPC subnets, IAM roles, KMS keys, EKS version upgrades) live in CDK and go through a CDK review cycle with cdk diff and cdk deploy. Application changes (new Helm chart versions, manifest updates, secret sync) live in kubernetes-bootstrap and go through an ArgoCD review cycle with a GitOps pull request.

Two concepts worth internalising before your next EKS project:

App-of-apps and sync waves. The root Application has sync-wave: -10. ArgoCD processes waves in ascending order, waiting for each wave to reach healthy status before starting the next. Wave -10 runs first and creates every other Application resource. Platform services land in waves 0-2. Workloads land in wave 3+. If an ExternalSecret is in wave 2 and the workload that consumes it is in wave 3, you never have a race condition where the workload starts before its secret exists. Sync waves are a contract rather than a sequencing hint.

EKS Pod Identity. The modern pattern for giving pods AWS permissions. No static tokens, no IAM access keys in Kubernetes Secrets, no CronJobs. One CDK binding maps a Kubernetes ServiceAccount to an IAM role, and EKS rotates the credentials automatically via IMDS. Kubernetes service account documentation covers the foundation; Pod Identity is the AWS-native layer on top. Learn it now. Any team running EKS in 2026 is either using it or migrating to it.

Where This Applies

Every team building an EKS platform for a real product faces the same three questions I started with. The patterns here transfer directly.

Startup platform teams moving from a single managed Kubernetes namespace to a real platform layer: the CDK + GitOps split reduces friction over who owns what. CDK is the infrastructure team's surface. GitOps is the product team's surface. The boundary is explicit and enforced by repository structure.

SRE teams migrating from self-managed Kubernetes to EKS: the app-of-apps pattern, sync waves, and Pod Identity bindings are operational patterns to learn before migrating production. The earlier kubeadm cluster this platform replaced gave me the internals knowledge to understand what EKS abstracts, etcd health, control plane availability, node bootstrap, and to make informed decisions about where to lean on managed services versus where to maintain explicit control.

Teams running Bedrock, Lambda, and RDS on EKS: this repo shows production patterns for each. Pod Identity for Bedrock model invocation, PgBouncer for RDS connection pooling, External Secrets Operator for Secrets Manager sync. The patterns are validated against the live running cluster.

Cost-conscious teams: two-tier node provisioning (system MNG always on, Karpenter workload pool scaling to zero at night) keeps base infrastructure cost low without sacrificing the ability to run workloads on demand. The EksSchedulerStack handles the nightly scale-down and morning restore automatically.

FAQ

What is the difference between EKS Pod Identity and IRSA?

Both bind an IAM role to a Kubernetes ServiceAccount, but IRSA does it through an OIDC identity provider and a role trust policy per ServiceAccount, while Pod Identity uses a first-class EKS association delivered through the instance metadata service. Pod Identity needs no OIDC provider setup, rotates credentials automatically, and is the AWS-recommended default for new clusters.

Why run a fixed system node group when Karpenter can provision everything?

Karpenter cannot schedule its own pods onto nodes it has not yet provisioned. The components that make the cluster function, CoreDNS, ArgoCD, and Karpenter itself, need capacity that exists before any autoscaling decision runs. The always-on system node group with a dedicated=system taint is that capacity; Karpenter then handles everything workload-shaped on demand.

What makes this a golden path rather than just a directory convention?

A golden path is paved end to end: a new service gets a Helm chart, an ArgoCD Application, and a Pod Identity binding, and every other concern, node capacity, secret sync, ingress, observability, migration Jobs, is inherited from the platform. The team never makes an unreviewed infrastructure decision by accident, because the only route to production runs through two well-defined review cycles.

Lessons and Next Steps

CloudFormation ownership is fragile for any Kubernetes resource whose identity properties are replace triggers. Pod Identity associations, PodDisruptionBudgets, and cert-manager Certificate resources all share this characteristic. The pattern that works: create them once via the AWS CLI or a Kubernetes operator, store the ARN or resource name in SSM, and let the application layer manage the lifecycle.

App-of-apps with sync waves is the only GitOps model that scales past a handful of services. Without explicit wave ordering, you're relying on luck for dependency satisfaction. Luck fails in proportion to the number of services.

The self-managed kubeadm cluster this platform replaced was the right starting point. Running etcd, configuring Calico CNI, debugging kubelet TLS bootstrapping: that experience lets me evaluate every EKS abstraction with eyes open. When the IMDS hop limit hit, I knew what IMDS was and why it mattered.

Next: extend the golden path with Open Policy Agent admission control, enforcing that every workload pod uses Pod Identity, runs as non-root, and cannot mount host filesystems. Policy-as-code at the admission layer is the natural progression from GitOps-as-code at the reconciliation layer. After that: observability for the platform itself, ArgoCD sync latency, reconciliation error rates, pod startup time distribution, surfaced in Grafana. Right now I monitor the workloads. The next milestone is monitoring the platform that runs them.

Ask the Lami chat assistant directly via the chat widget on the site. It runs on this exact platform, reconciled by the same app-of-apps tree described above.