Eliminating AWS Credentials from a Public Next.js Pod: The In-Cluster BFF Migration

TL;DR

Migrating to the in-cluster Backend-for-Frontend (BFF) pattern for Kubernetes credential isolation means your public-facing pod never touches AWS directly. Every read (articles, resume, chat, and engagement) proxies through a single in-cluster service over Kubernetes DNS. The frontend holds zero AWS credentials at runtime. The BFF owns RDS, Secrets Manager, and the Bedrock API key. When the BFF is unreachable the site degrades gracefully instead of hard-failing, because Docker builds and ISR prerenders succeed without cluster access.

The Problem: Credentials in the Wrong Place

The original portfolio application called DynamoDB and S3 directly from the Next.js pod. The Bedrock chatbot went through an API Gateway endpoint. The pod held AWS SDK credentials as environment variables — meaning a public-facing container sitting behind an ALB had the IAM permissions to read article content, fetch resume data, and invoke AI models.

That is a large blast radius for a read-only site.

Every time I wanted to add a new data domain — article engagement, chat history, project metadata — I had to open the portfolio repo, write new IAM policy statements, update the pod's service account annotations, re-deploy. Three PRs across two repositories for a data feature that the backend already supported. The infrastructure coupling was the bottleneck, not the application code.

More fundamentally: the frontend pod did not need to know about AWS. Its job is to render data, not own the credentials that fetch it. Putting IAM permissions on a public pod conflates two responsibilities that should live at different trust levels.

The result was a policy with seven AWS API actions on the pod's service account: reading items from DynamoDB (dynamodb:Query, dynamodb:GetItem), fetching objects from S3 (s3:GetObject), retrieving a secret from Secrets Manager (secretsmanager:GetSecretValue), and three Bedrock model-invocation actions, all attached to a pod that serves anonymous visitors. That policy dropped to zero after the migration.

Architecture: Consumer-Only BFF Pattern

The fix is straightforward to describe. Hard to get right in the details.

The nextjs pod (the container that runs the public-facing Next.js website) now holds no AWS data credentials. It talks to one address: http://public-api.public-api:<port>, a Kubernetes service DNS name that resolves to the public-api service regardless of which physical nodes the pods land on, how many replicas are running, or whether pods have restarted. The browser never sees this address. It only ever calls same-origin /api/* routes. The call to public-api happens server-to-server, inside the cluster.

public-api (a Hono service) owns every privileged dependency: RDS credentials via Secrets Manager, the Bedrock RAG Lambda API key, and the VPC-private RDS Postgres instance. RDS has no public IP and is reachable only from inside the VPC. That is the security boundary — not a firewall rule, but a network topology that makes the database unreachable by design.

ALB routing matters here. The shared public IngressGroup routes a dedicated API host to public-api and the main site host to the nextjs service at path /. Because routing is by host rather than path, the site's own /api/* handlers are not transparently rewritten to public-api. The portfolio uses the Next-proxy model: route handlers fetch public-api in-cluster and re-serve the result. The browser only ever calls the site's same-origin /api/* routes; the in-cluster service address never appears in client code.

Implementation: Data Flow and Security Boundaries

Articles, queryPublishedArticles fetches public-api's articles endpoint with next: { revalidate: 300 } for ISR. The BFF runs parameterised SQL against RDS and returns JSON. One migration seam worth noting: the article metadata Zod schema requires a non-empty contentRef, so the RDS layer synthesises rds://<slug> to satisfy validation. RDS content_md is Markdown and renders through the existing MDX renderer unchanged.

Chat, the site's own /api/chat route (the server-side endpoint the on-page chat widget calls) proxies to public-api's /api/chatbot/authenticated endpoint and normalises the { response } envelope to its own { message, sessionId } contract. The BFF injects the Bedrock API key from Secrets Manager before forwarding. Lami's retrieval uses pgvector and BM25 via Reciprocal Rank Fusion over the repository knowledge base. The consumer sends no x-api-key. It has no key to send.

Engagement (likes/comments), the consumer forwards the original client IP as x-forwarded-for so the BFF's per-IP rate limit applies to the actual visitor, not the pod. This matters: without forwarding, every request would originate from the same pod IP and rate limiting would be meaningless.

Resume, falls back to HTTP 204 if the BFF is unreachable. The UI uses hardcoded data. Writes use cache: 'no-store'.

Graceful degradation is a first-class contract, not an afterthought. Docker builds and Next.js ISR prerenders run with no cluster access. The build must succeed against an unreachable BFF. queryPublishedArticles returns [] on a non-OK response or network error; detail lookups return null. The site renders its shell, and ISR fetches real data at runtime.

Challenge Log

The Secrets Manager ARN Problem

The Bedrock Knowledge Base stores vectors in Pinecone, chosen to eliminate the ~£15–35/month OpenSearch Serverless idle cost at ~£2–8/month in token costs. Connecting Bedrock to Pinecone requires a Secrets Manager secret for its API key. The platform is now consolidating this managed vector store onto RDS PostgreSQL with pgvector, which retires the external Pinecone dependency and its API-key secret; the ARN lesson below still applies to any managed-secret integration.

Bedrock's Knowledge Base stack spent 90 minutes deploying successfully, passing all CloudFormation checks, and returning completely empty results on every query. No errors. Just silence.

The root cause: Bedrock requires the complete Secrets Manager ARN (a secret's full unique identifier), including the random six-character suffix AWS appends to every secret. Secret.fromSecretNameV2, the AWS CDK helper that imports an existing secret by its name, returns a partial ARN without that suffix. Bedrock uses the ARN to build the session policy that lets it read the secret's value (the secretsmanager:GetSecretValue API action). A partial ARN causes silent session policy evaluation failure.

1// lib/stacks/bedrock-kb-stack.ts
2// AwsCustomResource resolves the full ARN at deploy time.
3// onDelete is intentionally omitted — this is a read-only lookup resource
4// that holds no state. CDK will emit a warning on removal; that is acceptable.
5const describeSecret = new cr.AwsCustomResource(this, 'DescribePineconeSecret', {
6    onCreate: {
7        service: 'SecretsManager',
8        action: 'describeSecret', // camelCase — matches AWS JavaScript SDK v3 method name
9        parameters: { SecretId: props.pineconeSecretName },
10        physicalResourceId: cr.PhysicalResourceId.of(`${namePrefix}-pinecone-secret-lookup`),
11    },
12    onUpdate: {
13        service: 'SecretsManager',
14        action: 'describeSecret',
15        parameters: { SecretId: props.pineconeSecretName },
16        physicalResourceId: cr.PhysicalResourceId.of(`${namePrefix}-pinecone-secret-lookup`),
17    },
18    policy: cr.AwsCustomResourcePolicy.fromStatements([
19        new iam.PolicyStatement({
20            actions: ['secretsmanager:DescribeSecret'],
21            resources: [
22                `arn:aws:secretsmanager:${this.region}:${this.account}:secret:${props.pineconeSecretName}-??????`,
23            ],
24        }),
25    ]),
26});
27
28const pineconeSecretFullArn = describeSecret.getResponseField('ARN');

The fix is a deploy-time DescribeSecret lookup (the Secrets Manager API call that returns a secret's metadata, including its full ARN) run through AwsCustomResource, a CDK construct that executes a one-off AWS SDK call as part of the CloudFormation deployment. It resolves the full ARN during that deployment and passes it downstream. The Knowledge Base started returning results immediately.

Transferable value: any CDK stack (which deploys as CloudFormation) that passes a Secrets Manager secret to a service that builds its own session policy, such as Bedrock, MSK, or RDS Proxy, needs the full ARN. fromSecretNameV2 is the wrong tool for that. Build the lookup into the stack from day one.

Junior Corner

Why does the BFF hold the keys and not the frontend?

Think of a medieval castle. The keep is your database. The outer walls are your cluster. The public-facing Next.js pod is the castle's gate, visitors interact with it, so it must be exposed. The BFF (public-api) is the inner chamberlain: it lives behind the outer walls, holds the actual keys to the keep, and only fetches what a legitimate request needs.

If you give the gate-guard the master keys and someone finds a way past the gate, they have everything. If the gate-guard can only shout requests to the chamberlain inside, the keys never leave the inner walls.

In Kubernetes terms: the public pod's service account has no IAM annotations. It cannot call AWS. The BFF's service account has an IRSA annotation granting it exactly the Secrets Manager and RDS permissions it needs. Kubernetes service DNS (public-api.public-api:<port>) is the stable internal address that makes this possible, it resolves to the BFF regardless of pod restarts or scaling events, no hardcoded IPs required.

Adding a new data domain now means adding a BFF endpoint. No IAM policy changes. No frontend infra PRs.

Where This Applies

This pattern fits any multi-tier Kubernetes deployment where a public-facing application needs to read from private data stores: SaaS products with tenant-isolated RDS, internal tooling behind ALB, or portfolio sites that serve static and dynamic content from separate layers.

The operational benefit compounds quickly. Before the migration, adding a data feature required changes in two repositories and a new IAM policy statement on the frontend service account. After: one PR in the application repository, no infrastructure touch. That reduction in coordination cost is the real win for any team running more than two services.

The graceful degradation contract is equally transferable. Build pipelines that require live infrastructure connections will eventually fail during maintenance windows or cluster upgrades. Designing read paths to return empty collections on unreachable upstreams, rather than throwing, keeps deployment pipelines green regardless of cluster state.

For any deployment beyond a single-service prototype, putting AWS credentials directly on a public pod is a liability that outweighs the convenience. The BFF boundary is the correct abstraction.

FAQ

What is the difference between the BFF pattern and a standard API gateway?

A Backend-for-Frontend is a dedicated aggregation and adaptation layer built for one specific consumer, whereas a general API gateway is a shared entry point for many consumers. The BFF owns the translation logic, credential injection, and rate limiting that the consumer needs. A generic API gateway would need to expose those concerns to every client it serves.

How does Kubernetes service DNS work for in-cluster communication?

Kubernetes DNS creates a stable DNS name for every Service object in the cluster. For a Service named public-api in the public-api namespace, the fully qualified address is public-api.public-api.svc.cluster.local, shortened in practice to http://public-api.public-api:<port>. This name resolves to the service's ClusterIP regardless of which pods back it or how many replicas are running.

Why can't the frontend just use IAM Roles for Service Accounts with least-privilege?

IRSA with tight scoping is better than no scoping, but it still attaches AWS credentials to a public pod. Credential rotation, token expiry, and audit trail all become frontend concerns. The BFF pattern moves those concerns to the BFF pod entirely. The frontend does not interact with AWS at all at runtime, there is no credential surface to harden.

Lessons and Next Steps

The core shift is architectural, not technical: stop thinking of the frontend as a data client and start treating it as a pure presentation layer. The BFF is the data client. The frontend is its consumer.

Kubernetes service DNS makes this cheap to implement. There is no service mesh, no sidecar, no additional infrastructure, just a stable DNS name the frontend fetches over HTTP inside the cluster.

Next: mTLS between the nextjs pod and public-api for defence-in-depth in zero-trust environments. The BFF pattern gives credential isolation at the IAM layer; mTLS would add transport-layer identity verification so a compromised pod cannot impersonate the frontend. This experience positions me to evaluate credential isolation strategies, IRSA scoping, BFF proxying, mTLS, network policies, as independent layers of a defence-in-depth posture, rather than treating IAM as the only control.

Ask the Lami chat assistant directly via the chat widget on the site. It runs through this exact proxy path.