Side Project

AI Applications Platform with Infrastructure-as-Code

Tucaken: production SaaS on Kubernetes-on-AWS that turns a developer's real GitHub code into a job-tailored, evidence-backed resume.

Sole BuilderMulti Repo

Overview

Tucaken is a SaaS for software engineers who want a resume that is honest and tailored per job posting — grounded in what their actual code shows, not keyword stuffing. A job-seeker connects their GitHub account; Tucaken verifies which skills they can prove from their repositories, reads each job description into a canonical required-skill list, assesses the candidate's evidence (verified / partial / gap), and generates a tailored resume through a multi-agent Bedrock pipeline. The platform spans four repositories: the AI/ML backend (ingestion, skill-evidence extraction, 5-agent synthesis), the tucaken-app web app and BFF, the tucaken-infra 16-CDK-stack infrastructure layer, and the kubernetes-bootstrap in-cluster GitOps repo. I built the infrastructure as a 16-CDK-stack monorepo (tucaken-infra) that provisions an EKS cluster with Karpenter autoscaling, Pod Identity (replacing IRSA), Argo Rollouts blue/green deployments, a regional WAFv2 WebACL on the shared internet-facing ALB (CloudFront was fully retired and removed), and a full observability suite — Prometheus, Grafana, Loki, Tempo, Pyroscope — deployed GitOps-style via ArgoCD. The kubernetes-bootstrap repo owns the in-cluster GitOps manifests, Helm chart values, and purpose-built Grafana dashboard JSON files covering RDS health, Bedrock spend attribution, RAG evaluation scores, Redis cache effectiveness, RUM (Faro), ingestion funnels, and tracing. Key engineering challenges solved include: surviving large context windows in multi-repo case-study generation, replacing the silently-deprecated ADOT Lambda layer with self-contained X-Ray tracing, driving skill-canonicalisation recall from near-zero (2.2% canonical corpus) to a measured working system through six enrichment iterations — controlled-vocabulary enrichment, content-hash dedup cache, chunk-packing, Tier 1 deterministic tech→skill rules, and query-side canonicalisation — designing a filter-then-rank retrieval system with file-grained evidence stamps, building an ATS honesty pipeline with vendor-provenance guards and doc-vs-code drift prevention, hardening Grafana dashboards against SQL injection and double-scrape bugs, and migrating the cluster edge from a decommissioned Traefik NLB to the EKS ALB while correctly fixing IAM policies where bedrock:Rerank silently fell back to cosine retrieval.

Stack

Language: TypeScript
Framework: Reactaws_cdkhonozodopentelemetryjest
Database: postgresqlRedis
Infrastructure: awsargocdargo_rollouts
External Service: aws_bedrockstripeaws_account
Observability: grafana
Ci Cd: GitHub Actions

Architecture

Hover to zoom

Highlights

Controlled-vocabulary enrichment activated in production: skills overlap lane lifted from 2.2% to full operation
The skills overlap lane (d.skills && query.skills) was near-dead — measured at 2.2% canonical coverage — with 17,138 distinct LLM surface-forms against a hand-seeded vocabulary too small to match them. I built the full canonicalisation stack across six iterations: skill_ontology foundation with Titan embedding backfill, nearest-canonical resolver, controlled-vocabulary enricher emitting skills only from the shared ontology, content-hash dedup cache skipping Haiku for unchanged chunks, chunk-packing (many chunks per LLM call under one system prompt), Tier 1 deterministic tech→skill rules, and query-side canonicalisation resolving JD skills through the same cascade. A hand-labelled golden eval set (21 real chunks, 4 repos, 7 file types) provides ground truth for measuring recall without sampling bias. Enabled in production via PR #153 in tucaken-app after the method-aware cache fix in PR #315.
EKS platform layer: Pod Identity, Karpenter autoscaling, WAFv2 ALB edge, and Traefik NLB retired
The tucaken-infra 16-CDK-stack monorepo provisions the entire platform. The decommissioned Traefik NLB (zero traffic post-EKS migration) and all CloudFront residue were fully removed — dead S3 OAC bucket policy, NLB construct, SSM exports, stale nag suppressions — so a future CDK deploy can no longer silently recreate them. IAM bugs were found and fixed: the admin-api Pod Identity role lacked ce:GetCostAndUsage (FinOps tab silently empty), and the pipeline role's bedrock:Rerank was scoped to ARNs that the action doesn't support at resource level — both found and corrected. The deploy-api.yml CI pipeline and _deploy-kubernetes.yml reusable workflow orchestrate all container builds and GitOps rollouts.
Purpose-built Grafana dashboards with RAG eval panels, per-user Bedrock spend attribution, and SQL-injection hardening
kubernetes-bootstrap ships provisioned Grafana dashboards covering RDS health, Bedrock spend attribution, RAG evaluation scores (persisted via rag_eval_runs), ingestion funnels, cluster health, tracing, RUM (Faro web vitals), resume-import, and auto-bootstrap — visible in the heavily-changed dashboard JSON files. Granular Bedrock cost attribution was added (application_id, project_id, sync_kind columns on prompt_invocations). Dashboard data-correctness bugs were fixed: the db_identifier freetext variable blanked all CloudWatch panels; per-user spend panels interpolated variables directly into SQL strings (switched to Grafana's ${var:sqlstring} safe-quoting); and RDS log panels required the ARN form for logGroups.
Full-stack ATS honesty pipeline: Skill Evidence Ledger, 3-tier ATS coverage, and code-grounded case studies
End-to-end job-application intelligence: JD-extractor produces structured signal → KB-matcher research agent assesses evidence → 3-tier ATS keyword coverage (ontology + embedding tiers) → Skill Evidence Ledger (per-tool, file-cited, verified/partial/gap) → F-pattern resume validator → Haiku rewrite loop → deterministic resume guard. A self-improving role ontology learns from JDs and auto-promotes at quorum. Case-study generation now uses code-grounded component kinds (deterministic classifier for backend/frontend/infra/ml), derives DepthMarkers from fileClass lane counts, and surfaces most-changed-files as evidence in sourceSignals — all visible in the heavily-churned TechnicalWorkspace.tsx.
GitHub repo rename/transfer handled end-to-end with immutable ID re-keying and webhook self-heal
A production incident (a repo rename) exposed that all repo-scoped data was keyed on mutable full_name strings. I built the full rename-safety stack: a nullable github_repo_id column with backfill resolving by name via 301-follow, dual-write on connect and all five hot-path tables (document_embeddings, repo_sync_state, repo_file_state, repo_commits, repo_pull_requests), a webhook handler for repository.renamed/transferred events using an idempotent single-transaction reconcileRepoName, connect ON CONFLICT flipped to github_repo_id with legacy name-unique constraint dropped, and an ingestion worker self-heal that resolves current full_name by immutable ID at sync start.

Challenges

The bedrock:Rerank IAM action was silently failing — every KB query fell back to cosine top-K retrieval with no user-visible error. The pipeline Pod Identity role's policy scoped bedrock:Rerank to foundation-model/* + inference-profile/* ARNs, but bedrock:Rerank does NOT support resource-level permissions in IAM. Confirmed via simulate-principal-policy: InvokeModel on the same ARN returned 'allowed'; Rerank returned 'implicitDeny / MatchedStmt:null'. A separate issue had the reranker pointed at cohere.rerank-v3-5:0 in us-east-1 whose AWS Marketplace agreement was NOT_AVAILABLE in the account.
Granted bedrock:Rerank on '*' (the only valid form per the IAM service-authorization reference) while keeping InvokeModel ARN-scoped — deployed via CDK in tucaken-infra. Verified the first-party amazon.rerank-v1:0 in us-west-2 as ACTIVE + AUTHORIZED with agreement AVAILABLE via live Rerank API test, and defaulted the reranker to that model and region with RERANKER_MODEL_ID / RERANKER_REGION overrides for flexibility.
Chunk enrichment at production scale was prohibitively expensive: a large repo cost roughly $3.65 per run with $3.49 coming from over 2,500 Haiku calls, the 267-term canonical vocabulary prompt was re-sent on every call, and force-reindex of an unchanged repo re-invoked Haiku for every byte-identical chunk. Large repos also exceeded the 15-minute pod budget — the live ai-applications repo stopped early with 1,964 chunks still pending; kubernetes-bootstrap had 941 remaining — leaving roughly half the corpus without canonical skills.
Deployed a multi-layer cost reduction: content-hash dedup cache stores content_hash→skills per user+model so a cache hit copies skills with no model call; chunk-packing batches many chunks per Haiku call under the same system prompt for a several-times call reduction; the cache is method-aware (modelId#canon:N key) so flipping to canonical never serves stale free-text. The pod deadline was raised from 15 to 30 minutes with INGESTION_DEADLINE_SECONDS env aligned, giving 27 minutes of enrichment budget for large repos in one pass. Verified live that the 500-chunk re-enrich had been logging pool-close errors on every cost record — fixed with bookCost()/flushCosts() awaited before every pool close.
Grafana dashboards in kubernetes-bootstrap suffered multiple data-correctness bugs simultaneously: the RDS dashboard's db_identifier was a freetext variable that resolved to empty on stale browser sessions, blanking all CloudWatch panels and erroring the logs panel; per-user/per-repo spend panels interpolated dashboard variables directly into SQL strings, enabling SQL injection via variable inputs; RDS log panels used the name form for logGroups but Grafana's CloudWatch backend requires the ARN form; and the pgvector index-health query referenced non-existent column names.
Converted db_identifier to a single-option custom variable pre-selected to the live RDS instance identifier so it can never be empty. Fixed logGroups panels to use the ARN form (verified via Grafana's /api/ds/query). Corrected pgvector index-health SQL column names (indexrelname, relname, pg_relation_size(indexrelid)). Switched all spend panels to Grafana's ${var:sqlstring} safe-quoting format as a defense-in-depth measure. Documented a dedicated read-only grafana_ro Postgres role in DATASOURCE-HARDENING.md as a further hardening step.
Multi-repo case-study generation was hitting context-window limits with two large repositories. The incremental refine loaded every repo's commits, PRs, and KB chunks even when only one new repo was being added — observed at around 204k input tokens on a live refine run. Separately, when first persisting generated case studies, regenerating never pruned stale rows — the reference project accumulated 10 highlights and 10 challenges (the schema caps each run at 5), two stacked generations with near-duplicate text.
Scoped evidence on refine runs: scopeEvidenceToRepos() drops commits/pulls/kbChunks for repos other than newly-added ones since old repos' sourceSignals are already in the prior case study. Added a COVERAGE rule to the refine prompt ensuring newly-added repos appear in at least one highlight and one challenge. Fixed case-study persistence to reconcile each section to exactly the current run — inserting new content-hashes and pruning stale ones — while never touching user-authored NULL-content_hash rows.
Karpenter was evicting long-running coach/strategist/ingestion Job pods mid-Bedrock-call during node consolidation. The Job completed with no coaching_content written and pipeline_runs status stuck at 'coaching', causing the UI to spin forever. A compounding schema bug — sectionOrder added to the persona output without updating the separate strict TailoredResumeSchema — caused every analyse run to fail AFTER the roughly six-minute Sonnet writer call. With backoffLimit:2 set on model Jobs, each deterministic failure re-spent Sonnet up to three times.
Annotated the shared pod template with karpenter.sh/do-not-disrupt so Karpenter won't voluntarily evict batch Jobs while running. Added a SIGTERM handler to mark the run 'failed' before exit within K8s's grace period. Fixed sectionOrder in TailoredResumeSchema and hardened it to .strip() rather than .strict() to absorb future additive schema drift. Centralised backoffLimit to MODEL_JOB_BACKOFF_LIMIT=0 across all dispatch sites so no model Job re-spends on deterministic failures.

Key Decisions