Preventing Agent Hallucination: Grounding Verification and Deterministic Guards

TL;DR

RAG retrieval does not guarantee truthful output. A model can receive six relevant chunks and still invent a claim that appears nowhere in them. In my Bedrock-powered resume generation and portfolio chatbot pipelines, hallucination prevention runs after generation, in two layers. The first is a grounding verifier: Claude Haiku in tool-use mode, forced to return exactly one enum value, GROUNDED or NOT_GROUNDED, for every candidate answer. The second is a chain of deterministic guards: pure TypeScript functions that demote vendor claims backed only by example docs, reconcile stale documentation against actual code, and strip any metric the model invented. Neither layer relies on prompt wording. Both are reversible, logged, and unit-tested.

The Problem: Hallucination Survives Good Retrieval

Retrieval-augmented generation retrieves relevant content. It does not guarantee that the model uses only that content.

The failure mode I kept hitting was subtle. The retrieval pipeline would return accurate chunks. The model would read them, synthesise a plausible-sounding answer, and then elaborate slightly beyond what the chunks actually said. For a general-purpose chatbot, this is annoying. For a resume generation agent, it is a credibility-destroying error.

Two concrete examples from this codebase:

Vendor fabrication. The job-strategist agent matched a skill to a candidate's documentation. The doc contained an OpenAI Python example snippet, written as a reference, not as production code. The matcher marked the vendor verified. The writer stated it as first-person production experience. The resume now claimed the candidate worked with OpenAI in production. They never had.

Stale-doc drift. A repo migrated from self-hosted Kubernetes to EKS. The README was not updated. The knowledge base ingested the stale README. The matcher cited the README as current. The code extractor already knew the truth. Nothing in the pipeline reconciled the two.

The instinct is to fix the prompt. Lower the temperature, strengthen the instruction, add "never invent a metric" to the system message. This helps at the margins. The generation side in this project already runs at a low temperature of 0.3. Hallucinations still happen. Instructions only shift probabilities; a deterministic safeguard removes the failure outright.

The asymmetry is the design driver. Blocking a grounded answer is recoverable: the user sees a fallback message and tries again. Leaking a hallucination erodes trust in a way that is harder to repair, especially when the audience is a recruiter reading a resume. Preventing that leak, rather than reducing its odds, is what the two layers below are for.

Architecture: Two Layers After Generation

The fix is straightforward to describe. Hard to get right in the details.

The first layer is the grounding verifier. It runs post-generation, taking the candidate answer and the retrieved source chunks, and asks Claude Haiku a single question: is every claim in this answer directly supported by these chunks? The verdict comes back through Bedrock tool use. Tool use is the Converse API mechanism that forces the model to respond by calling a declared tool with a schema-validated payload. The verdict arrives as a typed enum instead of free text.

The second layer is the deterministic guard chain. The resume generation pipeline has a fabrication surface the verifier does not cover: the LLM rewrites text, and rewriting invents things. Three pure TypeScript functions run after the matcher and before the strategist writer, each targeting one fabrication mode. A prompt that says "never invent a metric" is probabilistic. The backstop is not.

The chatbot runs the verifier in mode: 'block' by default, replacing an ungrounded answer with a fallback message. A warn mode exists that annotates without blocking. Block is the right default for a recruiter-facing surface: an accurate answer matters more than a complete one.

Implementation: The Verifier and the Guards

The verifier prompt asks for per-chunk support explicitly. It shows the model the numbered source chunks and the generated answer. It then asks whether every claim is directly supported, and requests the verdict, a brief reason, and any unsupported claims (applications/shared/src/grounding/bedrock-grounding-verifier.ts, lines 44-58). A claim that is valid but spread across two adjacent chunks is a known edge case: the verifier sees no single supporting chunk and marks it NOT_GROUNDED. The verdict is technically correct per its instructions, but the answer was truthful. I keep the stricter form in production and handle boundary claims through retrieval tuning instead. The relaxed alternative, counting claims that the chunks support collectively, invites Haiku to stitch unrelated chunks together to back a hallucinated claim.

Retrieval tuning is the complement. The knowledge base uses HIERARCHICAL_TITAN chunking with a 60-token overlap (lines 182-194 of infra/lib/stacks/bedrock/kb-stack.ts). When claims span chunk boundaries, the overlap is what brings adjacent context into the same window. Multi-query retrieval defaults to TOP_K=8; lifting it to TOP_K=12 brings adjacent chunks into the verifier's view at the cost of more context tokens per call. I treat both as dials to adjust when the false-positive rate on the GroundingFalsePositive CloudWatch metric climbs.

Vendor provenance (vendor-provenance.ts) demotes any vendor match backed only by reference or example documentation rather than authored production work. This is the OpenAI snippet problem from the opening: the guard inspects match provenance and downgrades matches sourced exclusively from example files.

Code truth (code-truth.ts) reconciles documentation claims against the deterministic code-derived technology truth. If the README says self-hosted Kubernetes but the code extractor found EKS, the guard marks the doc claim stale. The code extractor is the source of truth.

Number provenance (number-provenance.ts) is a strict net over any AI-rewritten text. Every number in the output must exist in an allowed set: numbers from the original resume plus numbers extracted from grounding facts. Nothing outside that set survives in experience highlights, summary, or key achievements.

1// applications/job-strategist/src/ats/number-provenance.ts
2export function stripUngroundedNumbers(
3  text: string,
4  allowed: Set<string>
5): string {
6  return text.replace(/\b\d+(?:[.,]\d+)?\b/g, (match) =>
7    allowed.has(match) ? match : '[NUM_REDACTED]'
8  );
9}

All three guards are pure functions with no side effects. They are unit-tested independently. They log every redaction for audit. They run in sequence in the pipeline entrypoint, and reversing any one of them is a one-line change.

Challenge Log

The Verifier That Blocked Grounded Answers

The first version of the verifier asked Haiku to reply with GROUNDED or NOT_GROUNDED as plain text on the first line, then parsed the response with a regex. It blocked answers that were perfectly grounded. No errors. Just fallback messages where real answers should have been.

The root cause was free-text output. It fails in two distinct ways. Verdict-token placement: Haiku sometimes writes a chain-of-thought paragraph first, mentions both tokens in the reasoning, then gives its actual verdict, and a regex that scans the full response finds both tokens and falls through to the fail-safe. Unparseable output: Haiku occasionally substitutes synonyms, "supported", "yes", "consistent", none of which match the expected tokens, so the parser defaults to NOT_GROUNDED. The fail-safe fires, the answer is blocked, and it was actually grounded. A false positive created by the parser rather than the model's judgement.

The fix is tool-use enforcement:

1// applications/shared/src/grounding/bedrock-grounding-verifier.ts
2const command = new ConverseCommand({
3  modelId: 'eu.anthropic.claude-haiku-4-5-20251001-v1:0',
4  messages: [{ role: 'user', content: [{ text: verifierPrompt }] }],
5  toolConfig: {
6    tools: [{
7      toolSpec: {
8        name: 'grounding_verdict',
9        inputSchema: {
10          json: {
11            type: 'object',
12            properties: {
13              verdict: { type: 'string', enum: ['GROUNDED', 'NOT_GROUNDED'] },
14              reason: { type: 'string' },
15              ungroundedClaims: { type: 'array', items: { type: 'string' } },
16            },
17            required: ['verdict', 'reason'],
18          },
19        },
20      },
21    }],
22    toolChoice: { tool: { name: 'grounding_verdict' } },
23  },
24  inferenceConfig: { maxTokens: 400 },
25});

When toolChoice forces a specific tool, the model must call it. The schema enforces enum: ['GROUNDED', 'NOT_GROUNDED']. Unparseable output stops being a runtime possibility and becomes a schema violation the API rejects. The cost is slightly more Bedrock payload for the tool-use bytes; the reliability gain makes that the right trade for any surface where honesty is non-negotiable.

Diagnosing a suspected false positive is a three-step path. First, filter the calling Lambda's log group in CloudWatch Logs Insights for grounding-verifier lines. A successful pass does not log. An unparseable-output run logs the first 200 characters of the model output before defaulting to NOT_GROUNDED. Second, replay the verifier call directly against Bedrock Runtime with the original answer and chunks. Inspect where the verdict landed in the response. Third, check the calling mode where the chatbot handler constructs the verifier: block mode replaces the answer, warn mode annotates it. The GroundingFalsePositive metric is emitted whenever an operator overrides a NOT_GROUNDED verdict during review. If that count rises while the logs show clean tool-use output, the cause is chunk boundaries. Investigate retrieval overlap and TOP_K rather than the verifier logic.

Transferable value: any LLM verdict that gates production output should arrive through a forced tool call with an enum schema, never through free text and a regex. The failure mode is silent, the parser fails safe, and the fail-safe itself becomes your false-positive generator.

Junior Corner

Why do you need a grounding verifier if retrieval already found the right chunks?

Think of a courtroom. Retrieval is the discovery process: it finds the relevant documents and puts them in front of the witness. Grounding verification is the cross-examination: it checks that every statement the witness makes is actually in those documents. A witness with perfect documents can still embellish. The model could read six perfect chunks and invent a seventh claim from nowhere. Discovery does not catch that. Cross-examination does.

Retrieval is the database query: given a question, find the most relevant chunks from the knowledge base. RAG does this well. Grounding is the audit: given those passages and the model's answer, check whether every claim is actually supported by them.

Tool-use enforcement is the key engineering insight. Free-text output forces you to extract the verdict yourself and trust the model to format it correctly. Models sometimes do not. With toolChoice: { tool: { name: 'grounding_verdict' } } and an enum schema, the model cannot respond in any other shape. It is a typed function signature instead of a string parameter: you could accept "grounded" as a string and parse it at runtime, or you could accept 'GROUNDED' | 'NOT_GROUNDED' as a union type and let the compiler enforce it. Tool-use enforcement is the compiler.

Where This Applies

The two-layer pattern applies anywhere an LLM writes user-facing claims in a high-stakes context.

Resume generation from GitHub portfolios. Vendors, skills, and metrics must be truthful. A candidate who lists fabricated production experience faces rejection at the interview stage, or worse, after hire.

Portfolio chatbots for recruiters. A recruiter asking about a candidate's AWS experience and receiving a hallucinated answer loses trust in the entire portfolio. The grounding verifier in block mode is the right default here.

Self-healing infrastructure agents. The self-healing agent in this codebase uses write-and-verify enforcement: after any remediation tool is called, a [REFLECT] prompt asks the model to assess success before proceeding, and a [VERIFICATION REQUIRED] injection forces at least one verification tool call before the final report is allowed. The same asymmetry applies: claiming a fix was applied when it was not is worse than saying nothing.

The operational cost is small and the architecture is not complex. The guards are small functions. The verifier is one API call per answer. The cost of not having them scales with how many users trust the output.

FAQ

What is the difference between retrieval and grounding verification?

Retrieval finds the most relevant chunks for a question; it runs before generation. Grounding verification audits the generated answer against those chunks; it runs after generation. A pipeline can have excellent retrieval and still emit hallucinations, because the model is free to elaborate beyond what the retrieved chunks say.

Why force tool use instead of parsing the model's text reply?

A free-text verdict has to be extracted with string matching, and the model controls the string. It can bury the verdict after reasoning, or substitute synonyms the parser does not expect, and every parse failure triggers the fail-safe and blocks a good answer. A forced tool call with enum: ['GROUNDED', 'NOT_GROUNDED'] makes the verdict schema-validated at the API layer, so the parser has nothing to guess about.

What happens when the verifier blocks an answer that was actually correct?

The person asking gets the fallback response and can simply ask again, which is the recoverable side of the failure asymmetry. Operationally, a manual override during review emits the GroundingFalsePositive CloudWatch metric, and a rising count points at chunk-boundary effects, tuned via chunking overlap and TOP_K, rather than at the verifier logic.

Lessons and Next Steps

The most durable lesson from building this system is about failure-mode asymmetry. False positives, blocking a grounded answer, are annoying. False negatives, leaking a hallucination, damage trust in ways that accumulate invisibly until they surface in the worst possible context. Design fail-safe defaults around that asymmetry rather than around minimising user friction.

The deterministic backstop pattern shows up in every AI writing surface I have built: the grounding verifier, the number-provenance guard, the write-and-verify loop in the self-healing agent. The shape is always the same, a probabilistic generator followed by a deterministic check the model cannot talk its way past. Learning to design these checks, rather than chasing better prompts, is what separates production AI engineering from demo-day AI engineering.

Preventing hallucination in production is less about a better model and more about the guard you put after it. Next: deploy tool-use enforcement universally across all grounding surfaces, land the IaC value-scanner and the code-comment prose scanner to close the multi-word AWS service phrase gap that currently requires LLM fallback, and expand the GroundingFalsePositive metric into a dashboard that ties detector performance directly to chunking strategy changes. Building this positions me to evaluate any AI feature a team is considering shipping with one question first: what is the worst-case output, and is there a deterministic guard between the model and the user?

Ask the Lami chat assistant directly via the chat widget on the site. Every answer it gives you has already been through this exact verifier.