Primer AI
Making AI Verifiable for Intelligence Analysts
Claim-level verification that lets an analyst scan an AI summary, trust it, and recover anything verification removed.
Overview
- Context
- Analysts needed AI to digest overwhelming volumes of intelligence, but a single hallucinated claim could compromise a mission, and the industry standard of sentence-level citation hides errors at the claim level.
- Decision
- I designed RAG-V to break every AI summary into individual claims, verify each against source evidence, and present the result so an analyst could scan it, trust it, and recover anything verification removed.
- Impact
- RAG-V was cited as best-in-class explainability in competitive evaluations, every customer asked to extend it beyond its original use, and the inline source-context pattern it introduced became the platform standard for AI-generated content.
The Challenge
Hallucination is a non-starter.
Zero Tolerance for Error
A standard error rate is disqualifying in high-stakes intelligence, where a single fabricated entity can derail critical assessments.
AI Speed vs. Hallucination
Generative models drastically accelerate the reading process for massive data volumes, but analysts rightly distrust the unverified, hallucination-prone output.
The Manual Verification Bottleneck
The time-consuming process of manually cross-referencing footnotes against original sources completely cancels out the speed AI was supposed to provide.
Data is illustrative, based on trends from Ke et al. (2025), Retrieval augmented generation for 10 large language models and its generalizability in assessing medical fitness, Nature.
Research
Where Every Citation Pattern Breaks
Academic citation
Grounds claims in footnotes and inline references, a familiar and credible convention.
Analyst workflows
Lean on the same citations but add excerpts and metadata, organization, author, and timestamp, to validate and share what they find.
LLM products
Like Perplexity and Gemini attach hoverable chips to sentences.
Critical Finding
A citation proves a sentence has a source. It does not prove every claim inside that sentence is true.
A single sentence can carry several claims with different verification statuses. Sentence-level citation treats it as one unit, so a hallucination hides in plain sight, attached to a perfectly real source.
Strategy
Verify the Claim, Not the Sentence
One sentence, several claims
Take a single generated sentence. Sentence-level citation verifies it as one unit; RAG-V decomposes it into the separate claims it actually makes, then checks each on its own.
“Israeli forces are engaged in heavy fighting with Hamas in the northern Gaza Strip, and air forces are intensifying their military operations in southern Gaza.”
Ground every individual claim in source evidence and you get trust precise enough for high-consequence work.
Working with applied research, I shaped a pipeline that decomposes generated text into individual claims, cross-checks each against retrieved sources, and flags or regenerates the ones that fail. A human stays in the loop at the end. The system never signs off on itself.
RAG
RAG-V
Capture Market Signal
Prototyped a working pipeline to validate market signals and solve the core issue of AI trust.
Scannable Design
Designed for analysts under pressure to quickly gauge reliability before digging in.
Transparent Reasoning
Exposed the AI's reasoning for passed and failed claims so users kept full context and confidence.
Design
Designing for scannability, transparency, and control
Scannability Over Comprehensiveness
Analysts review dozens of documents at once, so the status marker itself had to carry the signal. I put inline, color-coded verification status directly on the text: verified, unverifiable, contradicted. One status per claim tells the analyst where to look, so they open evidence only where it matters instead of re-reading everything.
Transparency for Trust Calibration
Clicking a claim opens its reasoning: the exact claim, the matching source excerpt with document metadata, and why it passed or failed. Exposing the model's limitations rather than hiding them increased confidence rather than denting it. It trained analysts to treat the AI as a probabilistic partner to be checked, not an oracle to be obeyed.
Mitigating Information Loss
User feedback surfaced a specific fear: that verification would scrub relevant context along with the hallucinations. Correction is lossy. So I designed a safety net. Analysts toggle between the original and corrected versions, see a diff of what changed, and restore anything verification took out. Nothing disappears silently.
The Pivot
Inline Beats Click-Through
The first design still asked analysts to click through to confirm evidence. Testing showed that step was unnecessary friction. So I surfaced the source name and key excerpt inline, in a popover on the claim itself, so an analyst could verify without leaving the summary.
Latency & Data Loss
The verification speed was too slow, and filtering unverified claims meant losing potentially valuable information.
Trust Requires Manual Validation
AI summaries provide great overviews, but analysts inherently require manual verification to build trust in the output.
The realization
That inline pattern mattered more than the full claim-level machinery behind it.The way an analyst confirmed a source, without breaking their read, was the part that every team wanted next.
Result
Trust Became the Baseline
Retrospective
Speed vs. Truth
Transparency Earned Trust
Designing for trust is not about removing uncertainty. It is about making uncertainty visible and giving analysts the tools to act on it. Claim-level granularity gave them precision; color-coded status gave them speed.
Simplify Sooner
The inline source-context pattern was the right answer, and the click-through model before it added friction I only caught in testing. Start closer to the simplest trustworthy version.
Design for Calibration, Not Perfection
The goal was never a perfect summary. It was an analyst who knows exactly how much to trust an imperfect one.