Your AI Assistant Is Lying: Mapping Jagged Intelligence in Investigations

Your AI assistant just confidently cited a court case that does not exist, and you only discovered the fabrication because your editor asked for the docket number. This is not a glitch or a rare malfunction; it is the defining characteristic of current generative models. The consensus advice tells you to simply "verify everything," but that platitude ignores the economic reality of modern newsrooms facing severe funding scarcity. Blind trust leads to retractions, yet total rejection of automation makes deep investigation impossible in an era of shrinking resources.

The problem is not that AI is stupid. It is that AI is jagged. My own experience running autonomous research teams shows that these systems excel at summarizing dense regulatory filings while simultaneously inventing plaintiff names in routine civil litigation. We cannot automate verification safely until we stop treating intelligence as a smooth gradient and start mapping it as a rugged terrain with predictable cliffs. This post translates academic theory into a field-tested framework for ai verification workflows journalists can actually use on deadline.

What is jagged intelligence?

Jagged intelligence is the phenomenon where generative AI systems display highly uneven performance across tasks that appear semantically nearby, excelling at one prompt while failing confidently on another with only minor context changes. Researchers Joshua Gans and colleagues formally define this as Artificial Jagged Intelligence (AJI) in their economic model of AI reliability, submitted in January 2026. Unlike traditional software bugs, AJI means local error rates vary wildly even when global quality signals remain high.

This definition matters because it destroys the assumption that a model passing a bar exam or coding test will reliably verify a county clerk record. The paper treats truth as a rough Brownian process where the model knows scattered points drawn from a Poisson process but must interpolate between them. In investigative reporting, those interpolation gaps are exactly where sources get misidentified and legal claims get invented. Understanding jagged intelligence meaning requires accepting that capability is not a single dial you can turn up; it is a topographic map with safe valleys and deadly sinkholes.

Newsrooms pressured by economic necessity often adopt AI tools prematurely to survive funding cuts, creating a dangerous tension between efficiency and accuracy. The Global Investigative Journalism Network notes that reliable funding sources rarely match the growing need for investigative reporting. When budgets shrink, the temptation to let AI handle source verification becomes overwhelming. But AJI means that blind automation is not just imperfect—it is structurally hazardous in ways that standard quality assurance misses. A calibrated user who conditions on local uncertainty can still extract positive expected value, but only if they know precisely where the ground is unstable.

How do ai verification workflows journalists use to map risk?

Effective ai verification workflows journalists employ treat AI outputs as probabilistic samples requiring targeted human validation at specific high-risk nodes rather than as final drafts. This approach translates the abstract concept of AJI into a tactical Verification Heatmap that categorizes investigative sub-tasks by their failure predictability. Instead of asking "is this AI good?", we ask "is this specific sub-task within the model's safe interpolation zone?" The answer determines whether a human checks every output or only audits statistically significant samples.

Building the Verification Heatmap

We developed our heatmap by tracking failure modes across hundreds of research tasks. The pattern that emerged contradicts most vendor marketing: tasks involving structured data extraction from unstructured text are often safer than simple factual lookups in niche domains. Entity resolution in corporate filings tends to be lower risk because the training data contains millions of similar examples. Conversely, verifying obscure local court records or drafting FOIA requests for specific municipal agencies carries extreme jaggedness risk because the model has few anchor points to interpolate from.

Task Type	Jaggedness Risk	Required Human Validation
Summarizing federal appellate rulings	Low	Spot-check citations and holding
Drafting FOIA requests for state agencies	High	Verify statute numbers and recipient titles manually
Entity resolution in SEC filings	Medium	Cross-reference against official registry
Verifying local county court dockets	Extreme	Mandatory primary source confirmation for every claim

This heatmap is the operational core of ai hallucination prevention journalism. Notice that "extreme" risk does not mean "do not use AI." It means the AI serves only as a lead generator, never as a verifier. For low-risk tasks like summarizing well-documented federal cases, human oversight shifts from line-by-line review to statistical sampling. This differentiation is what allows resource-constrained desks to scale without collapsing under retraction pressure. The goal is matching validation intensity to local posterior variance, not applying a blanket skepticism that burns out reporters.

Designing Fail-Safe Human Loops

Human oversight ai newsroom tools function best when integrated directly into the generation pipeline rather than applied as an afterthought. We embed validation checkpoints at the exact moments where AJI predicts highest uncertainty. When an AI system drafts a FOIA request, our workflow automatically flags any statute citation that lacks a corresponding hyperlink to an official government domain. The reporter cannot proceed without either confirming the link or marking the citation as unverified. This friction is intentional; it forces engagement with the jaggedness rather than allowing smooth-sounding prose to bypass critical thinking.

For entity resolution tasks, we require dual-source confirmation before any name enters a published story. If the AI identifies a shell company director from leaked documents, a human must independently locate that person in at least one other primary record set. This mirrors the methodology of organizations like CLIP, which connects databases across 21 Latin American countries through its NINA platform to cross-verify claims. As noted in their 2026 Free Media Pioneer Award recognition:

"In creating CLIP, Ronderos aimed to bring together investigative journalists across Latin America to tackle stories far bigger than any one country."

That collaborative verification model is essentially a human solution to artificial jagged intelligence. No single database or model covers the entire terrain. Cross-referencing across jurisdictions and sources smooths out the local errors that doom isolated automated systems. Our internal workflows now mimic this distributed validation approach even for domestic investigations, treating each AI output as one node in a larger verification network rather than an authoritative endpoint.

What tools support investigative reporting ai reliability?

Reliable investigative reporting ai reliability depends less on model selection and more on integrating external ground-truth sources that constrain AI interpolation. Tools like the Google Search Console API help monitor how indexed content performs over time, revealing whether AI-assisted research maintains search visibility or triggers algorithmic penalties for low-quality synthesis. County Clerk Public Access Portals serve as non-negotiable primary sources for local court verification, providing the scattered truth points that models lack internally.

FOIA Request Trackers maintain institutional memory of agency response patterns, helping reporters identify when AI-generated request language deviates from historically successful phrasing. Spreadsheet-based Verification Logs create audit trails that document every human check performed on AI outputs, which is essential for defending editorial decisions during legal challenges. These tools do not replace judgment; they structure it. They transform vague concerns about AI reliability into discrete, trackable validation events that can be reviewed and improved over time.

We avoid recommending specific large language model vendors because the jaggedness problem persists across all current architectures. Whether you access models via Anthropic API, OpenRouter, or Networkr, the fundamental topology of capability remains uneven. The tool choice matters less than the verification architecture wrapped around it. A superior model used without a heatmap will still fail catastrophically on niche tasks, while a mediocre model constrained by rigorous human-in-the-loop protocols can produce publishable work safely.

How does scar tissue prove the verification gap compounds?

Our own publishing data demonstrates that small verification gaps compound over time, degrading archive integrity even when individual articles pass initial editorial review. This site has published 89 articles in the last 90 days, each subjected to varying levels of AI assistance and human validation. Google URL Inspection shows 47% of this site's 75 pages that have been live at least 14 days are indexed, a rate that reflects both technical SEO factors and content quality signals. Median time from publish to confirmed Google indexing on this site is 7 days, across 48 posts we measured.

Google Search Console recorded 1,149 search impressions and 5 clicks for this site across 13 weeks. Those numbers tell a story beyond traffic volume. The impression-to-click ratio suggests that while our content appears relevant to search queries, something prevents conversion. Part of that gap stems from the inevitable friction of writing about AI verification while practicing it. Early in our experimentation, we trusted AI summaries of technical papers too readily. Several posts required substantial revision after readers flagged interpolated claims that sounded plausible but misrepresented source methodology. That scar tissue now informs every heatmap we build.

The compounding effect is subtle. A single unverified entity name in an article about crypto regulation might escape notice initially. But when future AI-assisted research cites that article as a source, the error propagates. We documented a similar dynamic in our analysis of abandoned repositories as supply chain risks, where outdated code references created cascading vulnerabilities. In journalism, the cascade is epistemic rather than technical, but the structural pattern is identical. Verification debt accumulates interest.

This is why our editorial methodology now mandates retrospective audits of AI-assisted content at regular intervals. We cannot assume that passing initial review guarantees long-term accuracy. The jaggedness of AI means that errors are not randomly distributed; they cluster in predictable domains that may only become apparent when multiple articles interact. Monitoring our public audit feed reveals patterns that individual article reviews miss, showing where our heatmap needs recalibration based on actual failure data rather than theoretical risk assessment.

Can we ever fully automate source verification?

Full automation of source verification remains impossible under current AI architectures because jaggedness is a structural feature of probabilistic interpolation, not a temporary limitation awaiting more compute. The economic model of AJI demonstrates that even optimally calibrated users face irreducible local uncertainty in domains lacking dense training data. For investigative journalism, this means the question is not when AI will achieve perfect verification, but how to permanently design workflows that account for permanent imperfection.

The open question for newsrooms is calculating the threshold where human verification costs outweigh AI efficiency gains for different record types. Federal appellate summaries might justify heavy automation because the cost of error is moderate and the training data is rich. Local police misconduct records might never justify automation because the cost of error is catastrophic and the data is sparse and fragmented. This calculus varies by beat, by jurisdiction, and by publication risk tolerance. There is no universal answer, only local optimization under constraint.

Some researchers argue that scaling laws will eventually smooth out the jaggedness. I remain skeptical based on current evidence. The Brownian motion model suggests that adding more data points reduces global error but does not eliminate local variance. New capabilities create new interpolation zones with new failure modes. As models gain ability to synthesize across languages or modalities, they also gain new ways to confidently fabricate connections that never existed. The jaggedness evolves; it does not disappear.

This reality demands humility from both AI vendors and newsroom leaders. Vendors must stop marketing smooth intelligence curves and start publishing detailed reliability maps for specific professional domains. Newsrooms must stop treating AI adoption as a binary switch and start treating it as a continuous calibration exercise. The organizations that thrive will be those that build institutional muscle for navigating jagged terrain, not those that wait for the terrain to flatten.

Here are two concrete experiments you can run this week to ground this framework in your own workflow:

• Blind Verification Test: Ask your current LLM to verify 10 obscure local court records relevant to your beat. Manually check each against the official county clerk database. Calculate the error rate and compare it to the heatmap predictions above. If your observed failure rate diverges significantly from the expected risk level, your mental model needs updating.
• Workflow Mapping Exercise: List every step in your current investigation process. Label each as High Jaggedness Risk or Low Jaggedness Risk based on AI performance consistency in your domain. Identify the three highest-risk steps that currently lack mandatory human validation. Design a checkpoint protocol for those steps before your next major story ships.

Stop chasing flawless automation. Start designing fail-safe human validation steps that respect the jagged reality of artificial intelligence. Your credibility depends on it.

MOBILIZR -- Writing at mobilizr.org

1. Step 1: Audit your investigative workflow for 'jaggedness hotspots' where AI confidence exceeds actual accuracy.
2. Step 2: Implement a 'Two-Source Rule' for all AI-generated facts involving public records, requiring independent human confirmation.
3. Step 3: Use AI for draft generation and pattern recognition, but restrict it from final source citation without manual docket cross-referencing.
4. Step 4: Create a 'Hallucination Log' to track specific types of AI errors in your niche, building a custom reliability map over time.
5. Step 5: Train reporters on 'prompt engineering for verification,' focusing on asking AI to identify uncertainty rather than just answers.