My research agent fabricated 4 of 5 citations, so I built a verification gate

I run an extraction pipeline that reads research papers and pulls out structured claims. Thousands of papers go in, claims with citations come out. The citations are the whole point: a claim without a real source is just the model's opinion.

One evening I decided to spot-check five citations by hand. I wasn't worried. The output looked immaculate: author names, years, journals, the texture of real scholarship.

Four of the five were wrong. One paper didn't exist. One existed but said something different from what was attributed to it. Two had real authors attached to titles those authors never wrote. If I'd published that batch, anyone who checked my sources would have been right to never trust the project again.

Nothing about the output looked suspicious, because fabricated citations look exactly like real ones. A language model produces the most plausible-looking thing, and a citation reads as plausible when it's well-formed, whether or not the paper exists.

“Citation 1”

SUPPORTED

The one that held up: real paper, matching metadata.

“Citation 2”

NOT VERIFIABLE

The paper doesn't exist.

“Citation 3”

CONTRADICTED

The paper exists but says something different from what was attributed to it.

“Citation 4”

NOT VERIFIABLE

Real author, attached to a title they never wrote.

“Citation 5”

NOT VERIFIABLE

Real author, attached to a title they never wrote.

Fixing the prompt doesn't fix it

My first instinct was to fix the prompt. Be more careful, only cite real papers, double-check your sources. If you've built anything with language models you know how that story ends: the fabrication rate goes down and the confidence goes up. The problem gets worse, because now the errors are rarer and you've stopped looking for them.

It took me a day to accept that being more careful in the prompt wasn't going to fix it. If a property of the output has to hold every time, something other than the model has to enforce it. So I stopped asking the model to be trustworthy and built a gate.

The verification gate

Between extraction and publication sits a checkpoint no claim can skip.

1

Parse the citation into structured fields

Title, authors, year, identifier.

2

Check it against an authority that can't hallucinategate

In my case the OpenAlex and Crossref APIs, which index the real scholarly record.

3

Pass only on a full matchgate

The paper exists, the metadata matches, and the identifier resolves.

4

Quarantine anything that fails, with the reason attachedgate

A human looks at the quarantine queue, not the happy path.

5

Stamp what passed

Every published claim carries its verification timestamp, so I can audit it later.

The model can still be wrong all it wants, it just can't publish. That changed the economics of the whole pipeline. Before, every output needed suspicion, which meant slow manual review of everything.

Now my attention goes only where the gate says it should. The check doesn't make the model better, it makes the model's failures cheap.

The pattern underneath

Once I saw it I started seeing it everywhere. Any time a wrong output is expensive, something with no imagination should stand between the model and the world. In my fitness app the meal numbers are computed by plain code and the model only chooses between validated options, because a hallucinated macro total is a lie someone eats. Same pattern, different domain.

This is now the first question I ask about any AI system I'm building or reviewing: where's the gate? What property has to always hold, and what enforces it, given that the model can't? If the answer is "the prompt asks nicely," the system isn't done.

Build the gate before you scale

If you ship anything where citations, numbers, identifiers, or legal references come out of a model, build the gate before you scale the pipeline. Mine took a weekend.

Parse the load-bearing fields out of the output, verify them against a source of truth that can't hallucinate, quarantine failures with their reasons, and stamp what passed. Your authority might be Crossref, a price database, a statute index, or your own ledger. Anything but the model.