When an AI cites a paper, does the paper exist?

Every model gets the same questions and a protocol that lets it decline a citation. Each citation it does give is resolved by sourcecheck, an open-source source-integrity gate, against OpenAlex and Crossref. Does the DOI resolve to a real work? A DOI that resolves to nothing is a fabrication, and resolution is precision-first, so there's no fuzzy fallback to a paper that merely looks similar. Has the work been retracted? Checked against OpenAlex's retraction flag and Crossref's retraction notices, sourced from Retraction Watch. Both are lookups.

No language model grades anything in the headline path, and that's deliberate. For these two checks a language model is the worst possible judge: it's the thing being checked. It hallucinates DOIs, and it can't know a retraction that happened after its training cutoff. A registry can. The presence check, whether the claim is actually in the resolved source, is the one axis that shades into judgment, so it ships as an experimental secondary and never enters the headline.

These resolved to real papers that have since been retracted for data fraud or error. Models cited them, often with no note that they were withdrawn, because the retraction is newer than the training data. Even models that fabricated nothing did this. sourcecheck flagged every one. That gap, the model cites it and the registry knows it's dead, is the whole reason the library exists.

sourcecheck checks whether a claim is sourced, not whether it's true. A real, non-retracted paper can still be cited for something it never said. The library verifies the citation exists, that it hasn't been retracted, and (experimentally) that the claim is present in it, and it doesn't adjudicate truth. That scope is the design: these are exactly the checks a language model can't make for itself.

On fairness: fabrication rate is the ranking, but retraction is a safety showcase, not a pure penalty. Asking which study reported the Surgisphere hydroxychloroquine result requires naming a retracted paper to answer correctly, so citing it isn't itself a failure. The value is that sourcecheck surfaces the retraction the model omitted. One politically charged vaccine-safety retraction is deliberately excluded from the question bank and every example; retraction coverage comes from data-fraud and error cases only. Scorer validation for the experimental presence axis is in progress, so the board wears a beta label until it ships.

Fabrication rates at scale are well covered: GhostCite verified 375,440 citations across 13 models, and a Clemson-led cross-model audit ran registry checks over ten models. On retractions, Thelwall tested three open-weight models against 161 high-profile retracted articles (over 80% claimed the paper wasn't retracted), and his follow-up generated 6,510 ChatGPT quality reports on retracted or discredited work; not one mentioned a retraction.

What this board adds: the pre/post-training-cutoff split, on frontier commercial models, re-run as new models release, with deterministic registry verification and a per-citation audit trail. A model can't have memorized a retraction that postdates its training data, so the post-cutoff column can't be gamed by memorization. That split is the number to watch here.

What this board measures in the lab keeps landing in production: Deloitte refunded a government report, EY pulled one, KPMG had 40 of 45 citations exposed, South Africa withdrew a draft national policy, and courts across 41 countries have put over 1,700 cases of AI-hallucinated material on the record. The incidents board aggregates all of it, searchable, refreshed automatically every week, every entry linking its source document.

mikias.io/citations/incidents →