Extracting structured claims from 11,000 papers on a desk, not a datacenter

My extraction pipeline has sourced and triaged 90,111 research papers across its two big corpora (65,074 in consciousness research, 25,037 in eczema treatment), and fully extracted 11,490 of them (2,233 and 9,257) into structured claims, treatments, methods, and findings. It runs on hardware that fits on a desk, with cheap batch APIs picking up what local models can't. Nothing about it is exotic, and that's what this essay is about: at corpus scale, the wins came from plumbing.

Why not just rent a datacenter

The first version of the cost math was sobering. Extracting a single corpus with a frontier model at interactive prices would've cost more than the hardware I already owned, and I wanted to run many corpora, repeatedly, as the schemas improved. Two decisions changed the economics.

First, local inference for the high-volume, low-difficulty work. Page classification, relevance triage, and first-pass extraction just need a competent model running at zero marginal cost, so you can afford to be wrong and rerun.

Second, batch APIs for the work that needs a stronger model. Batch pricing is roughly half of interactive pricing, and extraction is the perfect batch workload: thousands of independent documents, no user waiting. One corpus of 25,037 papers cost about twenty five dollars to triage and extract this way. If your pipeline has a human waiting on the other end you pay interactive prices. If it doesn't, you almost never should.

The stages

Every corpus runs the same five stages, and each stage quarantines its failures instead of passing them along.

1

Acquisition

OpenAlex and PubMed queries plus citation chaining. Everything gets an identifier on the way in.

2

Triage

A cheap model classifies relevance before anything expensive happens. On the largest corpus: 25,037 candidates down to 13,222 relevant.

3

Text extraction

PDFs parse, or they don't. Extraction quality is recorded per document and confidence downgraded accordingly.

4

Claim extractiongate

A strict schema, enforced outside the model. Failing outputs are rejected and retried; unfillable fields stay empty.

5

Entity resolution and publicationgate

Deterministic normalization, conservative merges. Published claims carry source identifier, timestamp, verification status.

Acquisition. OpenAlex and PubMed queries plus citation chaining. Everything gets an identifier on the way in, because every later stage will need to prove where a claim came from.

Triage. A cheap model classifies relevance before anything expensive happens. On the largest corpus this cut 25,037 candidates to 13,222 relevant papers. Triage is the highest-leverage stage in the pipeline: every dollar spent here saves several downstream.

Text extraction. PDFs are the swamp. Some parse cleanly, some are scans, some are forty-year-old typewriter pages. The pipeline records extraction quality per document and downgrades its confidence accordingly, so bad text doesn't get treated as trustworthy text.

Claim extraction. The model reads each paper against a strict schema: claims with statistics, populations, methods, evidence levels, quotes. The schema is enforced outside the model. Outputs that fail validation get rejected and retried, and fields the model can't fill stay empty rather than guessed.

Entity resolution and publication. Authors, treatments, and concepts get normalized so the same thing mentioned two ways becomes one thing. Published claims carry their provenance: source identifier, extraction timestamp, verification status.

What moved throughput

I expected model upgrades to matter most. They mattered least. The changes that made the difference:

Caching by content hash. Papers reappear across corpora and reruns. Hashing the content and skipping anything already processed turned reruns from days into hours.

Batching with bounded concurrency. A few parallel workers with backpressure beat both the single-threaded version and the unbounded version that fell over on rate limits.

Chunking tuned to the document, not the model. Respecting section boundaries in papers produced better extractions than any prompt improvement on badly chunked text. I've since relearned this on legal documents, where it matters even more.

Where clever lost to conservative

Entity resolution was the stage where I tried to be smart and paid for it. The clever version used a model to decide whether two author names referred to the same person. It merged people who shouldn't be merged, which is the worst failure mode in a research tool, because it silently corrupts everything built on top.

The version that works is deterministic normalization with conservative merge rules: two records that might be the same person stay separate until evidence says otherwise. That gives up some completeness, and I'll take that over being confidently wrong.

The numbers

As of this writing: 65,074 papers sourced and triaged in the consciousness-research corpus with 2,233 fully extracted, 25,037 in the eczema corpus with 9,257 extracted, and every published citation machine-verified against OpenAlex and Crossref before it ships, a gate I built after discovering my own pipeline fabricating citations. Total extraction spend is tens of dollars per corpus, not thousands, because triage is aggressive, the cheap model does the bulk work, and nothing expensive runs twice.

If you're building anything similar, here's the order I'd hand you: get provenance on every document first, build the quarantine paths second, tune triage third, and only then start arguing about which model to use. The model is the most replaceable part of the whole machine.