2026-06-02 · fundamentals · fundamentals week 4 · from NoeticMap
Chunking decides what your AI can know
Chunk boundaries dominate retrieval quality more than model choice. A live visualizer shows a height limit getting separated from the exception that overrides it, and what that failure looks like in legal text.
Chunk boundaries decide what a retrieval system can know about a document, and in my own pipelines they've mattered more than model choice. I keep watching people, including past me, tune prompts for weeks before checking their chunk boundaries even once, so I built a demo to show where the cut destroys a fact.
Why the chunk is the atom
A retrieval-augmented system doesn't retrieve documents, it retrieves chunks. The chunk is what gets embedded, what gets scored, what gets stuffed into the model's context. Whatever facts a chunk boundary separates, the system now knows separately or not at all. No downstream model, however good, can reason over a sentence that never arrived.
The retrieval literature settled early on smallish passages. DPR (Karpukhin et al., 2020), the retriever behind the original RAG paper (Lewis et al., 2020), split Wikipedia into 100-word blocks. Small chunks embed cleanly, one topic per vector, and they're cheap to stuff into context. The papers are quieter about the cost: 100 words is smaller than a lot of meanings.
Watch a fact get cut in half
Here's the build, running live. One sample document shaped like the legal text I work with, three chunking strategies, and a two-part fact planted in it: a height limit, and the exception that can override that limit. The check at the bottom tells you whether a retriever could ever see them together.
Section 4. Building Height 4.1 In a residential zone, subject to the exceptions and site-specific provisions as set out elsewhere in this section and in the applicable zoning by-law, the maximum permitted height of a de
tached house is 10.0 metres where the lot frontage is 12.0 metres or greater. 4.2 Despite 4.1, where a site-specific exception applies to the lot in question, the maximum permitted building height is the height set out
in the text of the exception, and the exception prevails over the general limit in this section. Section 5. Lot Coverage 5.1 The maximum lot coverage in a residential zone is 33 percent of the lot area. 5.2 A covered
platform attached to the main wall is excluded from lot coverage where its area does not exceed 4.0 square metres.
Walk through what each strategy does to it.
Fixed size is the default in most tutorials because it's one line of code. Slide the chunk size around and watch the limit sentence get bisected mid-number at some sizes. A chunk ending in "the maximum permitted height of a detached house is 10.0" is worse than useless: it embeds and retrieves like any healthy chunk, and delivers a truncated fact with total confidence.
Fixed size with overlap is the standard patch, and it fixes the bisection problem: with 60 characters repeated across each boundary, a sentence split by one chunk usually survives whole in its neighbor. Notice what it doesn't fix. The limit and the exception are different sentences in different paragraphs, and no plausible overlap makes them travel together. Overlap repairs cuts, it doesn't preserve structure.
Structure-aware splits on the document's own boundaries, whole paragraphs packed into chunks, sections kept together where they fit. It's the only strategy in the demo where "Section 4. Building Height" arrives as one unit, limit and exception side by side.
And that's the failure that matters. A chunk containing only clause 4.1 answers "what's the height limit" confidently and wrong for any lot carrying an exception, because clause 4.2, the one that says the exception prevails over the general limit, went to a different chunk with a different embedding. The retriever and the generator both did their jobs. The knowledge was destroyed upstream, at the cut.
Where I learned this twice
The first time was in my research extraction pipeline. Papers have a structure, sections and headings that the authors put there because the units mean something, and early on I chunked past them at fixed sizes. Respecting section boundaries produced better extractions than any prompt improvement I made on badly chunked text, which is a sentence I wrote in the pipeline essay and building this demo finally let me explain rather than just report. The section boundary is the author telling you where one claim's context ends. Cutting elsewhere means every chunk leaks context into its neighbors.
The second time was legal text, where the stakes are sharper. Ontario regulations are full of the demo's shape: a general rule, then "despite subsection (1)" doing the overriding two paragraphs later. I've written about what instance-level exceptions did to my permit tool's design in the average answer is a trap. Chunking is where that failure gets manufactured mechanically: cut a statute at a fixed size and you've built a machine for serving general rules divorced from their exceptions, which in a regulated domain is a machine for confident wrong answers.
How to pick
The real procedure is a measurement, and it's cheap. Collect a handful of real questions with known answers in your corpus. For each chunking strategy, check whether the chunk that should answer each question (a) exists intact and (b) gets retrieved for that question. That hit rate is the number that matters, and it's a golden set like the one that caught my model choice being wrong, just aimed one layer down the stack.
What the measuring has taught me:
- Chunk to the document's structure when the document has one. Papers and statutes arrive pre-chunked by their authors. Use it.
- Overlap repairs cuts you couldn't avoid. It doesn't preserve structure across paragraphs.
- When a fact's meaning depends on another passage, no chunking fully saves you, and the fix lives elsewhere: either attach the dependency at extraction time (my pipeline stores the exception with the limit as one structured record) or retrieve more than one chunk and let the model reconcile, and then test that it does.
- The chunk size your corpus needs is an empirical fact about your corpus. Mine differ between papers and regulations. Nobody else's default was measured on your documents.
If you're debugging a retrieval system that "has the document but keeps missing the answer," look at the boundaries before you look at the model. The answer may be in two pieces.