What is Chunking?
RAG & RetrievalSplitting source documents into indexed pieces before embedding. Chunk boundaries decide what retrieval can return: a clause cut off from its qualifying condition is a failure mode baked in at ingestion.
Why It Matters
Chunking is the least glamorous and most consequential decision in a RAG pipeline. The model only ever sees chunks, never the original document, so every boundary drawn at ingestion time is a boundary in what the system can know. A clause separated from the condition that qualifies it, a table split from its header, a section cut mid-thought: each becomes a retrieval result that reads plausibly and answers wrong.
Because it happens once, silently, at ingestion, bad chunking is invisible until retrieval starts failing. And by then the debugging instinct points at the retriever or the prompt, never upstream at the split.
How It Works
Fixed-size splitting. Chunks at a character or token count, usually with overlap. Simple, predictable, and blind to structure; it cuts wherever the count runs out.
Structure-aware splitting. Follows the document’s own shape: headings, sections, paragraphs, table boundaries. A table stays a table; a clause keeps its conditions.
Semantic splitting. Splits where the topic shifts rather than where the layout does. More expensive at ingestion, better boundaries downstream.
Most production pipelines combine them: structure first, size limits second, overlap to soften the seams.
Where It Breaks
TrustNLP’s failure taxonomy catalogs ingestion-stage failures separately from retrieval ones for exactly this reason: the damage is done before the query arrives. Fixed-size chunking on a structured corpus produces chunks that match queries and answer them incompletely. And chunking decisions do not age well: a split tuned for last year’s document format quietly degrades as the corpus changes shape.
The compounding break is inconsistency: different splitters across document types means retrieval quality varies by document format, and the failure looks random.
How Flytebit Handles It
Chunking in our RAG builds follows the document’s structure, not a token count, and it is tested, not assumed: eval questions hit the corpus and surface which chunks fail to carry their own context. When documents change shape, the split gets re-evaluated like any other pipeline change. The measurement method is in Evaluating Agentic AI.