BLUF: Semantic chunking is the default upgrade path every RAG vendor recommends, and the published evidence does not support it as a default. Across four independent evaluations, embedding-similarity splitting either matched or lost to plain fixed-size and page-level baselines while costing between four minutes and fifteen hours of preprocessing per corpus instead of under one second. The one result that reproduces everywhere is different and less convenient: the optimal chunk size is a property of your documents and your embedding model, not of the algorithm. Measured on their own corpora, teams routinely find a mid-range sweet spot between 512 and 1,024 tokens — and teams with short factoid queries find 64 tokens. You cannot know which you are without running the measurement. Budget two engineering weeks for a recall@k evaluation on your own corpus before you accept any vendor default.
What the term "semantic chunking" actually means in an implementation
The vocabulary is loose enough to hide real architectural differences, so it is worth being precise about what is being compared.
Fixed-size chunking splits a document into uniform token windows, usually with 10 to 20 percent overlap to avoid cutting mid-sentence. NVIDIA tested 10, 15 and 20 percent overlap and found 15 percent performed best on FinanceBench at 1,024 tokens, which is consistent with the overlap range in common industry practice. Cost: effectively zero.
Recursive character splitting applies a separator hierarchy — paragraph breaks, then line breaks, then spaces — and recurses into any segment still above the size limit. It is a heuristic that respects document structure without understanding it. Cost: milliseconds.
Breakpoint-based semantic chunking (the widely deployed LangChain-style percentile-threshold splitter, originally from Greg Kamradt) embeds consecutive sentences, measures the similarity drop between neighbours, and inserts a boundary where the distance exceeds a threshold. It makes each decision from two sentences at a time, which is locally greedy. Cost: one embedding call per sentence, plus threshold tuning.
Clustering-based semantic chunking groups semantically similar units globally rather than sequentially. Chroma’s ClusterSemanticChunker composes chunks up to a size ceiling on the basis of embedding similarity; Vectara’s variant combines positional and cosine distance with a weighting term so that proximity is not discarded entirely.
Page-level chunking treats each source page as one chunk. It sounds unsophisticated. It is the strongest single performer in the largest enterprise-document evaluation published to date.
LLM-based chunking asks a language model to place the boundaries directly. Chroma’s LLMChunker prompts GPT-4o over indexed candidate breakpoints; LumberChunker aggregates paragraphs and asks an LLM where the semantic shift occurs. Cost: one or more LLM inferences per document, which is the variable that decides whether this is viable at enterprise corpus scale.
The evidence, dataset by dataset
Four evaluations are worth reading in full because they disagree in useful, specific ways.
NVIDIA (June 2025) ran the largest enterprise-document study: five corpora including 767 annotated public PDFs, 512 internal earnings and consulting decks, FinanceBench, the Docugami KG-RAG set and a collection of Deloitte tax reports. Pipeline held constant — llama-3.2-nv-embedqa-1b-v2 for embedding, llama-3.2-nv-rerankqa-1b-v2 for reranking, top-k of 10, llama-3.1-nemotron-70b-instruct for generation — with end-to-end answer accuracy scored by a two-model judge council. Page-level chunking took the highest average accuracy at 0.648 with the lowest standard deviation at 0.107. Every token-based size landed between 0.603 and 0.645. Page-level also beat section-level chunking on average, even when both used the same extraction model to eliminate extraction artefacts.
The per-dataset numbers are the interesting part, because they contradict the headline. FinanceBench peaked at 1,024 tokens (0.579) against page-level 0.566. The internal Earnings corpus peaked at 512 tokens (0.681), declining to 0.663 at 1,024 and 0.651 at 2,048. KG-RAG peaked with page-level (0.520) and bottomed out at 128 tokens (0.421). RAGBattlePacket peaked at 1,024 (0.804) with page-level close behind (0.790) and both 128 and 2,048 tokens at 0.749. Three financial-document corpora, three different optimal strategies.
Chroma (July 2024) introduced the metric contribution that matters most for procurement: token-level evaluation. Standard IR benchmarks such as MTEB and BEIR score whole-document relevance with nDCG@10, which structurally cannot see chunking at all. Chroma instead scores retrieved tokens against ground-truth excerpt tokens using precision, recall and Intersection over Union — the bounding-box analogy from computer vision applied to text spans. With text-embedding-3-large at five retrieved chunks: RecursiveCharacterTextSplitter at 200 tokens with no overlap returned 88.1 percent recall at 7.0 percent precision and 6.9 IoU. ClusterSemanticChunker at a 400-token ceiling reached 91.3 percent recall; at a 200-token ceiling it took the best precision and IoU in the table at 8.0 each. LLMChunker took the highest recall at 91.9 percent. The default KamradtSemanticChunker — the breakpoint method most teams actually deploy — scored 83.6 percent recall at 1.5 percent precision and 1.5 IoU, below average on every metric, because unconstrained similarity splitting produced a mean chunk length around 660 tokens. Applying a size floor lifted recall to 87.1 percent even though mean chunk length dropped.
One number from that table deserves separate attention: the documented OpenAI Assistants file-search default of 800 tokens with 400 overlap scored 87.9 percent recall but 1.4 percent precision and 4.7 percent Precision-Omega, the worst efficiency in the entire evaluation. A widely used platform default is a poor recall-efficiency trade, and nothing in the product surface tells you that.
Vectara (October 2024) asked the cost question directly across document retrieval, evidence retrieval and answer generation. Fixed-size chunking won 3 of 5 evidence-retrieval datasets, with the margins between fixed-size, breakpoint and clustering chunkers minimal throughout. On document retrieval the pattern was sharper and more instructive: breakpoint-based semantic chunking won decisively on artificially stitched corpora — Miracl at 81.89 F1@5 against fixed-size 69.45, Natural Questions at 63.93 against 43.79 — and lost on every real, unstitched corpus: HotpotQA 87.37 against 90.59, MS MARCO 92.23 against 93.58, ConditionalQA 64.44 against 68.11, Qasper 89.27 against 90.99. The authors are explicit about why: stitched documents were assembled by concatenating unrelated short documents, giving them artificially high topic diversity. Semantic splitting recovers the original document boundaries. Real enterprise documents are rarely that internally heterogeneous.
A 2026 arXiv evaluation added the operational dimension that the earlier work omitted. Measuring chunking wall-clock time per corpus: fixed-size averaged under one second, recursive semantic 4.9 minutes, GraphSeg 3.09 hours, LumberChunker 8.37 hours and DenseX 15.05 hours. On evidence retrieval at recall@10, recursive semantic averaged 89.36 and fixed-size 87.71 — a 1.65-point gap for a roughly 300-fold cost increase. More usefully for anyone sizing a production ingest: several expensive methods did not finish at all on the larger corpora, hitting execution-time limits or triggering out-of-memory failures in the underlying sentence-segmentation library. Implementation fragility at corpus scale is a real selection criterion, not a footnote.
Decision table: selecting a chunking strategy by corpus and constraint
Corpus is paginated PDFs with meaningful page-level information units — reports, filings, technical manuals. Query mix is broad. — Start with: page-level chunking. — Why: highest average accuracy (0.648) and lowest variance (0.107) in the NVIDIA evaluation, plus stable citation boundaries, because a page reference survives a re-chunk while a token index does not. — Fallback: 512 or 1,024 tokens if page-level extraction quality is poor.
Corpus is continuous prose without reliable page structure — wiki exports, transcripts, knowledge-base articles. Query mix is short factoid lookups. — Start with: fixed-size 256 to 512 tokens with 15 percent overlap, or recursive character split at roughly 200 tokens. — Why: NVIDIA’s factoid-heavy corpora peaked at 256 to 512; SQuAD in the Fraunhofer study peaked at 64 tokens with recall@1 of 64.1 percent and lost 10 to 15 points by 512. — Fallback: increase size if answers are dispersed rather than localised.
Corpus is long technical or narrative documents where answers span paragraphs — incident reports, contracts, research. — Start with: fixed-size 512 to 1,024 tokens. — Why: TechQA recall@1 rose from 4.8 percent at 64 tokens to 71.5 percent at 1,024; NarrativeQA from 4.2 to 10.7 percent across the same range. — Fallback: hierarchical retrieval with small child chunks and a parent-context expansion step.
Corpus is heterogeneous with genuinely abrupt topic shifts inside single documents — aggregated newsletters, concatenated meeting notes, merged ticket threads. — Consider: breakpoint-based semantic chunking with an enforced minimum chunk size. — Why: this is the one condition under which semantic splitting won decisively in the Vectara evaluation. — Caveat: verify the heterogeneity is real. If your documents are internally coherent, you are paying for a benchmark artefact.
Constraint is token efficiency — long context is expensive, or the generator degrades with distractors. — Start with: clustering-based semantic chunking at a 200-token ceiling, or recursive split at 200 tokens with zero overlap. — Why: Chroma’s Cluster-200 took the best precision and IoU (8.0 each); reducing overlap improves IoU directly because the metric penalises redundant tokens. — Trade-off: recall drops. Verify against your answer-accuracy target, not just retrieval scores.
Constraint is ingest latency or a corpus in the millions of documents. — Start with: fixed-size or page-level. — Why: sub-second versus hours per corpus, and LLM-based and DenseX-style methods failed to complete on the largest corpora in the 2026 evaluation. — Trade-off: you give up 1 to 2 points of retrieval recall for an ingest pipeline that finishes and can be re-run.
The embedding model is a variable, not a constant
This is the finding most likely to invalidate a chunking decision that was made carefully but once.
The Fraunhofer IAIS study compared a decoder-based embedding model (Stella, with a context window above 130,000 tokens) against an encoder-based one (Snowflake arctic-embed-l-v2.0, 8,194 tokens) across the same chunk sizes. On COVID-QA, Stella peaked at 64 tokens with recall@1 of 52.1 percent and declined monotonically with larger chunks; Snowflake improved monotonically and peaked at 1,024 tokens with 54.2 percent. Same corpus, same chunk sizes, opposite optimal configuration.
On long-document corpora Stella gained 5 to 8 points of recall@1 over Snowflake at 512 to 1,024 tokens, consistent with a model pretrained on long inputs exploiting global chunk context. Snowflake held within 1 to 2 points on the short-answer corpora, where fine-grained entity matching dominates.
Chroma’s results show the same dependency from the other direction. Repeating the evaluation with all-MiniLM-L6-v2, TokenTextSplitter at 250 tokens with 125 overlap took the highest recall at 82.4 percent — and the identical splitter with zero overlap dropped to 77.1 percent. With text-embedding-3-large, overlap mattered far less. Smaller-context embedding models need overlap to preserve boundary context; larger ones do not, and the redundant tokens simply cost you precision.
The operational consequence is a governance one. If your chunk size was tuned for the embedding model you deployed eighteen months ago, and someone has since upgraded the embedder for a documented quality gain, your chunking configuration is now unvalidated. Chunk size and embedding model must be versioned and re-evaluated together, in the same way I argued retrieval decisions themselves need to be reconstructible in the RAG security operating model.
Failure modes you should expect
1. Fragment collapse from unconstrained similarity splitting. A breakpoint chunker with no size floor produces chunks whose length is set entirely by the similarity signal. Chroma measured a mean of roughly 660 tokens for the default configuration, which sounds fine as an average and hides the tail: very short fragments embed cleanly and retrieve well in isolation, then starve the generator of the surrounding context it needs to answer. Detection: plot the chunk-length distribution after ingest and look at the fifth percentile, not the mean. Mitigation: enforce a minimum chunk size. Adding that floor raised Kamradt recall from 83.6 to 87.1 percent even though mean length fell.
2. Vendor defaults adopted as engineering decisions. The OpenAI Assistants file-search default of 800 tokens with 400 overlap produced the worst token efficiency in Chroma’s evaluation — 1.4 percent precision, 4.7 percent Precision-Omega. Defaults are chosen to be safe across all customers, which means optimal for none. Mitigation: treat every default as a hypothesis with a documented alternative.
3. Benchmark-shape mismatch. Semantic chunking’s clearest wins in the literature come from corpora built by stitching unrelated short documents together. If you evaluate on synthetic or aggregated documents and deploy against coherent single-topic documents, you will have measured the wrong thing and paid for the privilege. Mitigation: evaluate on unmodified samples of your production corpus, and if you must synthesise long documents, report both stitched and unstitched results separately.
4. Chunk boundaries as unstable citation anchors. Token-index chunk references change whenever chunk size changes, so every stored citation, every evaluation ground-truth mapping and every user-visible source link breaks on re-chunk. NVIDIA calls this out explicitly as an advantage of page-level chunking: page boundaries are static. Mitigation: anchor citations to a stable document coordinate — page, section identifier, character offset in the source — never to a chunk index.
5. Ingest pipelines that cannot be re-run. If a full re-chunk takes fifteen hours of LLM inference, you will not re-run it when the embedding model changes, when a bug is found in the extractor, or when a document class turns out to be mis-parsed. The pipeline becomes a one-way door. Mitigation: measure full-corpus re-ingest cost as an explicit non-functional requirement before selecting the method, and require that the number is small enough to run quarterly.
6. Chunking optimised for retrieval scores rather than answers. Chroma’s Cluster-200 configuration takes the best precision and IoU while landing at average recall. Whether that is a win depends entirely on whether your generator is more damaged by missing context or by distractors. Mitigation: score end-to-end answer accuracy as the decision metric, with retrieval metrics as diagnostics. NVIDIA’s conclusion runs the same way — chunking is one hyperparameter among embedding choice, reranking and generation parameters, and it should not be tuned in isolation.
7. Chunking that silently crosses an access boundary. Merging or expanding chunks across document boundaries can place content from two differently classified sources into a single retrievable unit. That is a permission problem, not a relevance problem, and it is why chunk-level metadata has to carry classification and access labels — the design point in RAG access control: enforce permissions before retrieval.
A two-week evaluation you can run on your own corpus
This is the only reliable way to select a chunking strategy, and it is cheap relative to the cost of being wrong at scale.
Step 1 — Sample by document class, not at random. Draw 30 to 50 documents per distinct class in your corpus (paginated reports, wiki prose, transcripts, tickets). The NVIDIA result that three financial-document corpora had three different optima is the reason class-level sampling matters more than sample size.
Step 2 — Build ground truth at the span level. For each of 50 to 100 real user questions, record the exact text span in the source that answers it. Span-level ground truth is what allows token-level IoU and precision; document-level labels cannot distinguish chunking strategies at all. Chroma’s generation pipeline — prompt an LLM for a query plus verbatim excerpts, accept only excerpts with exact full-text matches, then filter duplicates and low-similarity pairs by cosine threshold — is a reproducible way to bootstrap this, and their code is public.
Step 3 — Fix everything except chunking. One embedding model, one reranker, one top-k, one generator, one prompt. NVIDIA held all four constant precisely so that accuracy differences were attributable to chunking rather than to extraction or retrieval artefacts. Any variation you leave in place will absorb the signal you are looking for.
Step 4 — Sweep the size axis first. Fixed-size at 128, 256, 512, 1,024 and 2,048 tokens with 15 percent overlap, plus page-level if your documents are paginated. This sweep alone will locate your sweet spot, and in every published evaluation the size axis moves the metric more than the algorithm choice does.
Step 5 — Add exactly two algorithmic candidates. Recursive character split at your best size, and one semantic method with an enforced minimum chunk size. Two candidates keep the matrix tractable and cover the plausible upside.
Step 6 — Record four numbers per configuration. Token-level recall and IoU on the span ground truth, end-to-end answer accuracy, and chunking wall-clock time for the full corpus. The fourth number is the one procurement discussions omit and operations later regret.
Step 7 — Apply an explicit decision rule before you see the results. For example: adopt the more expensive method only if it improves end-to-end answer accuracy by more than 3 points, since the published gaps between semantic and fixed-size baselines sit between roughly 1 and 2 points. Setting the threshold in advance is what prevents a 1.5-point gain from justifying a 300-fold cost increase.
Step 8 — Version the result. Record chunk size, overlap, algorithm, embedding model version and the measured numbers as a configuration artefact tied to the index build. Without that record you cannot tell later whether a quality regression came from a model change, a chunking change or a corpus change — which is the same lineage argument that applies to RAG data governance for enterprise knowledge bases.
Verification checklist before an index goes to production
— Chunk-length distribution recorded, with fifth-percentile and maximum length, not only the mean.
— Minimum chunk size enforced, with the floor value documented and justified.
— Chunk size swept and the selected value backed by measured recall@k or IoU on your own corpus.
— End-to-end answer accuracy measured, not just retrieval metrics.
— Full-corpus re-ingest wall-clock time measured and confirmed re-runnable on a defined cadence.
— Chunk size, overlap, algorithm and embedding model version recorded together as one versioned configuration.
— Citations anchored to stable document coordinates rather than chunk indices.
— Chunk metadata carries source classification and access labels, verified by a negative test.
— A re-evaluation trigger defined for embedding-model upgrades and for the arrival of a new document class.
Trade-offs, stated plainly
Page-level chunking wins on average and loses specifically. It took the best mean accuracy and the lowest variance in the largest enterprise evaluation, and it was still beaten on two of five corpora by 512 and 1,024-token fixed sizing. It also requires reliable pagination, which rules out a large share of real enterprise content: wiki pages, chat exports, database records and CMS entries have no pages to speak of.
Semantic chunking is not useless — it is unjustified as a default. Clustering-based methods take the best token efficiency in Chroma’s evaluation, and LLM-based chunking takes the best recall. If your generator is expensive per token or your corpus genuinely mixes topics inside single documents, those wins are real. What the evidence does not support is adopting them before measuring, or expecting them to compensate for an unmeasured chunk size.
Small chunks trade recall for precision, and the exchange rate depends on your generator. Chroma’s Cluster-200 configuration takes the best IoU at average recall. Whether losing recall to gain token efficiency is correct depends on whether your model is more damaged by missing evidence or by distractors — which is an empirical question about your model, not a general principle.
Every one of these evaluations has limits you should carry into your own reading. Chroma’s dataset is LLM-generated and stylistically narrow, and their queries were filtered by cosine threshold, which introduces sampling bias they acknowledge. The Fraunhofer study evaluated retrieval by string matching rather than semantic relevance and stitched short corpora to reach realistic document lengths. Vectara’s work is sentence-level throughout, so its embeddings lack contextual information by construction. NVIDIA’s study is authored by a vendor whose extraction microservices are in the measured pipeline. None of that invalidates the results. It does mean the only evaluation whose limitations you fully control is the one you run yourself.
And chunking is a smaller lever than the discourse suggests. NVIDIA’s own conclusion places it alongside embedding-model choice, reranking strategy and generation parameters as one hyperparameter among several. Vectara found chunking effects frequently overshadowed by embedding quality. If your retrieval quality is poor, chunking is unlikely to be the largest available fix.
What to do next
If you are standing up or reworking a RAG pipeline this quarter, run the size sweep before you evaluate a single vendor chunker. Five sizes plus page-level, on 50 real questions with span-level ground truth, will tell you more about your corpus than any published benchmark — and it converts an architecture debate into a measurement, which is the only form that survives a quality regression six months later.
If you want a second pair of eyes on the evaluation design, the span-level ground-truth construction, or a chunking configuration that was tuned before your embedding model changed, that scoping work is exactly what I do with engineering teams. Reach out through the contact page and bring a sample of your corpus and twenty real user questions — those two artefacts determine the entire answer.
Sources
NVIDIA Technical Blog, "Finding the Best Chunking Strategy for Accurate AI Responses" (18 June 2025) — five-corpus evaluation, page-level average accuracy 0.648 with standard deviation 0.107, per-dataset optima, 15 percent overlap finding, fixed pipeline configuration (developer.nvidia.com/blog/finding-the-best-chunking-strategy-for-accurate-ai-responses).
Chroma Technical Report, Smith and Troynikov, "Evaluating Chunking Strategies for Retrieval" (3 July 2024) — token-level IoU, precision and recall metrics, up to 9 percent recall spread between strategies, ClusterSemanticChunker and LLMChunker results, KamradtSemanticChunker default versus size-floored variant, OpenAI Assistants 800/400 default efficiency (research.trychroma.com/evaluating-chunking).
Qu, Tu and Bao (Vectara), "Is Semantic Chunking Worth the Computational Cost?", arXiv:2410.13070 (October 2024) — fixed-size versus breakpoint versus clustering across document retrieval, evidence retrieval and answer generation; stitched-corpus versus real-corpus divergence (arxiv.org/abs/2410.13070).
Ramakanth Bhat, Rudat, Spiekermann and Flores-Herr (Fraunhofer IAIS), "Rethinking Chunk Size For Long-Document Retrieval: A Multi-Dataset Analysis", arXiv:2505.21700 (June 2025) — recall@k by chunk size across six corpora, embedding-model-dependent optima for Stella versus Snowflake (arxiv.org/abs/2505.21700).
"Chunking Methods on Retrieval-Augmented Generation — Effectiveness Evaluation Against Computational Cost and Limitations", arXiv:2606.00881 (2026) — chunking wall-clock time per corpus, recall@10 and LLM-as-judge answer scores across eight methods, documented execution-time and memory failures at corpus scale (arxiv.org/abs/2606.00881).
