RAG Chunking Strategies: Compare Fixed-Size, Recursive, Semantic, and Document-Aware Splitting
Choosing how to split documents is one of the most consequential design decisions in a retrieval-augmented generation system. This guide compares practical RAG chunking strategies, shows how to test them, and provides a reusable checklist for improving retrieval without losing context, metadata, or citations.
Overview
Chunking divides source documents into smaller passages that can be embedded, stored, retrieved, and supplied to a language model. The goal is not to create the smallest possible pieces. The goal is to create passages that are coherent enough to answer questions while remaining focused enough for retrieval to identify them accurately.
There is no universally best chunk size for retrieval-augmented generation. A useful starting point depends on the document type, the questions users ask, the embedding model's input limits, the language of the content, and the capabilities of the vector database. Treat chunking as an experimental retrieval decision rather than a one-time preprocessing step.
Before changing a splitter, inspect your documents. Note whether they contain headings, tables, code blocks, lists, footnotes, page numbers, legal clauses, or repeated headers. Also identify the unit of meaning that users are likely to ask about. For a technical manual, that unit may be a procedure or subsection. For a policy document, it may be a complete rule and its exceptions. For a transcript, it may be a topic segment rather than a page.
Comparison of common RAG chunking strategies
| Strategy | Typical starting point | Overlap | Metadata to preserve | Retrieval profile | Complexity | Best-fit documents |
|---|---|---|---|---|---|---|
| Fixed-size | Use a token or character window sized for the embedding model and query type | Small, consistent overlap when context crosses boundaries | Document ID, section, page, start and end offsets | Predictable baseline; may split ideas awkwardly | Low | Uniform notes, plain text, initial prototypes |
| Recursive | Set a target window, then split using progressively smaller separators | Moderate overlap only where needed | Heading path, paragraph index, source offsets | Usually more coherent than blind fixed-size splitting | Low to medium | Mixed prose, help content, general documentation |
| Sentence-based | Group complete sentences until a target size is reached | Often limited or sentence-aware | Sentence range, paragraph, page, source offsets | Preserves grammatical units but can separate headings and supporting context | Medium | Articles, reports, FAQs, narrative prose |
| Semantic | Place boundaries where adjacent text changes in meaning | Usually low; add targeted context when testing shows gaps | Semantic segment ID, heading path, source offsets | Can preserve topics well, but results depend on thresholds and models | Medium to high | Long prose with clear topic shifts |
| Markdown- or document-aware | Split by headings, sections, lists, tables, and code blocks before applying a size limit | Use sparingly to avoid duplicating structured content | Heading hierarchy, block type, page, anchor, offsets | Strong when structure carries meaning | Medium | Technical docs, repositories, wikis, product manuals |
| Parent-child | Store smaller searchable child chunks linked to larger parent sections | Child overlap is optional; parent supplies broader context | Parent ID, child ID, section, page, offsets | Precise matching with a context expansion step | Medium to high | Books, policies, long manuals, structured knowledge bases |
These approaches can be combined. For example, a document-aware pipeline may first isolate a heading and its content, then use recursive or sentence-based splitting when that section exceeds the target size. A parent-child design may use semantic or sentence-based child chunks while retaining the complete subsection as the parent.
Checklist by scenario
For plain text or lightly structured notes
- Start with fixed-size or recursive splitting to establish a measurable baseline.
- Use the tokenizer associated with the embedding workflow when possible; character counts do not represent token counts consistently across languages and formats.
- Preserve paragraph boundaries where they exist.
- Test a small range of chunk sizes rather than assuming one setting will fit every query.
- Record source offsets so retrieved text can be traced back to the original document.
For technical documentation and Markdown
- Extract heading hierarchy and attach the full heading path to each chunk.
- Keep code blocks, command examples, and their explanations together when they answer the same task.
- Avoid splitting a numbered procedure between the instruction and its conditions or expected result.
- Handle tables deliberately. A row without its column headers may be difficult to interpret after retrieval.
- Consider parent-child retrieval when users need a precise paragraph but the answer depends on the surrounding section.
For policies, contracts, and compliance material
- Preserve clause numbers, definitions, exceptions, and cross-references as metadata or visible text.
- Prefer boundaries based on clauses or sections rather than arbitrary windows.
- Keep a rule and its qualifications together when the target size allows.
- Store page numbers and document versions for citation and review.
- Test questions that ask for exceptions, effective dates, and relationships between clauses; these often expose weak boundaries.
For transcripts and conversational records
- Retain speaker labels and timestamps.
- Group turns by topic or time window, but avoid mixing unrelated conversations.
- Use overlap carefully because repeated dialogue can increase irrelevant matches.
- Consider a summary or topic label as metadata, while keeping the original wording available for evidence.
For multilingual or mixed-format collections
- Measure chunks with the tokenizer used by the embedding model rather than relying only on characters.
- Check whether sentence detection handles the languages in your collection.
- Separate boilerplate such as navigation, repeated headers, and disclaimers when it dominates retrieval results.
- Evaluate each major language and document format independently before applying one global configuration.
A reusable evaluation workflow
- Inspect the corpus. Sample documents from every important source, format, and length category. Identify natural boundaries and recurring noise.
- Define representative questions. Include direct lookups, multi-step questions, questions about exceptions, and queries using terminology that does not exactly match the source.
- Create a baseline. Begin with a simple recursive or fixed-size splitter and keep the embedding model, retrieval count, and answer prompt constant.
- Label expected evidence. For each test question, record which document, section, or passage contains the answer.
- Compare retrieval results. Examine whether the expected evidence appears, whether it is complete, and how much irrelevant text is returned.
- Diagnose the failure. A missed answer may result from a bad boundary, an incomplete query, poor metadata, an unsuitable embedding, or an incorrect retrieval filter.
- Change one variable. Test chunk size, overlap, boundary rules, or parent expansion separately so the result remains interpretable.
- Preserve citations. Carry document IDs, page or section references, offsets, and version information through indexing and generation.
For a broader evaluation process, use a repeatable test set and scoring method as described in How to Evaluate AI Outputs: A Practical Test Set and Scoring Template.
What to double-check
- Embedding-model limits: Confirm the model's documented input behavior and tokenizer. Do not assume that a character limit maps cleanly to a token limit.
- Chunk and query scale: Very large chunks may contain several unrelated topics; very small chunks may omit the conditions needed to interpret an answer.
- Overlap cost: Overlap can protect against boundary loss, but excessive overlap increases index size and may return duplicate evidence.
- Metadata filtering: Verify that source, tenant, language, date, permissions, and document version fields survive ingestion and are usable by retrieval.
- Vector-database behavior: Check how the store handles metadata filters, duplicate vectors, updates, deletes, namespaces, and hybrid or keyword search if those features are part of the design.
- Reranking: If a reranker is used, evaluate the complete pipeline. A splitter that looks weak with vector similarity alone may behave differently after reranking, but it still must preserve the required evidence.
- Citations: Make sure every chunk can be mapped to a stable source location. Do not rely on generated text to reconstruct page or section references.
- Security: Treat retrieved documents as untrusted input. Chunking does not remove malicious instructions embedded in source material; test the complete RAG flow with adversarial content using a prompt injection testing checklist for RAG and AI agents.
- Prompt behavior: Keep the answer prompt stable during chunking experiments. Otherwise, prompt changes can obscure whether retrieval actually improved.
Common mistakes
Choosing a popular chunk size without testing
A setting that works for short FAQs may perform poorly on code, contracts, or research reports. Use a starting value only as a baseline, then test it against real questions.
Splitting before understanding structure
Blindly cutting every document into equal windows can separate headings from content, table headers from rows, or exceptions from rules. Parse structure first when structure carries meaning.
Using overlap as a substitute for good boundaries
Overlap can reduce missed context, but it cannot reliably repair a poorly designed split. It also creates repeated content and may increase irrelevant retrieval.
Ignoring non-text elements
Tables, diagrams, code, footnotes, and scanned pages often need specialized extraction or metadata. A clean-looking text stream may still be missing the information users need.
Optimizing only for retrieval scores
A retrieved chunk can be technically relevant but unusable if it lacks the definition, exception, or source reference needed by the answer. Inspect context completeness and citation accuracy, not only whether a result appears in the top results.
Changing multiple pipeline components at once
If chunking, embeddings, retrieval count, reranking, and prompting all change together, you cannot identify what helped. Version each configuration and record the test set results. A documented prompt versioning workflow can be extended to retrieval and indexing settings.
When to revisit
Revisit your RAG chunking strategy whenever the inputs or objectives change. That includes adding a new document type, importing a different language, changing the embedding model, switching vector databases, modifying metadata filters, introducing reranking, or changing the context window used by the generation model.
Also review chunking after users report missing answers, irrelevant citations, duplicated context, incomplete procedures, or confusion between document versions. These symptoms may indicate a boundary problem, but they can also reveal extraction, indexing, access-control, query-rewriting, or prompt issues. Diagnose the full retrieval path before rewriting the splitter.
Before a seasonal planning cycle or a major knowledge-base refresh, run the representative test set again. Compare missed evidence, irrelevant matches, answer completeness, citation accuracy, latency, and index growth. Keep the previous configuration available so you can roll back if a new strategy improves one document class while degrading another.
Action checklist: sample the corpus, map document structure, choose a baseline splitter, preserve metadata and offsets, build representative questions, test one variable at a time, inspect retrieved context manually, measure both misses and irrelevant matches, and record the configuration. Repeating this checklist makes chunking an maintainable part of AI application engineering rather than a hidden preprocessing assumption.