RAG Chunking Strategies: Compare Fixed-Size, Recursive, Semantic, and Document-Aware Splitting
RAGretrievalembeddingsvector databasesAI development

RAG Chunking Strategies: Compare Fixed-Size, Recursive, Semantic, and Document-Aware Splitting

SSmart QBits Editorial Team
2026-11-21
8 min read

Compare RAG chunking strategies and use a practical checklist to improve retrieval quality, context, metadata, and citations.

RAG Chunking Strategies: Compare Fixed-Size, Recursive, Semantic, and Document-Aware Splitting

Choosing how to split documents is one of the most consequential design decisions in a retrieval-augmented generation system. This guide compares practical RAG chunking strategies, shows how to test them, and provides a reusable checklist for improving retrieval without losing context, metadata, or citations.

Overview

Chunking divides source documents into smaller passages that can be embedded, stored, retrieved, and supplied to a language model. The goal is not to create the smallest possible pieces. The goal is to create passages that are coherent enough to answer questions while remaining focused enough for retrieval to identify them accurately.

There is no universally best chunk size for retrieval-augmented generation. A useful starting point depends on the document type, the questions users ask, the embedding model's input limits, the language of the content, and the capabilities of the vector database. Treat chunking as an experimental retrieval decision rather than a one-time preprocessing step.

Before changing a splitter, inspect your documents. Note whether they contain headings, tables, code blocks, lists, footnotes, page numbers, legal clauses, or repeated headers. Also identify the unit of meaning that users are likely to ask about. For a technical manual, that unit may be a procedure or subsection. For a policy document, it may be a complete rule and its exceptions. For a transcript, it may be a topic segment rather than a page.

Comparison of common RAG chunking strategies

StrategyTypical starting pointOverlapMetadata to preserveRetrieval profileComplexityBest-fit documents
Fixed-sizeUse a token or character window sized for the embedding model and query typeSmall, consistent overlap when context crosses boundariesDocument ID, section, page, start and end offsetsPredictable baseline; may split ideas awkwardlyLowUniform notes, plain text, initial prototypes
RecursiveSet a target window, then split using progressively smaller separatorsModerate overlap only where neededHeading path, paragraph index, source offsetsUsually more coherent than blind fixed-size splittingLow to mediumMixed prose, help content, general documentation
Sentence-basedGroup complete sentences until a target size is reachedOften limited or sentence-awareSentence range, paragraph, page, source offsetsPreserves grammatical units but can separate headings and supporting contextMediumArticles, reports, FAQs, narrative prose
SemanticPlace boundaries where adjacent text changes in meaningUsually low; add targeted context when testing shows gapsSemantic segment ID, heading path, source offsetsCan preserve topics well, but results depend on thresholds and modelsMedium to highLong prose with clear topic shifts
Markdown- or document-awareSplit by headings, sections, lists, tables, and code blocks before applying a size limitUse sparingly to avoid duplicating structured contentHeading hierarchy, block type, page, anchor, offsetsStrong when structure carries meaningMediumTechnical docs, repositories, wikis, product manuals
Parent-childStore smaller searchable child chunks linked to larger parent sectionsChild overlap is optional; parent supplies broader contextParent ID, child ID, section, page, offsetsPrecise matching with a context expansion stepMedium to highBooks, policies, long manuals, structured knowledge bases

These approaches can be combined. For example, a document-aware pipeline may first isolate a heading and its content, then use recursive or sentence-based splitting when that section exceeds the target size. A parent-child design may use semantic or sentence-based child chunks while retaining the complete subsection as the parent.

Checklist by scenario

For plain text or lightly structured notes

  • Start with fixed-size or recursive splitting to establish a measurable baseline.
  • Use the tokenizer associated with the embedding workflow when possible; character counts do not represent token counts consistently across languages and formats.
  • Preserve paragraph boundaries where they exist.
  • Test a small range of chunk sizes rather than assuming one setting will fit every query.
  • Record source offsets so retrieved text can be traced back to the original document.

For technical documentation and Markdown

  • Extract heading hierarchy and attach the full heading path to each chunk.
  • Keep code blocks, command examples, and their explanations together when they answer the same task.
  • Avoid splitting a numbered procedure between the instruction and its conditions or expected result.
  • Handle tables deliberately. A row without its column headers may be difficult to interpret after retrieval.
  • Consider parent-child retrieval when users need a precise paragraph but the answer depends on the surrounding section.

For policies, contracts, and compliance material

  • Preserve clause numbers, definitions, exceptions, and cross-references as metadata or visible text.
  • Prefer boundaries based on clauses or sections rather than arbitrary windows.
  • Keep a rule and its qualifications together when the target size allows.
  • Store page numbers and document versions for citation and review.
  • Test questions that ask for exceptions, effective dates, and relationships between clauses; these often expose weak boundaries.

For transcripts and conversational records

  • Retain speaker labels and timestamps.
  • Group turns by topic or time window, but avoid mixing unrelated conversations.
  • Use overlap carefully because repeated dialogue can increase irrelevant matches.
  • Consider a summary or topic label as metadata, while keeping the original wording available for evidence.

For multilingual or mixed-format collections

  • Measure chunks with the tokenizer used by the embedding model rather than relying only on characters.
  • Check whether sentence detection handles the languages in your collection.
  • Separate boilerplate such as navigation, repeated headers, and disclaimers when it dominates retrieval results.
  • Evaluate each major language and document format independently before applying one global configuration.

A reusable evaluation workflow

  1. Inspect the corpus. Sample documents from every important source, format, and length category. Identify natural boundaries and recurring noise.
  2. Define representative questions. Include direct lookups, multi-step questions, questions about exceptions, and queries using terminology that does not exactly match the source.
  3. Create a baseline. Begin with a simple recursive or fixed-size splitter and keep the embedding model, retrieval count, and answer prompt constant.
  4. Label expected evidence. For each test question, record which document, section, or passage contains the answer.
  5. Compare retrieval results. Examine whether the expected evidence appears, whether it is complete, and how much irrelevant text is returned.
  6. Diagnose the failure. A missed answer may result from a bad boundary, an incomplete query, poor metadata, an unsuitable embedding, or an incorrect retrieval filter.
  7. Change one variable. Test chunk size, overlap, boundary rules, or parent expansion separately so the result remains interpretable.
  8. Preserve citations. Carry document IDs, page or section references, offsets, and version information through indexing and generation.

For a broader evaluation process, use a repeatable test set and scoring method as described in How to Evaluate AI Outputs: A Practical Test Set and Scoring Template.

What to double-check

  • Embedding-model limits: Confirm the model's documented input behavior and tokenizer. Do not assume that a character limit maps cleanly to a token limit.
  • Chunk and query scale: Very large chunks may contain several unrelated topics; very small chunks may omit the conditions needed to interpret an answer.
  • Overlap cost: Overlap can protect against boundary loss, but excessive overlap increases index size and may return duplicate evidence.
  • Metadata filtering: Verify that source, tenant, language, date, permissions, and document version fields survive ingestion and are usable by retrieval.
  • Vector-database behavior: Check how the store handles metadata filters, duplicate vectors, updates, deletes, namespaces, and hybrid or keyword search if those features are part of the design.
  • Reranking: If a reranker is used, evaluate the complete pipeline. A splitter that looks weak with vector similarity alone may behave differently after reranking, but it still must preserve the required evidence.
  • Citations: Make sure every chunk can be mapped to a stable source location. Do not rely on generated text to reconstruct page or section references.
  • Security: Treat retrieved documents as untrusted input. Chunking does not remove malicious instructions embedded in source material; test the complete RAG flow with adversarial content using a prompt injection testing checklist for RAG and AI agents.
  • Prompt behavior: Keep the answer prompt stable during chunking experiments. Otherwise, prompt changes can obscure whether retrieval actually improved.

Common mistakes

A setting that works for short FAQs may perform poorly on code, contracts, or research reports. Use a starting value only as a baseline, then test it against real questions.

Splitting before understanding structure

Blindly cutting every document into equal windows can separate headings from content, table headers from rows, or exceptions from rules. Parse structure first when structure carries meaning.

Using overlap as a substitute for good boundaries

Overlap can reduce missed context, but it cannot reliably repair a poorly designed split. It also creates repeated content and may increase irrelevant retrieval.

Ignoring non-text elements

Tables, diagrams, code, footnotes, and scanned pages often need specialized extraction or metadata. A clean-looking text stream may still be missing the information users need.

Optimizing only for retrieval scores

A retrieved chunk can be technically relevant but unusable if it lacks the definition, exception, or source reference needed by the answer. Inspect context completeness and citation accuracy, not only whether a result appears in the top results.

Changing multiple pipeline components at once

If chunking, embeddings, retrieval count, reranking, and prompting all change together, you cannot identify what helped. Version each configuration and record the test set results. A documented prompt versioning workflow can be extended to retrieval and indexing settings.

When to revisit

Revisit your RAG chunking strategy whenever the inputs or objectives change. That includes adding a new document type, importing a different language, changing the embedding model, switching vector databases, modifying metadata filters, introducing reranking, or changing the context window used by the generation model.

Also review chunking after users report missing answers, irrelevant citations, duplicated context, incomplete procedures, or confusion between document versions. These symptoms may indicate a boundary problem, but they can also reveal extraction, indexing, access-control, query-rewriting, or prompt issues. Diagnose the full retrieval path before rewriting the splitter.

Before a seasonal planning cycle or a major knowledge-base refresh, run the representative test set again. Compare missed evidence, irrelevant matches, answer completeness, citation accuracy, latency, and index growth. Keep the previous configuration available so you can roll back if a new strategy improves one document class while degrading another.

Action checklist: sample the corpus, map document structure, choose a baseline splitter, preserve metadata and offsets, build representative questions, test one variable at a time, inspect retrieved context manually, measure both misses and irrelevant matches, and record the configuration. Repeating this checklist makes chunking an maintainable part of AI application engineering rather than a hidden preprocessing assumption.

Related Topics

#RAG#retrieval#embeddings#vector databases#AI development
S

Smart QBits Editorial Team

AI and Developer Education Editors

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.