StackLift
All terms

Glossary

Chunking

Splitting documents into passages that are indexed and retrieved one at a time.

Chunking is the step in a RAG pipeline that splits documents into smaller passages, called chunks, before they are embedded and indexed. Retrieval then returns chunks, not whole files, so how you split decides whether the right answer can be found and whether it arrives with enough context to be useful.

Common approaches are fixed-size splits, recursive splits that follow headings, paragraphs and sentences, and semantic splits that cut where the topic changes. Newer methods add context to each chunk, such as a short generated summary of where it sits in the document, or embed the whole document first and split afterward.

There is no single best setting. Chunk size, overlap and structure depend on your documents, so teams compare options on a set of real questions. Metadata such as title, section and permissions should travel with every chunk.

Start in 60 seconds

Put this to work on your project.

Get an AI-assisted, architect-reviewed scope, cost range, and timeline. Free, no login.

The complete guide

Chunking: the complete guide

Updated November 18, 2026 ยท Published by , Founder, StackLift AI. Written with AI assistance; how we make content.

Short answer

Chunking is splitting documents into smaller passages, called chunks, before they are embedded and indexed for retrieval. A RAG system returns chunks, not whole files, so how you split decides whether the right answer can be found and whether it arrives with enough context. Chunk size, boundaries, overlap and metadata all matter.

What is chunking?

Chunking is the step in a retrieval pipeline where long documents are cut into smaller pieces. Each piece, or chunk, is turned into an embedding and stored in a vector database or search index. When a user asks a question, the system finds the most relevant chunks and gives them to the model.

It is a basic step of RAG, and often the one that decides quality. A model can only answer from what it is shown. If the chunk that contains the answer is cut in the wrong place, or buried inside a larger one that is mostly about something else, the answer is missed or diluted.

Do not confuse it with tokenization. Tokens are the small pieces a model reads internally. Chunks are passages that you choose, typically a paragraph or several, and they are far larger than a token.

Why do we need to chunk documents at all?

  • Models have a limited context window, so you cannot paste a whole library into every request.
  • Retrieval works best on focused passages. An embedding of a whole book blurs many topics into one point, which matches nothing well.
  • Smaller passages cost less to send, so each question uses fewer tokens. See AI cost optimization.
  • Focused context reduces distraction, which can reduce wrong answers. A model shown only relevant text is less likely to drift.
  • Chunks can carry their own source and permissions, so answers can cite where they came from and respect who may see what.

Larger context windows have not removed the need. Sending everything is slower and more expensive, and long inputs can still bury relevant detail.

What chunking strategies are there?

Common approaches
ApproachHow it splitsStrengthsWeaknesses
Fixed sizeEvery chunk has about the same length, with some overlapSimple, predictable, fastCuts through sentences and ideas
RecursiveTries to split at headings, then paragraphs, then sentences, until chunks fitRespects document structure, a strong defaultNeeds tuning of sizes and separators
Structure-awareSplits by the document's own units: sections, slides, table rows, code functionsKeeps meaning together, preserves headingsNeeds a parser per format
SemanticCuts where the topic changes, using embeddings to detect shiftsChunks follow ideasSlower, results vary, needs checking
Sentence or windowEmbeds small units and returns the surrounding text at query timePrecise matching with context on returnMore storage and logic
ContextualAdds a short generated description of where the chunk sits in the documentFixes chunks that make no sense aloneExtra model calls at indexing time
Parent and childSearch small chunks, return the larger section they belong toPrecision plus contextMore complex retrieval
Agentic or LLM-guidedA model decides the boundariesHandles messy documentsCostly, needs oversight

A common starting point is recursive splitting with modest overlap, followed by testing. Move to something more elaborate only when your test questions show a gap that the simpler method cannot close.

How big should a chunk be, and what is overlap?

There is no universal answer. Small chunks match questions precisely but may lack the context to answer them. Large chunks hold more context but blur topics and use more of the model's window. The right size depends on your content and the questions people ask.

Overlap repeats a little text from the end of one chunk at the start of the next, so an idea that straddles a boundary is not lost. Too little overlap risks broken ideas; too much wastes storage and returns near-duplicate results.

Trade-offs in chunk size
ChoiceHelpsHurts
Smaller chunksPrecise matches, lower cost per retrievalMissing context, more chunks to store and rank
Larger chunksMore context in each resultBlurred meaning, more irrelevant text sent to the model
More overlapFewer broken ideas at boundariesDuplicates and more storage
Less overlapCompact indexAnswers may split across two chunks

Test several settings on your own questions rather than copying a number from an article. The best setting for contracts is rarely the best for chat transcripts.

How do you chunk documents for a RAG system?

  1. Collect representative documents and a list of real questions that people will ask.
  2. Extract clean text, keeping structure such as headings, lists and tables. Poor extraction ruins everything after it.
  3. Choose a starting strategy, usually recursive or structure-aware splitting.
  4. Attach metadata to every chunk: source, title, section, date, language and who may view it.
  5. Embed and index the chunks.
  6. Run the test questions and check whether the right chunk is retrieved and whether it contains enough to answer.
  7. Adjust size, overlap and boundaries, and rerun the tests.
  8. Re-chunk when documents change, and track which version of the chunking produced which index.

The RAG implementation playbook walks through the surrounding steps, from sources to evaluation.

How should chunking differ by document type?

Matching the method to the content
ContentGood approachWhy
Policies, manuals, contractsSplit by section and clause, keep headings with the textMeaning depends on the section it sits in
FAQs and help articlesOne question and answer per chunkEach unit stands alone
Tables and spreadsheetsKeep rows with their column headers; consider storing as structured dataA number without its header is meaningless
Source codeSplit by function or classCode has natural units
Chat and email threadsGroup by conversation turns with the subjectReplies need what they reply to
SlidesOne chunk per slide, plus the title and speaker notesSlides are short and fragmentary
Scanned documentsRun text recognition first and check quality before chunkingErrors in text carry into every chunk
Long reportsParent and child chunks, with summariesNeeds both detail and the big picture

What does chunking look like in real projects?

  • A support assistant indexes help articles, one question and answer per chunk, with the product and date as metadata, so answers cite the right article and skip retired ones.
  • A legal team searches contracts. Splitting by clause, with the contract title and clause heading stored on each chunk, lets the assistant answer "what does the termination clause say?" with a source.
  • An internal knowledge assistant indexes wiki pages with the team and access group on every chunk, and filters on the signed-in user's permissions before retrieval.
  • A research tool stores small sentence chunks for precise matching and returns the surrounding section, so answers have the context to be understood.
  • A product catalog assistant keeps each product's attributes together, so a question about size and color retrieves one complete record.

The pattern repeats: keep together what belongs together, label it well and test with real questions.

Why does metadata matter as much as the text?

A chunk that says "the fee is waived" is useless without knowing which product, which year and which document. Metadata restores that. It also powers filtering, so a search can be limited to one product, region or date range.

  • Source and location, so answers can cite and link back.
  • Title, heading path and document type, so context travels with the chunk.
  • Dates and version, so outdated content can be filtered out.
  • Permissions, so a user is never shown content they may not see. Check permissions at retrieval time, before the model sees the text.
  • Language and other tags used for filtering.

Combining filters with hybrid search and reranking often does more than adjusting chunk size.

What mistakes should you avoid?

  • Using the default settings of a framework and never testing them on your documents.
  • Splitting tables and lists in the middle, so rows lose their headers.
  • Dropping headings and titles, which strips chunks of their context.
  • Chunks so large that every result is mostly irrelevant, or so small that none can answer anything.
  • Ignoring document updates, so the index returns old text.
  • Blaming the model for wrong answers that come from poor retrieval. Always read the retrieved chunks first.
  • Skipping permissions metadata, which risks exposing private content.

How do you measure chunking quality?

Measure retrieval separately from the final answer. If retrieval is poor, no prompt will fix it. Build a test set of real questions with the passages that should answer them.

  • Retrieval hit rate: is a chunk containing the answer among the results?
  • Rank: how high does the useful chunk appear?
  • Context sufficiency: does the chunk alone support a correct answer?
  • Answer groundedness: does the final answer match the retrieved text?
  • Noise: how much irrelevant text is sent to the model per question?
  • Cost and latency per question under each setting.

Use agent evaluation practices to run these on every change, and AI observability to see what retrieval returns in production.

What tools are used for chunking?

Most RAG frameworks include text splitters, including LangChain and LlamaIndex, which offer fixed, recursive, markdown-aware, code-aware and semantic splitters. Document parsing tools such as Unstructured and cloud document-analysis services extract text and structure from PDFs, slides and scans before splitting.

Vector stores and search engines, such as Qdrant, Weaviate, Elasticsearch and PostgreSQL with pgvector, store the resulting chunks and metadata. See vector database vs. pgvector. Some managed knowledge services also chunk for you, with settings you can adjust.

What does chunking mean for cost and timeline?

Chunking is a small part of the code and a large part of the quality. Effort depends on the variety of document formats, the quality of extraction, how much structure must be preserved, the metadata and permission model, and how much testing you do.

Re-indexing costs embedding calls each time you change the strategy, so test on a sample first. For model costs, see LLM pricing. To scope a retrieval project, use the free estimate and read RAG vs. fine-tuning.

How StackLift approaches chunking

StackLift treats retrieval as something to be tested, not assumed. Architect-reviewed scope comes first, then a governed delivery pipeline. Chunking choices, metadata, permissions and test questions are scoped as deliverables, and observability is in the plan from sprint one.

Agents work on feature branches and never touch production secrets, and engineers own the decisions. See LLM integration and AI chatbot development.

Questions people ask

What is the best chunk size?

There is no single best size. Start with a moderate size and modest overlap, then test on your own questions and adjust. Content type and question style matter more than any rule of thumb.

Is chunking the same as tokenization?

No. Tokenization splits text into the small units a model reads internally. Chunking splits documents into passages for retrieval, and it is a design choice you control.

Do I still need chunking with a long context window?

Usually yes. Sending everything is slower and costlier, and large inputs can still bury relevant details. Retrieval narrows the input to what matters.

What is semantic chunking?

A method that cuts where the topic changes, using embeddings to find the shifts. It can produce coherent chunks, but it costs more and should be compared with simpler methods on your data.

How does overlap help?

It repeats a little text across the boundary of two chunks so an idea that straddles the cut is not lost. Keep it modest to avoid duplicate results.

How do I chunk PDFs and tables?

Extract text with a parser that keeps structure, split by section, and keep table rows attached to their headers. Scans need text recognition first, and its output should be checked.

Do I need to re-chunk when my documents change?

Yes, for the changed documents. Track versions so you can update only what changed and remove outdated chunks from the index.

How is chunking related to hallucination?

Poor chunks can hide the right answer or show misleading fragments, and a model then fills the gap. Good chunking and retrieval are among the main defenses against hallucination.

Sources

Sources and further reading

The standards, rules, research and official documentation this page draws on.