This article continues the series of three articles I published in April on Retrieval-Augmented Generation (RAG). While the earlier articles focused on RAG fundamentals, this new series takes a deeper dive into the advanced concepts, techniques, and optimization strategies required to build enterprise-grade RAG solutions
Whether you're a practitioner, architect, engineer, or AI leader, this series is designed to help you move beyond the basics and develop a deeper understanding of modern RAG architectures and best practices. I hope you find it valuable, and I look forward to your feedback and insights throughout the series.
Introduction
Most teams obsess over which large language model they will use. Far fewer teams obsess over how they break their documents into chunks. That is backwards. In production RAG, chunking and document loading drive more variance in answer quality than the choice of generation model. This article is about the unglamorous, foundational work that separates a polished RAG system from a confusing one.
Why Chunking Matters
Chunking is the process of dividing each document into smaller fragments that will become individual entries in the vector store. The reason you do this is geometric. An embedding is the average meaning of whatever you embed. If you embed an entire 50 page policy as a single vector, that vector represents the average meaning of the whole document, and a precise question about page 27 is unlikely to match it well.
Chunk too aggressively and each fragment is so small that it loses the surrounding context that gives it meaning. Chunk too lightly and each fragment is so broad that retrieval becomes imprecise.
The Chunk Size Trade Off
Chunks that are too large dilute meaning. Chunks that are too small lose context. In practice, chunk sizes between 500 and 1500 characters with 150 to 250 character overlap are common starting points. The overlap matters. It ensures that ideas that straddle a chunk boundary are not amputated, because each idea will land fully inside at least one chunk
Why There Is No Universal Best Chunk Size
Different content types embed differently. Different question patterns retrieve differently. A regulatory document with long, dense paragraphs benefits from larger chunks. A FAQ database with self-contained question and answer pairs benefits from smaller, well bounded chunks. A codebase benefits from chunks that respect function boundaries. That is why chunking is, frustratingly, empirical. You guess, you measure, you iterate.
Splitter Strategies
Splitters are the algorithms that decide where to cut. The right splitter depends on the structure of your documents and the queries you expect.
1. Fixed Size Splitting
Cut every N characters or every N tokens, with an optional overlap window. Simplest, fastest, most predictable. Works well for prose with consistent paragraph structure. Tends to chop awkwardly across sentence boundaries.
2. Recursive Character Splitting
Try to split on paragraph breaks first, then sentences, then words, then characters, falling through to coarser cuts only when the target size is exceeded. Preserves structure better than fixed size. This is the default in most production stacks.
3. Semantic Splitting
Use an encoder to detect natural meaning shifts in the text and split on those boundaries. More expensive at index time, often higher quality at retrieval time. Useful for long form documents where topic shifts are subtle.
4. Document Structure Aware Splitting
Respect headings, sections, tables, and lists. A Markdown splitter cuts on heading levels. A code splitter cuts on function or class boundaries. A regulatory document splitter respects citation hierarchies like SR 11-7 paragraph references.
5. Late Chunking
A newer technique: embed the entire document first, then segment the resulting embeddings. This preserves cross chunk context that traditional chunk then embed approaches lose.
Document Loading by File Type
Before you chunk, you have to load. Each format has its quirks. The principle, again, is to use the right tool and not invoke a language model where a parser will do.
1. Markdown and Plain Text
Load directly. No conversion needed. These formats are already aligned with how language models consume content.
2. PDFs and Word Documents
Convert to Markdown or plain text using Python libraries. Tools include PyMuPDF, pdfplumber, python-docx, and unstructured. For scanned PDFs without an embedded text layer, run OCR first. Do not use an LLM to convert. It is slow, expensive, and rarely improves results.
3. Tabular and Structured Data
This is the one case where an LLM pass adds clear value. Embedding raw CSV rows performs poorly because the column names give little semantic context. Instead, generate a short natural language summary for each row or for each meaningful group of rows, embed the summary, and store the row reference in metadata. Retrieval then finds the right row by meaning, and downstream tools can fetch the structured record.
4. Images
Two options. The simpler option is to use an image to vector encoder, such as CLIP or one of its successors, that embeds images directly. The more flexible option is to use a vision language model to produce a textual caption or description of the image, then embed the text. Captioning is more controllable, more debuggable, and easier to evaluate.
5. Audio and Video
Transcribe with a speech to text model such as Whisper, then treat the transcript as text. Preserve timestamps in metadata so a retrieved passage can be linked back to the moment in the audio or video where it appeared.
Metadata, the Other Half of Chunking
A chunk is more than its text. Every chunk should carry metadata that supports filtering, governance, and citation. Common fields:
Source document, with a stable identifier.
Section heading and page number.
Date created and date last modified.
Author and approver.
Document classification, such as public, internal, confidential, or restricted.
Document type, such as policy, procedure, regulation, contract, or research.
Jurisdiction or business unit, where applicable.
Metadata enables three things you cannot live without in the enterprise:
Pre-filtering by attributes such as date or document type before vector search
Access control at query time so users only see chunks they are entitled to see
Trustworthy citations in the generated answer.
Practical Tuning Guidance
Practical Tuning Guidance
1. Adjust Top K (number of retrieved chunk per prompt) together with Chunk Size
When you experiment with chunk size, also adjust the number of chunks you retrieve so that the total amount of context passed to the LLM stays roughly constant across experiments. Otherwise you are measuring two changes at once and you will not know what caused the difference in quality.
2. Test with Real User Queries
Synthetic test questions usually look nothing like real user queries. Capture actual user questions, cluster them, and build your evaluation set around the top clusters. We will go deep on evaluation in Part 5.
3. Re Chunk When Source Documents Change Materially
If a policy is rewritten, do not just append. Re ingest. Stale chunks that reference deprecated wording will continue to surface and contaminate answers.
4. Keep an Eye on Token Cost
Larger chunks and larger top K both increase the size of the prompt sent to the generation model. That has a direct line to your monthly invoice. Chunking choices are also cost choices.
Common Mistakes
Using one chunking strategy across heterogeneous document types. Policies, emails, and dashboards each deserve a different approach.
Ignoring chunk overlap. Without it, ideas that straddle boundaries are lost.
Stripping headings during loading. Headings are some of the strongest semantic signals in a document.
Failing to enrich chunks with metadata. Filtering, governance, and citations all suffer when metadata is missing.
Embedding tables as raw rows. Always summarize first.
Treating chunking as a one-time decision. The right chunking strategy emerges from evaluation, and evaluation never ends.
A Working Mental Checklist
Before you press go on ingestion, ask:
Have I converted each format with the simplest viable parser, not with an LLM?
Have I picked a splitter that respects the structure of my documents?
Have I picked chunk sizes and overlaps appropriate to my data and queries?
Have I attached the metadata needed for filtering, access control, and citations?
Have I built a way to measure whether my chunking is actually working?
If you can answer yes to all five, you are ahead of most teams I have seen.
Closing Thought
Chunking and loading are the unfashionable parts of RAG and they are also where the biggest gains hide. The teams that take this layer seriously ship RAG systems that feel sharp. The teams that skim past it ship demos that disappoint in week three.
Explore more: Start Here · Topics · Archive · About