Document loaders and chunking strategies

Why this matters

Most "bad RAG" is bad chunking. If your chunks are too small, you lose context. Too big, you lose precision. Wrong split point, you cut a sentence in half and the embedding becomes noise. This single lesson is responsible for a 20-50% improvement on most production RAG systems.

Learning objectives

1. Pick the right document loader for your file type (PDF, HTML, DOCX, code, tables).
2. Apply 5 chunking strategies and know when to use each.
3. Implement parent-child and late chunking patterns.
4. Generate "contextual" chunks (Anthropic 2024) for 35-50% retrieval lift.

1. Document loaders — a tour by file type

Bad input = bad index. Pick the right loader for the file type.

# uv add unstructured[all-docs]
from unstructured.partition.auto import partition

elements = partition("./contract.pdf", strategy="hi_res")
for el in elements[:5]:
    print(el.category, "|", el.text[:80])

Unstructured returns typed Elements (Title, NarrativeText, Table, ListItem, ...) — extremely useful for layout-aware chunking later.

2. The 5 chunking strategies (with when-to-use)

A. Fixed-size with overlap (the baseline)

Split every N characters, keep an overlap window so context is not lost at boundaries.

# uv add langchain
from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,
    chunk_overlap=100,
    separators=["\n\n", "\n", ". ", "? ", "! ", " ", ""],
)
chunks = splitter.split_text(big_text)

RecursiveCharacterTextSplitter tries the separators in order: paragraph → newline → sentence → word → character. So you rarely cut mid-sentence.

When: unstructured prose, default starting point.

B. Sentence / semantic chunking

Split on natural sentence boundaries, then merge until you hit a token target.

# uv add nltk
import nltk; nltk.download("punkt_tab")
sents = nltk.sent_tokenize(big_text)
chunks = []; buf = ""
for s in sents:
    if len(buf) + len(s) > 800:
        chunks.append(buf); buf = ""
    buf += s + " "
if buf: chunks.append(buf)

When: clean prose where sentence boundaries matter (legal, scientific).

C. Semantic chunking (embedding-aware)

Embed each sentence; split where consecutive embeddings drop below a similarity threshold (i.e. the topic changes).

# uv add llama-index-core
from llama_index.core.node_parser import SemanticSplitterNodeParser
from llama_index.embeddings.openai import OpenAIEmbedding

splitter = SemanticSplitterNodeParser(
    buffer_size=1, breakpoint_percentile_threshold=95,
    embed_model=OpenAIEmbedding(model="text-embedding-3-small"),
)
nodes = splitter.get_nodes_from_documents(documents)

When: mixed-topic articles, blog posts, transcripts. More expensive but cleaner topical chunks.

D. Structure-aware (Markdown / HTML / code)

Use the document's own structure as the chunk boundary.

from langchain_text_splitters import MarkdownHeaderTextSplitter

splitter = MarkdownHeaderTextSplitter(headers_to_split_on=[
    ("#", "h1"), ("##", "h2"), ("###", "h3"),
])
docs = splitter.split_text(md_text)  # returns Documents with header metadata

For code:

splitter = RecursiveCharacterTextSplitter.from_language(
    language="python", chunk_size=600, chunk_overlap=80,
)

When: docs with strong structure — Wikipedia, READMEs, technical specs, source code. The metadata (h1, h2) is gold for retrieval filters and for showing the model where it is.

E. Late chunking (2024-25 hot trend)

Embed the whole document at once, then slice the embedding sequence into chunk-level pooled vectors. Each chunk vector "knows" the whole document context.

Requires a long-context embedder like jina-embeddings-v3 or nomic-embed-text-v2. Most accuracy per chunk, more compute.

When: dense, cross-referenced documents (legal, scientific). You want every chunk to "remember" the full document tone.

3. Parent-child chunking (a production must-know)

Idea: index small chunks for retrieval precision, but return their larger parent chunks to the LLM for synthesis.

[Parent: 2000-token section]
  ├─ child 1 (300 tokens)
  ├─ child 2 (300 tokens)
  └─ child 3 (300 tokens)

Retrieve children → look up parents → pass parents to the LLM. Best of both worlds.

# Implementation sketch with ChromaDB
parents = make_large_chunks(text, size=2000)
for pid, parent in enumerate(parents):
    children = make_small_chunks(parent, size=300, overlap=50)
    for cid, child in enumerate(children):
        collection.add(
            ids=[f"{pid}-{cid}"],
            documents=[child],
            metadatas=[{"parent_id": pid}],
        )
parent_store = {pid: parents[pid] for pid in range(len(parents))}

def retrieve(q):
    res = collection.query(query_texts=[q], n_results=10)
    parent_ids = {m["parent_id"] for m in res["metadatas"][0]}
    return [parent_store[pid] for pid in parent_ids]

LangChain has ParentDocumentRetriever, LlamaIndex has HierarchicalNodeParser if you do not want to roll your own.

4. Contextual chunking (Anthropic 2024) — the easiest +35-50% lift

Naive chunks lose document-level context. Contextual chunking prepends a 1-2 line summary of where the chunk fits in the document before embedding.

The recipe:

import anthropic
ac = anthropic.Anthropic()

CONTEXT_PROMPT = """\
<document>
{doc}
</document>

Here is the chunk we want to situate within the whole document
<chunk>
{chunk}
</chunk>

Please give a short succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk. Answer only with the succinct context and nothing else."""

def situate(chunk: str, document: str) -> str:
    r = ac.messages.create(
        model="claude-haiku-4-5",
        max_tokens=200,
        system=[{
            "type": "text",
            "text": CONTEXT_PROMPT.split("Here is the")[0] + "Document above is the full source.",
            "cache_control": {"type": "ephemeral"},
        }],
        messages=[{"role": "user", "content": CONTEXT_PROMPT.format(doc=document, chunk=chunk)}],
    )
    return r.content[0].text

Then store context + chunk as the document, and embed that.

Why prompt caching matters: you call this thousands of times per document, and the document is the same. Cache the document → 90% off → near-free at scale.

Combined with hybrid search (next lesson) and reranking, Anthropic measured a 67% reduction in failed retrievals on their internal benchmarks.

5. Tagging chunks with metadata (do not skip)

Every chunk you store should have:

• doc_id, chunk_id, parent_id (for parent-child)
• source ("github://..." or "drive://...")
• created_at / updated_at (for freshness filters)
• lang ("en", "hi", ...)
• tenant_id / acl_groups (for permissions)
• tags ([list of topics])
• chunk_index (position in doc — useful for ordering)

This metadata is what turns "search" into "search the right things."

6. Picking chunk size — the cheat sheet

Tune empirically with RAGAS (Lesson 2.5). What works for one corpus rarely works for another.

Hands-on lab (4 hours)

Take a real PDF (e.g., the OpenAI Cookbook or your company handbook).

1. Load it with Unstructured (strategy="hi_res" for layout).
2. Chunk in three ways: fixed (800/100), semantic (LlamaIndex), structure-aware (MarkdownHeaderTextSplitter after converting to MD).
3. For each strategy, embed with text-embedding-3-small into a separate Chroma collection.
4. Build a 30-question eval set (write expected answers manually).
5. Compare retrieval-recall@5 on each strategy.
6. Add a contextual chunking variant on top of the winner. Re-evaluate.
7. Print a markdown table of recall@5 per strategy.

Goal: produce data that lets you say "contextual chunking lifted recall@5 from 0.74 to 0.91" — a portfolio-grade story.

Common pitfalls

1. Forgetting overlap — answers near boundaries are missed.
2. Splitting tables in half — extract tables separately, embed as JSON.
3. Same chunk size for code and prose — code wants smaller, syntax-aware chunks.
4. Not tracking chunk_index — you cannot reconstruct context order.
5. Re-chunking on every change — version your chunking pipeline; expensive to redo.

Self-check

1. What is the difference between sentence chunking and semantic chunking?
2. Why does parent-child chunking improve precision and recall simultaneously?
3. What is the role of cache_control in contextual chunking?
4. When does late chunking beat semantic chunking?
5. Why must metadata include tenant_id for SaaS apps?

References

• Anthropic Contextual Retrieval blog
• Anthropic contextual cookbook
• Late chunking paper (Jina, 2024)
• LlamaIndex Semantic Chunking
• LangChain text splitters
• Unstructured.io docs
• Docling (IBM)