Document Chunking Strategies & Parsing

1. The Vector Dilution Dilemma

Why can't we simply embed an entire 30-page PDF document into a single vector? Because transformer embedding models (like text-embedding-3-small) compress all tokens into a fixed coordinate space (e.g., 1,536 floats).

2. Chunking Methodologies Compared

3. Tuning Chunk Size & Overlap

Two primary hyperparameters govern text splitting:

• chunk_size: The maximum number of characters or tokens per chunk. Too small (e.g., 50 tokens) loses vital surrounding context; too large (e.g., 2,000 tokens) dilutes semantic precision.
• chunk_overlap: The number of overlapping characters/tokens between adjacent chunks (typically 10% to 20% of chunk_size). Overlap ensures that a sentence split across a boundary does not lose its meaning.

4. Hands-On Python: Implementing Splitting Strategies

Here is a practical Python script demonstrating both recursive character splitting and markdown header parsing:

from langchain_text_splitters import (
    RecursiveCharacterTextSplitter,
    MarkdownHeaderTextSplitter
)

raw_markdown = """# Enterprise AI Architecture

## 1. Vector Database Selection
Vector databases provide approximate nearest neighbor search over dense embeddings.
Pinecone offers managed cloud scalability.
ChromaDB is ideal for embedded local workflows.

## 2. Ingestion Pipeline
Always sanitize and normalize text before computing embeddings.
Ensure chunk overlap is between 10% and 20% to avoid boundary context loss.
"""

# Strategy A: Structure-Aware Markdown Header Splitting
headers_to_split_on = [
    ("#", "Header_1"),
    ("##", "Header_2"),
]
markdown_splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=headers_to_split_on,
    strip_headers=False
)
header_chunks = markdown_splitter.split_text(raw_markdown)

print(f"=== Markdown Header Splits ({len(header_chunks)} chunks) ===")
for i, chunk in enumerate(header_chunks, 1):
    print(f"Chunk {i} Metadata:", chunk.metadata)
    print(f"Chunk {i} Text Preview:", chunk.page_content[:60], "...\n")

# Strategy B: Recursive Character Text Splitting (Secondary refinement)
recursive_splitter = RecursiveCharacterTextSplitter(
    chunk_size=150,
    chunk_overlap=20,
    separators=["\n\n", "\n", " ", ""]
)
sub_chunks = recursive_splitter.split_documents(header_chunks)
print(f"Total refined sub-chunks: {len(sub_chunks)}")

5. Parent Document Retriever (Small-to-Big Retrieval)

In production RAG, you often encounter a painful conflict:

The Parent Document Retriever solves this elegantly:

1. A large document is split into 1,500-token Parent Chunks and stored in a Key-Value Docstore (e.g. Redis, MongoDB, or disk).
2. Each parent is sub-split into small 200-token Child Chunks and indexed in the Vector Database with a parent_id pointer.
3. When a user query arrives, the Vector Database matches the child chunk, but the retriever fetches and injects the parent chunk into the LLM prompt.