Document Loaders & Text Splitters

1. The Core Document Abstraction

In LangChain, all textual content is represented as a standardized Document object with two attributes:

• page_content (str): The raw text extracted from the document.
• metadata (dict): Metadata dictionary preserving context (e.g., source, page, author, date).

# Example Document Object representation
Document(
    page_content="Distributed consensus algorithms like Raft and Paxos ensure...",
    metadata={"source": "whitepaper.pdf", "page": 4, "chapter": "Consensus Mechanics"}
)

2. Ingesting Multi-Source Formats

LangChain Community provides over 150+ native document loaders:

# Ingesting Web Documentation with WebBaseLoader
from langchain_community.document_loaders import WebBaseLoader, PyPDFLoader, CSVLoader

# 1. Ingest Web Content
web_loader = WebBaseLoader("https://docs.python.org/3/tutorial/classes.html")
web_docs = web_loader.load()
print(f"Loaded {len(web_docs)} web document(s). Length: {len(web_docs[0].page_content)} characters.")

# 2. Ingest Multi-Page PDF with page-level metadata
pdf_loader = PyPDFLoader("./sample_contract.pdf")
pdf_pages = pdf_loader.load()
print(f"Loaded {len(pdf_pages)} PDF pages. First page metadata: {pdf_pages[0].metadata}")

# 3. Ingest Structured CSV Rows
csv_loader = CSVLoader(file_path="./customers.csv", source_column="Customer_ID")
csv_records = csv_loader.load()

3. Chunking Strategies: RecursiveCharacterTextSplitter

LLMs have context window limits, and embedding entire 100-page documents into a single vector causes severe information dilution. You must chunk documents into concise, semantically coherent passages.

RecursiveCharacterTextSplitter is the gold standard chunker. It splits recursively by paragraphs (\n\n), then by sentences (\n), then by spaces ( ), and finally by characters ("") to keep semantically related sentences intact:

from langchain_text_splitters import RecursiveCharacterTextSplitter

# Configure recursive splitter with overlap
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,        # Maximum character count per chunk
    chunk_overlap=150,     # Overlap boundary to prevent severed context
    length_function=len,
    separators=["\n\n", "\n", " ", ""]
)

# Split long documents into passages while preserving original metadata
chunks = text_splitter.split_documents(pdf_pages)

print(f"Total original pages: {len(pdf_pages)}")
print(f"Total split chunks: {len(chunks)}")
print("Sample Chunk Metadata:", chunks[0].metadata)