LangChain PDF loaders compared (and a cleaner Markdown-first recipe)

LangChain ships several PDF loaders, and they are thin wrappers around the underlying extraction libraries — so choosing a loader is choosing an extractor. The default PyPDFLoader gives you one plain-text Document per page, which is fine for prototypes and mediocre for retrieval: page-based splits cut through sections and flatten tables. For production RAG, either pick the loader whose underlying library matches your documents, or skip loaders entirely and go Markdown-first.

What each loader wraps

The trade-offs are the libraries’ trade-offs — speed, tables, licensing — covered in depth in pdfplumber vs PyMuPDF vs pypdf.

The structural problem all page loaders share

Every per-page loader hands your text splitter pages, but retrieval wants sections. A RecursiveCharacterTextSplitter over page text cuts mid-sentence, splits tables from headers, and embeds the page furniture (headers/footers) that makes unrelated chunks look similar.

The Markdown-first recipe

Convert the PDF to Markdown first, then split on headings — LangChain’s MarkdownHeaderTextSplitter exists for exactly this:

import requests
from langchain_text_splitters import MarkdownHeaderTextSplitter

def load_pdf_markdown(path: str) -> str:
    with open(path, "rb") as f:
        r = requests.post("https://api.marklipi.com/convert", files={"file": f}, timeout=60)
    r.raise_for_status()  # 422 → scanned PDF, route to OCR instead
    return r.json()["markdown"]

md = load_pdf_markdown("annual-report.pdf")

splitter = MarkdownHeaderTextSplitter(headers_to_split_on=[
    ("#", "h1"), ("##", "h2"), ("###", "h3"),
])
docs = splitter.split_text(md)  # Documents with heading metadata attached

What this buys you over a page loader:

• Chunks are sections, with the heading path in metadata — free context for embeddings and citations.
• Tables survive as Markdown tables inside a chunk instead of collapsing to value soup.
• No key, no dependency choice — the conversion is one keyless HTTP call (free tier 100/day), so there’s no pypdf-vs-PyMuPDF decision or AGPL question in your requirements.txt.
• Deterministic ingestion — re-running the pipeline produces identical chunks, which makes index diffs and eval regressions tractable.

The full chunking strategy (table atomicity, size caps within sections, OCR routing) is in the RAG parsing guide.

When a loader is still the right call

• Air-gapped or compliance-bound: files can’t leave the machine → PDFPlumberLoader (tables, MIT) or PyMuPDFLoader (speed, mind the AGPL).
• Scanned documents: no text layer to extract → UnstructuredPDFLoader with OCR, or a dedicated OCR step; text-layer tools including marklipi will refuse scans by design.
• You need per-page provenance: page-level citations are a product requirement → keep a page loader alongside, or record page offsets at conversion time.

FAQ

What’s the best PDF loader in LangChain?

For plain prototyping, PyPDFLoader. For table-heavy documents, PDFPlumberLoader. For production RAG over digital-native PDFs, converting to Markdown and using MarkdownHeaderTextSplitter usually beats all page-based loaders on retrieval quality.

Does LangChain have a native marklipi integration?

Not yet — but none is needed: the pattern above is four lines with requests, and the output plugs into LangChain’s Markdown splitter directly.

How do I handle scans in a LangChain pipeline?

Branch on the converter’s 422 response: text-layer PDFs take the cheap deterministic path, scans route to an OCR-capable loader. How to tell the two apart.