How to parse PDFs for RAG: a practical chunking guide

The highest-leverage step in a PDF RAG pipeline is the one most teams rush: parsing. If the extracted text is scrambled — tables flattened, columns interleaved, headers repeating mid-sentence — no embedding model or reranker can recover the meaning. The reliable recipe is: convert to Markdown first, chunk on the Markdown structure, and route scans separately.

Step 1: Parse to Markdown, not plain text

Markdown preserves the two things retrieval needs and plain text destroys: heading hierarchy (your chunk boundaries) and tables (your facts). For digital-native PDFs, deterministic text-layer extraction gets you there in one keyless call:

import requests

def pdf_to_markdown(path: str) -> str:
    with open(path, "rb") as f:
        r = requests.post("https://api.marklipi.com/convert", files={"file": f}, timeout=60)
    if r.status_code == 422:
        raise ValueError(f"{path} is a scan — route to OCR")
    r.raise_for_status()
    return r.json()["markdown"]

That 422 branch matters: it’s your router between the cheap deterministic path (most business PDFs) and the expensive OCR/vision path (scans). Mixing them silently is how OCR noise ends up embedded in your index. (Text layer vs OCR, explained.)

Step 2: Chunk on structure, not character counts

Fixed-size chunking (every 1,000 characters) cuts sentences, splits tables from their headers, and separates answers from the questions above them. With Markdown you can split on structure instead:

1. Split on headings (#/##/###) so each chunk is one coherent section.
2. Keep tables atomic — a table split in half is worse than useless, because half-tables retrieve confidently and answer wrongly.
3. Prepend the heading path to each chunk ("Annual Report > Liquidity > Cash flows") so the embedding carries context the raw paragraph lacks.
4. Then apply a size cap (e.g., ~1,500–3,000 characters) within sections, splitting on paragraphs.

Most frameworks have this built in — LangChain’s MarkdownHeaderTextSplitter, LlamaIndex’s MarkdownNodeParser — which is itself an argument for parsing to Markdown: the whole ecosystem chunks it well. (The LangChain-specific version of this pipeline.)

Step 3: The three mistakes that poison retrieval

• Embedding page furniture. Headers, footers, and page numbers repeat on every page; embedded, they make unrelated chunks look similar. Good converters drop them — verify by eyeballing your chunks.
• Trusting OCR output like text-layer output. OCR digits are guesses. If financial or legal precision matters, tag OCR-derived chunks so answers can carry a confidence caveat.
• Chunking before parsing is right. Test the parser on your ugliest 10 documents before tuning chunk sizes. A perfect chunker over scrambled text is lipstick on garbage.

Why Markdown also wins at query time

Retrieved chunks go back into the prompt, and Markdown costs 60–90% fewer tokens than the raw-PDF alternatives — the token math is here. Tables arrive as tables, so the model reasons over rows instead of transcribing pixels.

FAQ

What chunk size is best for PDF RAG?

Structure first, size second: whole sections capped around 1,500–3,000 characters works well with current embedding models. The cap matters less than not cutting through tables and sentences.

Should I use a vision model to parse PDFs for RAG?

Only for documents that need it — scans, photos, layout-heavy forms. For text-layer PDFs, deterministic extraction is exact, ~free, and reproducible; vision parsing adds cost and a nonzero hallucination rate at the ingestion stage, which is the worst place to add errors.

How do I handle a corpus that mixes digital and scanned PDFs?

Try text-layer extraction first and treat 422 as the routing signal to your OCR path. You get exactness where it’s possible and OCR only where it’s necessary.