# How to parse PDFs for RAG: a practical chunking guide

> RAG quality is decided at parse time. How to convert PDFs to Markdown, chunk on structure instead of character counts, and avoid the three parsing mistakes that quietly poison retrieval.

Published: 2026-07-16 · Source: https://marklipi.com/blog/parse-pdfs-for-rag-chunking-guide


The highest-leverage step in a PDF RAG pipeline is the one most teams rush: parsing. If the extracted text is scrambled — tables flattened, columns interleaved, headers repeating mid-sentence — no embedding model or reranker can recover the meaning. The reliable recipe is: **convert to Markdown first, chunk on the Markdown structure, and route scans separately**.

## Step 1: Parse to Markdown, not plain text

Markdown preserves the two things retrieval needs and plain text destroys: **heading hierarchy** (your chunk boundaries) and **tables** (your facts). For digital-native PDFs, deterministic text-layer extraction gets you there in one keyless call:

```python
import requests

def pdf_to_markdown(path: str) -> str:
    with open(path, "rb") as f:
        r = requests.post("https://api.marklipi.com/convert", files={"file": f}, timeout=60)
    if r.status_code == 422:
        raise ValueError(f"{path} is a scan — route to OCR")
    r.raise_for_status()
    return r.json()["markdown"]
```

That `422` branch matters: it's your router between the cheap deterministic path (most business PDFs) and the expensive OCR/vision path (scans). Mixing them silently is how OCR noise ends up embedded in your index. ([Text layer vs OCR, explained.](/blog/pdf-text-layer-vs-ocr))

## Step 2: Chunk on structure, not character counts

Fixed-size chunking (every 1,000 characters) cuts sentences, splits tables from their headers, and separates answers from the questions above them. With Markdown you can split on structure instead:

1. **Split on headings** (`#`/`##`/`###`) so each chunk is one coherent section.
2. **Keep tables atomic** — a table split in half is worse than useless, because half-tables retrieve confidently and answer wrongly.
3. **Prepend the heading path** to each chunk (`"Annual Report > Liquidity > Cash flows"`) so the embedding carries context the raw paragraph lacks.
4. **Then apply a size cap** (e.g., ~1,500–3,000 characters) *within* sections, splitting on paragraphs.

Most frameworks have this built in — LangChain's `MarkdownHeaderTextSplitter`, LlamaIndex's `MarkdownNodeParser` — which is itself an argument for parsing to Markdown: the whole ecosystem chunks it well. ([The LangChain-specific version of this pipeline.](/blog/langchain-pdf-loaders-compared))

## Step 3: The three mistakes that poison retrieval

- **Embedding page furniture.** Headers, footers, and page numbers repeat on every page; embedded, they make unrelated chunks look similar. Good converters drop them — verify by eyeballing your chunks.
- **Trusting OCR output like text-layer output.** OCR digits are guesses. If financial or legal precision matters, tag OCR-derived chunks so answers can carry a confidence caveat.
- **Chunking before parsing is right.** Test the parser on your ugliest 10 documents *before* tuning chunk sizes. A perfect chunker over scrambled text is lipstick on garbage.

## Why Markdown also wins at query time

Retrieved chunks go back into the prompt, and Markdown costs 60–90% fewer tokens than the raw-PDF alternatives — the [token math is here](/blog/markdown-vs-pdf-llm-token-costs). Tables arrive as tables, so the model reasons over rows instead of transcribing pixels.

## FAQ

### What chunk size is best for PDF RAG?

Structure first, size second: whole sections capped around 1,500–3,000 characters works well with current embedding models. The cap matters less than not cutting through tables and sentences.

### Should I use a vision model to parse PDFs for RAG?

Only for documents that need it — scans, photos, layout-heavy forms. For text-layer PDFs, deterministic extraction is exact, ~free, and reproducible; vision parsing adds cost and a nonzero hallucination rate at the ingestion stage, which is the worst place to add errors.

### How do I handle a corpus that mixes digital and scanned PDFs?

Try text-layer extraction first and treat `422` as the routing signal to your OCR path. You get exactness where it's possible and OCR only where it's necessary.