RAG preparation guide
A PDF is not ready for RAG after extraction alone
Useful retrieval depends on reading order, heading structure, tables, provenance, chunk boundaries, and measured answer quality.
Why Markdown helps
Markdown makes headings, lists, code blocks, and tables explicit. Those markers are easier to inspect and split than PDF coordinates. Conversion does not solve retrieval by itself; it creates a reviewable intermediate representation.
Preserve provenance
Keep the source filename, page boundary, document version, and conversion date beside each chunk. Page comments in scanned-PDF output help trace an answer back to the scan. Do not remove them before the indexing pipeline has created stable citations.
Chunk from structure
- Split at meaningful headings before applying a token limit.
- Keep a table with its caption and explanatory paragraph.
- Repeat the nearest useful heading in chunk metadata.
- Use a small overlap only where sentences or lists cross the boundary.
- Never mix documents merely to fill a target chunk size.
Measure extraction before retrieval
| Check | What failure means |
|---|---|
| Heading accuracy | Chunks may lose their subject or hierarchy |
| Reading order | Sentences from columns or sidebars may be interleaved |
| Table cell accuracy | Retrieved numbers may be attached to the wrong label |
| Page provenance | Answers cannot be verified against the source |
Evaluate with real questions
Build a small set of questions whose answers and source pages are known. Measure whether retrieval returns the correct passage and whether the final answer cites it. Include cases with tables, footnotes, multi-column pages, and scans. A cleaner-looking Markdown file is not evidence of better retrieval.
Start with the digital PDF converter or the scanned PDF OCR converter, then retain the source for verification.