references/opendataloader.md
2.9 KB查看排版效果| 1 | # OpenDataLoader PDF Integration — Reference |
| 2 | |
| 3 | > Text-layer PDF parsing: JSON with bounding boxes, Markdown, HTML, Text, Tagged PDF. |
| 4 | > Supports table detection, reading order (XY-Cut++), sanitization. |
| 5 | |
| 6 | ## Environment |
| 7 | |
| 8 | Requires Java 11+ and the `opendataloader-pdf` PyPI package (~21.5MB, includes 23MB Java CLI JAR). |
| 9 | |
| 10 | ```bash |
| 11 | java -version 2>&1 | head -1 # must be 11+ |
| 12 | python3 -c "import opendataloader_pdf; print('✅')" 2>&1 || echo "⚠️ not installed" |
| 13 | ``` |
| 14 | |
| 15 | ### Installation |
| 16 | |
| 17 | ```bash |
| 18 | # 1. Java (if missing) |
| 19 | sudo apt-get install -y -qq default-jre |
| 20 | |
| 21 | # 2. Python SDK (use Tsinghua mirror for speed — official PyPI often times out) |
| 22 | python3 -m venv ~/.venvs/opendataloader |
| 23 | ~/.venvs/opendataloader/bin/pip install -i https://pypi.tuna.tsinghua.edu.cn/simple opendataloader-pdf |
| 24 | |
| 25 | # 3. Optional: add to PATH |
| 26 | ln -sf ~/.venvs/opendataloader/bin/opendataloader-pdf ~/.local/bin/ |
| 27 | ``` |
| 28 | |
| 29 | **Key Finding**: Official PyPI (files.pythonhosted.org) downloads frequently time out; Tsinghua mirror delivers at 60+ MB/s. |
| 30 | |
| 31 | ## CLI Usage |
| 32 | |
| 33 | ```bash |
| 34 | # Basic: JSON + Markdown |
| 35 | opendataloader-pdf input.pdf -f json,markdown -o ./output_dir |
| 36 | |
| 37 | # All formats |
| 38 | opendataloader-pdf input.pdf -f json,markdown,html,text,tagged-pdf -o ./output_dir |
| 39 | |
| 40 | # Stdout (single format only) |
| 41 | opendataloader-pdf input.pdf -f markdown --to-stdout 2>/dev/null |
| 42 | |
| 43 | # Page range |
| 44 | opendataloader-pdf input.pdf -f json -o ./output --pages "1,3,5-7" |
| 45 | |
| 46 | # Sanitize (redact emails, phones, bank card numbers) |
| 47 | opendataloader-pdf input.pdf -f json -o ./output --sanitize |
| 48 | |
| 49 | # Table detection mode |
| 50 | opendataloader-pdf input.pdf -f json -o ./output --table-method cluster |
| 51 | |
| 52 | # Keep original line breaks |
| 53 | opendataloader-pdf input.pdf -f markdown -o ./output --keep-line-breaks |
| 54 | ``` |
| 55 | |
| 56 | ## Python SDK |
| 57 | |
| 58 | ```python |
| 59 | from opendataloader_pdf import convert |
| 60 | convert( |
| 61 | "/path/to/contract.pdf", |
| 62 | output_dir="/path/to/output", |
| 63 | format="json,markdown,html", |
| 64 | reading_order="xycut", # XY-Cut++ reading order (default) |
| 65 | table_method="default", # or "cluster" |
| 66 | pages="1,3,5-7", |
| 67 | sanitize=False, |
| 68 | keep_line_breaks=False |
| 69 | ) |
| 70 | ``` |
| 71 | |
| 72 | ## Output Formats |
| 73 | |
| 74 | | Format | Content | Use Case | |
| 75 | |--------|---------|----------| |
| 76 | | `json` | type/hierarchy/content/bounding box/font/size/color per element | Programmatic processing, positional traceability | |
| 77 | | `markdown` | Structured text with headings, lists, tables | LLM input, quick reading | |
| 78 | | `html` | Layout-preserving page | Web preview | |
| 79 | | `text` | Plain text | Search, simple analysis | |
| 80 | | `tagged-pdf` | Tagged PDF | Accessibility (PDF/UA) | |
| 81 | |
| 82 | ## Performance |
| 83 | |
| 84 | - **Local deterministic mode**: ~0.015s/page (pure Java, no API calls) |
| 85 | - **AI hybrid mode**: Complex pages (OCR) need extra Hybrid service |
| 86 | - **No network required**: Local mode fully offline |
| 87 | - **23MB JAR**: Large initial download |
| 88 | |
| 89 | ## Limitations |
| 90 | |
| 91 | - Source PDF must have extractable text layer (scanned docs need PaddleOCR path — see main pdf-ocr-md skill) |
| 92 | - Scanned OCR recommended path: PaddleOCR (Chinese) or OpenDataLoader hybrid mode |
文件内容是平台检查时读出的文本，只供查看。平台不会运行包里的任何脚本或程序。
