Large Document Processing & Intelligent Text Chunking

## ⚠️ Repo Reality Check (read this first)

The real components are:
- Top-level pipeline: [LargeDocumentProcessor](../../../src/pipeline/largedocumentprocessor.py#L43)
- Structure-aware parser: [AdvancedDocumentParser](../../../src/ocr/advanceddocumentparser.py#L66)
- Streaming OCR with progress: [EnhancedOCRProcessor](../../../src/ocr/enhancedocrprocessor.py#L34)
- Chunker: [IntelligentTextChunker](../../../src/utils/intelligent_chunker.py#L47) — see the [intelligent-text-chunking](../intelligent-text-chunking/SKILL.md) skill.
- Training data generation: [AITrainingDataGenerator](../../../src/training/aitraininggenerator.py#L39)
- Setup helper: [scripts/setuplargedocumentprocessing.py](../../../scripts/setuplargedocumentprocessing.py)

The NWT EPUB parser exposes only getverse(booknum, chapter, verse) ([nwtepubparser.py](../../../src/utils/nwtepubparser.py#L63)) — there is no getchapter / getbook. See the [bible-epub-processing](../bible-epub-processing/SKILL.md) skill.

Source data lives under [config/data/](../../../config/data/) (NOT a top-level data/).

Always wrap chunking calls with protectscripturereferences / restorescripturereferences from [src/utils/scriptureparser.py](../../../src/utils/scriptureparser.py#L77) when input may contain Bible references.

Overview

Two tightly related concerns combined here:

1. Large document parsing — DOCX/PDF/EPUB ingestion with structure preservation
2. Intelligent text chunking — splitting parsed text into semantically coherent pieces for AI training or RAG

Source Files

Document Processing

Supported Formats

• DOCX via python-docx
• PDF via PyMuPDF (import as fitz) — note: fitz==0.0.1.dev2 is NOT in requirements; use PyMuPDF only
• EPUB via ebooklib + NWTEpubParser
• Plain text / CSV — direct read

EPUB Pattern (NWT Bible)

from src.utils.nwt_epub_parser import NWTEpubParser

parser = NWTEpubParser('data/bible/nwt_E.epub')
verse_text = parser.get_verse('John', 3, 16)
chapter_verses = parser.get_chapter('Genesis', 1)

PDF/DOCX Pattern

import fitz  # PyMuPDF — installed as PyMuPDF, exposed as fitz

doc = fitz.open('large_document.pdf')
for page_num, page in enumerate(doc):
    text = page.get_text()
    # process text...

Intelligent Text Chunking

Strategy Selection

Implementation Pattern

# Sentence-boundary-aware chunking
def chunk_text(text: str, max_chars: int = 1024, overlap: int = 100) -> list[str]:
    sentences = re.split(r'(?<=[.!?])\s+', text)
    chunks, current = [], ''
    for sent in sentences:
        if len(current) + len(sent) > max_chars and current:
            chunks.append(current.strip())
            current = current[-overlap:] + ' ' + sent  # overlap
        else:
            current += ' ' + sent
    if current.strip():
        chunks.append(current.strip())
    return chunks

Chuukese-aware chunking

# Chuukese uses the same sentence terminators as English
SENTENCE_ENDINGS = re.compile(r'(?<=[.!?])\s+')

def detect_language(text: str) -> str:
    has_accents = bool(re.search(r'[áéíóú]', text))
    return 'chuukese' if has_accents else 'english'

Memory Efficiency

• Process large PDFs page-by-page, not loading the full DOM into memory
• Stream EPUB chapters — do not load the entire book at once
• Write chunk output incrementally to JSONL files rather than accumulating in RAM

Output Formats

• JSONL: one JSON object per line — best for large training datasets
• JSON array: for smaller batches consumed by the frontend
• Plain text: cleaned extracted text for inspection

Dependencies

• PyMuPDF==1.23.8 — PDF processing (do NOT add fitz==0.0.1.dev2)
• python-docx>=1.2.0
• ebooklib>=0.18
• beautifulsoup4>=4.12.0