indexion segment

Split text into contextual segments using divergence-based, TF-IDF, or punctuation strategies.

When to Use

• User needs to chunk text for RAG or embedding pipelines
• User wants to split a document into meaningful sections
• User asks to segment text for processing
• Preparing text for similarity analysis at sub-document level

Usage

# Default window divergence strategy
indexion segment <input-file> <output-dir>

# TF-IDF based segmentation
indexion segment --strategy=tfidf <input-file> <output-dir>

# Punctuation-based segmentation
indexion segment --strategy=punctuation <input-file> <output-dir>

# Custom segment sizes
indexion segment --min-size=200 --max-size=3000 --target-size=800 document.txt output/

# Custom divergence threshold
indexion segment --threshold=0.5 document.txt output/

# Adaptive threshold mode (default)
indexion segment --adaptive document.txt output/

# Hybrid NCD+TF-IDF mode
indexion segment --hybrid --ncd-weight=0.6 --tfidf-weight=0.4 document.txt output/

# Custom window size
indexion segment --window-size=5 document.txt output/

# Custom output prefix
indexion segment --prefix=chunk document.txt output/

Options

Strategies

Workflow

1. Run indexion segment <input-file> <output-dir> to split text with defaults
2. Adjust --threshold and --target-size to tune segmentation granularity
3. Use --hybrid mode for better accuracy on mixed-content documents