CSV Wrangling with qsv

Standard Workflow Order

Always follow this sequence when processing CSV data:

1. Setup (Cowork) - If relative paths don't resolve, call mcpqsvqsvgetworkingdir and mcpqsvqsvsetworkingdir to sync
2. Index - index (enables fast random access for subsequent commands)
3. Discover - sniff (detect format, encoding, delimiter) -> headers -> count
4. Profile - stats --cardinality --stats-jsonl (creates cache used by smart commands)
5. Inspect - slice --len 5 (preview rows), frequency --frequency-jsonl (value distributions with cache for reuse)
6. Transform - select, sort, dedup, rename, replace, search, sqlp, etc.
7. Validate - validate (against JSON Schema), stats (verify results)
8. Export - tojsonl, table, mcpqsvqsvtoparquet, to (xlsx/sqlite/postgres/ods/datapackage)
9. Document - describegpt --all (AI-generated Data Dictionary, Description & Tags)

Tool Selection Matrix

qsv Selection Syntax

Used by select, search, sort, dedup, frequency, and other commands:

Common Pipeline Patterns

Clean and Deduplicate

sniff -> index -> safenames -> fixlengths -> sqlp (TRIM) -> dedup -> validate

Profile and Analyze

sniff -> index -> stats --cardinality --stats-jsonl -> read .stats.csv -> frequency (on key columns) -> sqlp (GROUP BY queries)

Before writing SQL: read .stats.csv to learn column types, cardinality, nullcount, min/max, sort order. Run frequency on columns you'll GROUP BY or filter on. Use this to write precise WHERE clauses, correct type casts, and avoid unnecessary COALESCE.

For repeated SQL queries on large CSV (> 10MB), consider converting to Parquet: sniff -> index -> stats -> toparquet -> sqlp (using readparquet()). Note: sqlp can query CSV of any size directly.

Join and Enrich

index (both files) -> stats (both) -> joinp -> select (keep needed columns) -> sort

Profile and Document

sniff -> index -> stats --cardinality --stats-jsonl -> describegpt --all

Convert and Export

excel (to CSV) -> index -> stats -> select -> tojsonl / qsv_to_parquet

Batch Convert to Multiple Formats

excel (to CSV) -> index -> stats -> to xlsx report.xlsx
excel (to CSV) -> index -> stats -> to sqlite report.db
excel (to CSV) -> index -> stats -> to parquet parquet_output_dir

File Integrity Verification

blake3 file.csv > checksums.b3 (before transfer) -> blake3 --check checksums.b3 (after transfer)

Delimiter Handling

• CSV (,): default, no flag needed
• TSV (\t): use --delimiter '\t' or file extension .tsv
• SSV (;): use --delimiter ';' or file extension .ssv
• Auto-detect: set QSVSNIFFDELIMITER=1 environment variable

Important Notes

• Column indices are 1-based, not 0-based
• --no-headers flag changes behavior significantly - most commands assume headers exist
• Output goes to stdout by default; use --output file.csv to write to file
• Many commands auto-detect .sz (Snappy compressed) files transparently
• cat rows requires same column order; use cat rowskey for different schemas
• dedup loads all data into memory and sorts internally; use --sorted flag if input is already sorted to enable streaming mode with constant memory
• sort loads entire file into memory; for huge files use sqlp with ORDER BY
• For repeated SQL queries on large CSV (> 10MB), consider converting to Parquet with mcpqsvqsvtoparquet for faster performance. Parquet works ONLY with sqlp and DuckDB — all other qsv commands need CSV/TSV/SSV input

Tool Discovery

Use mcpqsvqsvsearchtools to discover commands beyond the initially loaded core tools. There are 55 qsv skill-based commands covering selection, filtering, transformation, aggregation, joining, validation, formatting, conversion, and more.

Operational Notes

• Timeout: Default operation timeout is 10 minutes (configurable via QSVMCPOPERATIONTIMEOUTMS, max 30 min). Allow operations to run to completion.
• Memory: dedup, sort, reverse, table, transpose, pragmastat, and stats (with extended stats) load entire files into memory. For files >1GB, prefer extdedup/extsort via mcpqsvqsv_command.
• Cowork path architecture: qsv runs on the HOST machine. File paths must be valid on the host. Always verify with mcpqsvqsvgetworking_dir.
• Sequential operations: Prefer sequential over parallel qsv calls to avoid queuing delays: index → stats → analysis.
• Large files (>5GB): Let mcpqsvqsvfrequency run to completion. Only fall back to mcpqsvqsvsqlp with GROUP BY if the server timeout is exceeded.
• Context window: Save outputs to files rather than returning to chat. Use mcpqsvqsvslice or mcpqsvqsvsqlp with LIMIT to inspect subsets.