On-disk RAG evaluation (corpus/ + train.json)

Purpose

Guide agents through NVIDIA RAG Blueprint filesystem benchmarks: preparing corpus/ and train.json, running scripts/eval/evaluate_rag.py, tuning retrieval and generation flags for quality comparisons, interpreting RAGAS JSON outputs, and triaging failures (HTTP/stream errors, empty contexts, collection mismatch, judge API).

For latency, throughput, and load testing, use the rag-perf skill (scripts/rag-perf, docs/performance-benchmarking.md) — not this skill.

When not to use

Do not use this skill for: deploying or repairing services (use rag-blueprint); evaluating APIs without the corpus/ + train.json layout; general ML experimentation unrelated to this evaluator; production monitoring/alerting; or latency/throughput benchmarking (use rag-perf).

Prerequisites

• Repo cloned; run commands from repo root (imports and paths assume this).
• Python 3.11+ and uv; eval deps: uv sync --project scripts/eval.
• Reachable RAG server and ingestor (defaults often localhost:8081 / 8082).
• NVIDIAAPIKEY for RAGAS (see [credential hygiene](references/benchmark-execution.md#credential-hygiene-nvidiaapikey)); optional RAGEVALJUDGE_MODEL.
• Dataset roots passed to --dataset-paths each contain corpus/ and train.json.

Instructions

1. Prepare data — Ensure each dataset directory matches the layout and train.json rules in [references/dataset-and-conversion.md](references/dataset-and-conversion.md). When sources arrive as public links (sites or dataset pages), materialize documents under corpus/—prefer PDF for multimodal content so images stay embedded; convert CSV/JSONL/etc. using the patterns there.
2. Run eval — uv run --project scripts/eval python scripts/eval/evaluate_rag.py with --dataset-paths, --host, and --port. See [references/benchmark-execution.md](references/benchmark-execution.md) for command examples, outputs, and errors. Use [references/evaluate-rag-cli.md](references/evaluate-rag-cli.md) for flag-level detail.
3. Tune quality — Adjust --topk / --vdbtop_k, reranker and query-rewriting toggles, and generation overrides (--temperature, --top-p, --max-tokens) as documented in [references/benchmark-execution.md](references/benchmark-execution.md) when comparing retrieval/generation configs for RAGAS scores.
4. Analyze results — Use [references/result-analysis.md](references/result-analysis.md) for scripts; scan rag*evaluation_summary.json for headline RAGAS metrics.
5. Triage errors — Use the [error signal table](references/benchmark-execution.md#common-error-cases-and-signals) and the Troubleshooting section below.

Examples

Set API key without putting secrets in shell history (preferred patterns): load from a gitignored env file or secrets manager; avoid committing .env; rotate keys if exposed. Details: [references/benchmark-execution.md#credential-hygiene-nvidiaapikey](references/benchmark-execution.md#credential-hygiene-nvidiaapikey).

Minimal eval (key already in environment):

uv sync --project scripts/eval
uv run --project scripts/eval python scripts/eval/evaluate_rag.py \
  --dataset-paths /path/to/my_dataset \
  --host localhost \
  --port 8081

Pretty-print summary JSON:

python3 -m json.tool results/my_dataset/rag_my_dataset_evaluation_summary.json

More examples (skip ingestion, quality sweeps): [references/benchmark-execution.md](references/benchmark-execution.md).

Limitations

• Evaluator behavior is fixed to the filesystem contract and evaluate_rag.py; it does not substitute for custom offline judges or non-RAG benchmarks.
• Vector DB / embedding choices follow deployed ingestor and RAG env — not overridden by this CLI alone.
• Scores depend on retrieval quality, judge model availability, and NVIDIAAPIKEY; empty contexts yield partial RAGAS metrics (see references).
• Large procedural detail lives under references/ to keep routing concise; read those files when the user needs step-by-step conversion, full flags, or error tables.

Troubleshooting

Full signal table: [references/benchmark-execution.md#common-error-cases-and-signals](references/benchmark-execution.md#common-error-cases-and-signals).

Gotchas

• Run from repo root: paths and imports in scripts/eval/evaluate_rag.py assume this; a wrong directory silently breaks imports.
• --ingestorserverurl: pass http://host:port without /v1—the code appends /v1/ automatically. Including /v1 causes 404s on ingestor calls.
• Vector DB / embedding settings: not set by this CLI; configure via the deployed ingestor and RAG server env vars (e.g. APPVECTORSTOREURL, embedding model).
• --model / --llm_endpoint: forwarded verbatim only when explicitly set; omit to keep the server's configured LLM.
• Stale collections: a previous run's ingested data persists unless you use --force_ingestion. Use --collection with a unique name when comparing quality across isolated runs.
• Empty context metrics: if all generatedcontexts are empty, RAGAS scores only nvaccuracy and leaves the other two metrics blank—this is not a silent success.

Source of truth

Agent playbook

1. Run eval — uv sync --project scripts/eval then uv run --project scripts/eval python scripts/eval/evaluaterag.py with required --dataset-paths, --host, and --port (and env NVIDIAAPIKEY). Argument --ingestorserver_url is optional (defaults to http://localhost:8082); pass it only when overriding the ingestor endpoint.
2. Quality tuning — See [references/benchmark-execution.md](references/benchmark-execution.md): --topk/--vdbtop_k, reranker and query-rewriting toggles, --temperature, --top-p, --max-tokens.
3. Data conversion — Follow [references/dataset-and-conversion.md](references/dataset-and-conversion.md).
4. Analyze results — [references/result-analysis.md](references/result-analysis.md); quick scan: python3 -m json.tool results/<dataset>/rag<dataset>evaluation_summary.json.
5. Error triage — [references/benchmark-execution.md#common-error-cases-and-signals](references/benchmark-execution.md#common-error-cases-and-signals).