Testing Pipeline Templates

Three-phase process: dry run → batch → quality iteration.

Phase 1: Dry Run (Single Seed)

Verify the pipeline executes and produces structurally correct output.

# list pipelines
curl -s http://localhost:8000/api/pipelines | python -m json.tool

# validate seeds against pipeline
curl -s -X POST http://localhost:8000/api/seeds/validate \
  -H 'Content-Type: application/json' \
  -d '{"pipeline_id": <ID>, "seeds": [{"repetitions": 1, "metadata": {"content": "test"}}]}' \
  | python -m json.tool

# execute single seed
curl -s -X POST http://localhost:8000/api/pipelines/<ID>/execute \
  -H 'Content-Type: application/json' \
  -d '{"content": "test input"}' | python -m json.tool

Analyze the response:

• result — expected output fields present?
• trace — each entry has blocktype, executiontime, output
• accumulated_state — data flowing correctly between blocks?

Check review readiness:

• Look at the last trace entry's accumulated_state — these are the fields visible in the Review UI
• All reviewer-relevant fields should be first-level keys (not nested under generated or other objects)
• If useful fields are nested, add a FieldMapper as the last block to flatten them (see creating-pipeline-templates skill)

Red flags: missing fields, metadata pollution (extra fields like samples, targetcount), executiontime >30s, empty/null generator outputs, reviewer-relevant data buried in nested objects.

Phase 2: Small Batch

Check output consistency and diversity across multiple seeds.

# prepare diverse seed file (3-5 seeds)
cat > test_seeds.json << 'EOF'
[
  {"repetitions": 2, "metadata": {"content": "topic A"}},
  {"repetitions": 2, "metadata": {"content": "topic B"}},
  {"repetitions": 1, "metadata": {"content": "edge case"}}
]
EOF

# start batch job
curl -s -X POST http://localhost:8000/api/generate \
  -F "pipeline_id=<ID>" -F "file=@test_seeds.json" | python -m json.tool

# monitor (poll until status=completed)
curl -s http://localhost:8000/api/jobs/<JOB_ID> | python -m json.tool

# review all records
curl -s "http://localhost:8000/api/records?job_id=<JOB_ID>" | python -m json.tool

Analyze patterns: output diversity, consistent field structure, validation pass/fail rates, common failure patterns.

Phase 3: Quality Iteration

Iteration loop:

1. Identify weakest block from traces
2. Adjust ONE thing (prompt, temperature, schema, or validation rules)
3. Re-run single seed → verify improvement
4. Re-run small batch → confirm consistency
5. Repeat

Prompt tips:

• Be specific: "Generate a 2-3 sentence description" not "Generate a description"
• Add examples in system prompt for few-shot guidance
• Use constraints: "Do not include personal opinions"
• Reference state: "Based on: {{ content }}"

Cleanup

curl -s -X DELETE "http://localhost:8000/api/records?job_id=<JOB_ID>"
curl -s -X DELETE http://localhost:8000/api/pipelines/<ID>

Key Endpoints

Related Skills

• creating-pipeline-templates — build the template first
• debugging-pipelines — when execution fails with errors
• configuring-models — set up LLM/embedding providers