diegosouzapw/omniroute

cli-eval

Create and run evaluation suites, watch live benchmark progress, view scorecards, compare model performance, and integrate eval runs with CI workflows from the CLI.

Trending #5262 First seen Jun 10, 2026

Installation

$ npx skills add diegosouzapw/omniroute --skill cli-eval

Similar popular skills

Related neighbors and high-traction skills in the same topics — useful to compare before installing.

Also in this package

Other skills from diegosouzapw/omniroute · top by installs.

npx skills add diegosouzapw/omniroute

Browse all from diegosouzapw/omniroute

More details

Agent compatibility

Declared targets from SKILL.md / docs. Unmarked agents are not listed — the skill may still install via the CLI.

Claude Code Not declared
Cursor Not declared
Codex Not declared
GitHub Copilot Not declared
Windsurf Not declared
Gemini CLI Not declared
Cline Not declared
OpenCode Not declared

Repository health

Stars 62.5K
License LICENSE
Default branch release/v3.8.51
Open issues 279
Status Active

Package contents

Files included with this skill beyond the listing page.

  • skill md SKILL.md 5,295 B
  • docs SUMMARY.md 3,678 B

History

  1. First seen on skills.sh
  2. First recorded snapshot · 377 installs

SKILL.md

<!-- generated by src/lib/agentSkills/generator.ts; manual edits will be overwritten -->

Overview

Create and run evaluation suites, watch live benchmark progress, view scorecards, compare model performance, and integrate eval runs with CI workflows from the CLI.

Quick install

npm install -g omniroute   # or: npx omniroute
omniroute --version

Subcommands

eval

Example:

omniroute eval

eval suites

Example:

omniroute eval suites

eval list

Example:

omniroute eval list

eval get <suiteId>

Example:

omniroute eval get <suiteId>

eval create

Flags:

  • --file <path>

Example:

omniroute eval create

eval run <suiteId>

Flags:

  • -m, --model <id>
  • --combo <name>
  • --concurrency <n>
  • --tag <tag>
  • --watch

Example:

omniroute eval run <suiteId>

eval list

Flags:

  • --suite <id>
  • --status <s>
  • --since <ts>
  • --limit <n>

Example:

omniroute eval list

eval get <runId>

Example:

omniroute eval get <runId>

eval results <runId>

Flags:

  • --failed

Example:

omniroute eval results <runId>

eval cancel <runId>

Flags:

  • --yes

Example:

omniroute eval cancel <runId>

eval scorecard <runId>

Example:

omniroute eval scorecard <runId>

simulate [prompt]

Flags:

  • --file <path>
  • -m, --model <id>
  • --combo <name>
  • --reasoning-effort <level>
  • --thinking-budget <n>
  • --explain

Example:

omniroute simulate [prompt]

<!-- skill:custom-start --> <!-- Migrated from skills/omniroute-cli-eval/SKILL.md (preserved curated content) -->

OmniRoute — CLI Evals

Requires the omniroute CLI. See CLI entry-point skill for install + global flags.

What are evals?

Evals are automated test suites that score LLM outputs against expected answers or rubrics. OmniRoute stores suites and run results in its local database.

Eval suites

omniroute eval suites list                       # List all eval suites
omniroute eval suites list --json                # JSON output

omniroute eval suites get <suiteId>              # Full suite definition

Create a suite

omniroute eval suites create \
  --name "code-quality" \
  --rubric "exact-match" \
  --samples-file ./samples.jsonl                 # JSONL: {input, expected_output}

Rubric options: exact-match, contains, llm-judge, regex.

--samples-file format (one JSON object per line):

{"input": "What is 2+2?", "expected_output": "4"}
{"input": "Translate 'hello' to Spanish", "expected_output": "hola"}

Run an eval

omniroute eval suites run <suiteId> \
  --model claude-sonnet-4-6                      # Run suite against a specific model

omniroute eval suites run <suiteId> \
  --model gpt-4o \
  --watch                                        # Live TUI progress (EvalWatch)

The run is asynchronous. Use --watch for a live terminal dashboard or poll manually:

RUN_ID=$(omniroute eval suites run <suiteId> --model claude-sonnet-4-6 --output json | jq -r '.id')
omniroute eval get $RUN_ID

Manage runs

omniroute eval list                              # List all eval runs
omniroute eval list --json

omniroute eval get <runId>                       # Run details (status, model, score)
omniroute eval results <runId>                   # Per-sample results
omniroute eval scorecard <runId>                 # Full scorecard with pass/fail per sample
omniroute eval cancel <runId>                    # Cancel a running eval

Scorecard output

omniroute eval scorecard <runId> --output json

Response fields per sample:

{
  "id": "sample-1",
  "score": 0.95,
  "passed": true,
  "input": "What is 2+2?",
  "output": "4",
  "expected": "4"
}

Comparing models

Run the same suite against multiple models and compare:

for MODEL in claude-sonnet-4-6 gpt-4o gemini-2.0-flash; do
  omniroute eval suites run $SUITE_ID --model $MODEL --output json | jq '{model: .model, score: .score}'
done

CI integration

# Run and fail CI if score drops below threshold
SCORE=$(omniroute eval suites run $SUITE_ID --model claude-sonnet-4-6 --output json | jq -r '.score')
python3 -c "import sys; score=float('$SCORE'); sys.exit(0 if score >= 0.90 else 1)"

Errors

  • suites create fails with invalid rubric → use one of: exact-match, contains, llm-judge, regex
  • suites run returns model not found → verify model ID with omniroute models --search <name>
  • eval get shows status: failed → check omniroute logs --search eval for error details
  • scorecard returns empty results → the run may still be running; poll omniroute eval get <runId> until status is completed

<!-- skill:custom-end -->