Harbor CLI Reference

Harbor CLI manages the full evaluation lifecycle: creating tasks, running agents, viewing results. Install with uv tool install harbor. Global option: --version / -v.

For complete flag tables with types and defaults for every command, read references/flags.md.

Quick Reference

harbor run / harbor jobs start

The primary command. harbor run is an alias for harbor jobs start.

Defaults: agent = oracle, n-concurrent = 1, environment = docker, output = ./jobs.

Essential flags:

Additional flags for environment kwargs (--ek), environment env vars (--ee), verifier kwargs (--vk), orchestrator kwargs (--ok), agent images, retry config, trace export, and per-phase timeout multipliers are in references/flags.md.

Examples

# Run local task with oracle (validates solution + tests)
harbor run -p ./my-task

# Run with a real agent
harbor run -p ./my-task -a claude-code -m anthropic/claude-sonnet-4-1

# Dataset evaluation with concurrency
harbor run -d [email protected] -a claude-code -m anthropic/claude-sonnet-4-1 -n 8

# Cloud environment
harbor run -d my-dataset -a claude-code -m anthropic/claude-sonnet-4-1 -e daytona

# From config file
harbor run -c eval-config.yaml

# Pass API key to agent
harbor run -p ./my-task -a claude-code -m anthropic/claude-sonnet-4-1 \
  --ae ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY

# Run subset of tasks
harbor run -d [email protected] -a claude-code -m anthropic/claude-sonnet-4-1 \
  -t "bash-*" -l 10

# Dry run
harbor run -d [email protected] -a claude-code -m anthropic/claude-sonnet-4-1 --dry-run

harbor jobs resume

Resume an interrupted or partially completed job.

harbor jobs resume -p ./jobs/my-job-2025-01-15
harbor jobs resume -p ./jobs/my-job -f AgentTimeoutError

harbor jobs summarize

Generate AI-powered failure summaries for trials in a job.

harbor jobs summarize ./jobs/my-job
harbor jobs summarize ./jobs/my-job -m sonnet --all --overwrite

harbor trials start

Run a single trial. Useful for debugging and task development.

Key difference from harbor jobs start: the environment flag is --environment-type (not --env), and output goes to --trials-dir (default: ./trials).

Full flag list (task kwargs, git options, etc.) in references/flags.md.

# Test with oracle (validates solution + tests)
harbor trials start -p ./my-task

# Test with real agent
harbor trials start -p ./my-task -a claude-code -m anthropic/claude-sonnet-4-1

# Keep container for inspection
harbor trials start -p ./my-task -a claude-code -m anthropic/claude-sonnet-4-1 --no-cleanup

The oracle agent runs solution/solve.sh inside the environment, then the verifier runs tests/test.sh. If oracle does not get reward 1.0, your tests or solution have a bug.

harbor trials summarize

harbor trials summarize ./trials/my-trial
harbor trials summarize ./trials/my-trial -m sonnet --overwrite

harbor datasets

harbor datasets list

harbor datasets list
harbor datasets list --registry-url https://custom-registry.example.com
harbor datasets list --registry-path ./local-registry.json

Flags --registry-url and --registry-path are mutually exclusive. Default: Harbor's public registry.

harbor datasets download

harbor datasets download [email protected]
harbor datasets download [email protected] -o ./my-tasks --overwrite

harbor adapters

harbor adapters init

harbor adapters init my-benchmark

Interactive wizard that prompts for benchmark name, adapter ID, class name, description, source URL, and license. Creates the adapter directory with template files.

harbor adapters review

harbor adapters review -p adapters/my-benchmark
harbor adapters review -p adapters/my-benchmark --skip-ai -o report.md

harbor tasks

harbor tasks init

harbor tasks init my-task
harbor tasks init my-task -p ./custom-tasks

harbor tasks check

AI-powered quality assessment against a rubric.

harbor tasks check ./my-task
harbor tasks check ./my-task -m opus -o results.json
harbor tasks check ./my-task -r custom-rubric.toml

harbor tasks start-env

Launch task environment interactively for manual inspection.

harbor tasks start-env -p ./my-task
harbor tasks start-env -p ./my-task -e daytona
harbor tasks start-env -p ./my-task --agent claude-code -m anthropic/claude-sonnet-4-1

Full flag list in references/flags.md.

harbor tasks debug

Analyze failing trials for a task using AI.

harbor tasks debug my-task-id -m sonnet
harbor tasks debug my-task-id --job-id my-job --jobs-dir ./jobs

harbor tasks migrate

Convert Terminal-Bench tasks to Harbor format.

harbor tasks migrate -i ./old-tasks -o ./harbor-tasks
harbor tasks migrate -i ./old-tasks -o ./harbor-tasks --cpus 2 --memory-mb 4096

harbor view

Browse job/trial results in a web UI.

harbor view ./jobs/my-job
harbor view ./jobs --port 9000
harbor view ./jobs --dev

Shows job summaries, per-trial reward values, agent trajectories, and test output.

harbor sweeps run

Run successive sweeps. Each sweep drops tasks that already have at least one success, focusing effort on remaining failures.

harbor sweeps run -c sweep-config.yaml
harbor sweeps run -c sweep-config.yaml --max-sweeps 5 --trials-per-task 3
harbor sweeps run -c sweep-config.yaml --hint "Try grep to find the file first"

Additional export flags in references/flags.md.

harbor traces export

Export ATIF trajectories as training datasets.

harbor traces export -p ./jobs/my-job
harbor traces export -p ./trials --filter success --push --repo my-org/traces
harbor traces export -p ./jobs/my-job --sharegpt --episodes last

Additional flags (--subagents, --instruction-metadata, --verifier-metadata, --verbose) in references/flags.md.

harbor cache clean

Remove Harbor Docker images and ~/.cache/harbor.

harbor cache clean
harbor cache clean --force
harbor cache clean --dry
harbor cache clean --no-docker      # Only clean ~/.cache/harbor
harbor cache clean --no-cache-dir   # Only clean Docker images

harbor admin upload-images

Build and push task Docker images to a container registry. Updates task.toml with docker_image for pre-built image workflows. Hidden from harbor --help but fully functional.

harbor admin upload-images -r my-registry.io --tag 20260317
harbor admin upload-images -r my-registry.io --dry-run -f "bash-*"

Key flags: --tasks-dir / -t (default: tasks), --registry / -r (required), --tag (default: today YYYYMMDD), --dry-run, --filter / -f, --push/--no-push (default: push), --delete/--no-delete, --parallel / -n (default: 1), --update-config/--no-update-config, --override-config/--no-override-config. Full details in references/flags.md.

Supported Agents

Environment Backends

Note: harbor run uses --env / -e. harbor trials start uses --environment-type / -e.

Common Workflows

Create and test a new task

harbor tasks init my-task
# Edit: task.toml, instruction.md, Dockerfile, tests/test.sh, solution/solve.sh
harbor tasks check ./my-task
harbor trials start -p ./my-task                    # oracle validates tests
harbor trials start -p ./my-task -a claude-code -m anthropic/claude-sonnet-4-1
harbor view ./trials

Run a benchmark evaluation

harbor datasets list
harbor datasets download [email protected]
harbor run -d [email protected] -a claude-code -m anthropic/claude-sonnet-4-1 -n 8
harbor view ./jobs
harbor traces export -p ./jobs/my-job --push --repo my-org/traces

Debug a failing task

harbor tasks start-env -p ./my-task         # interactive shell in the container
# Inside: bash /tests/test.sh && bash /solution/solve.sh
harbor tasks debug my-task-id -m sonnet     # AI analysis of failures
harbor trials start -p ./my-task            # re-test with oracle

Resume a failed job

harbor jobs resume -p ./jobs/my-job-2025-01-15
harbor jobs resume -p ./jobs/my-job -f AgentTimeoutError
harbor jobs summarize ./jobs/my-job

Common Gotchas

API keys: Most agents need keys passed via --ae. Claude Code needs ANTHROPICAPIKEY, OpenHands needs LLMAPIKEY, Goose needs provider-specific keys. If the agent fails immediately, check the key.

Docker must be running. Harbor uses Docker for sandboxed environments. Network exhaustion from many concurrent trials can cause failures -- harbor cache clean helps.

Model name format varies by agent. Claude Code uses anthropic/claude-sonnet-4-1, Cline CLI uses provider:model-id (e.g., openrouter:anthropic/claude-opus-4.5), Goose uses provider/model_name. A ValueError about model format usually means the wrong format for that agent.

--env vs --environment-type: harbor run uses --env / -e. harbor trials start uses --environment-type / -e. Same short flag, different long name.

Default agent is oracle. Forgetting -a gives you the oracle agent, which just runs solve.sh. Fine for task validation, not for real evaluations.

--no-cleanup for debugging. Pass this to harbor trials start to keep the container running after a failed trial so you can inspect it.

Retry defaults skip common errors. --retry-exclude defaults to AgentTimeoutError, VerifierTimeoutError, RewardFileNotFoundError, RewardFileEmptyError, VerifierOutputParseError. These usually indicate task bugs, not transient failures.