HUD Skill Reference

Product Summary

HUD is a platform for building RL environments for AI agents. An environment wraps your code as tools agents can call and defines scenarios that evaluate what agents do. Environments spin up fresh and isolated for every evaluation—no shared state, fully reproducible. Use HUD to evaluate model performance, build frontier-grade post-training data, and train specialized agents on your tasks.

Key files and commands:

• env.py — Define tools and scenarios
• tasks.py or tasks/ — Define evaluation tasks
• Dockerfile.hud — Container definition (auto-generated)
• hud init — Scaffold new environment
• hud deploy — Build and deploy to platform
• hud eval — Run agents locally or remotely
• hud sync tasks — Push tasks to platform
• Gateway: inference.hud.ai — One endpoint for all models (Claude, GPT, Gemini, Grok, etc.)

Primary docs: https://docs.hud.ai

When to Use

Reach for HUD when:

• Building evaluation environments — You need isolated, reproducible sandboxes where agents can take actions and receive reward signals
• Comparing model performance — You want to run the same tasks across multiple models (Claude, GPT, Gemini) and see which performs best
• Creating benchmarks — You're building a taskset to evaluate agent capabilities (coding, computer use, tool use, research)
• Generating training data — You need traces from successful agent runs to fine-tune your own models
• Running at scale — You need to run hundreds of parallel evaluations without local compute
• Iterating on scenarios — You're refining prompts, grading logic, or task difficulty and need fast feedback loops
• Deploying agent infrastructure — You need a production-ready system for running agents with full tracing and monitoring

Do not use HUD for: simple LLM API calls without evaluation, chat applications without structured tasks, or one-off agent experiments (use the SDK directly instead).

Quick Reference

Environment Structure

from hud import Environment

env = Environment("my-env")

# Define a tool
@env.tool()
def my_tool(arg: str) -> str:
    """Tool description for the agent."""
    return f"Result: {arg}"

# Define a scenario (evaluation)
@env.scenario("my-scenario")
async def my_scenario(param: str):
    # First yield: send prompt to agent, get answer back
    answer = yield f"Do something with {param}"
    
    # Second yield: score the result (0.0 to 1.0)
    yield 1.0 if answer else 0.0

Task Definition

from env import my_scenario

# Create a task (scenario + specific arguments)
task = my_scenario.task(param="value")
task.slug = "unique-task-id"
task.columns = {"category": "test", "difficulty": "easy"}

CLI Commands

Running Evaluations

# Local: single task
hud eval tasks.py claude

# Local: all tasks
hud eval tasks.py claude --full

# Local: specific tasks
hud eval tasks.py claude --task-ids task1,task2

# Local: with variance (run each task 3 times)
hud eval tasks.py claude --full --group-size 3

# Remote: on HUD infrastructure
hud eval "My Taskset" claude --full --remote

# With model override
hud eval tasks.py openai --model gpt-4o

Native Tools (Pre-built)

Decision Guidance

When to Use X vs Y

Workflow

1. Create & Test Locally

# Initialize
hud init my-env && cd my-env

# Edit env.py: add tools and scenarios
# Edit tasks.py: define evaluation tasks

# Test a single task
hud eval tasks.py claude

# Test all tasks
hud eval tasks.py claude --full

2. Iterate on Scenarios

# Run with hot-reload (edit env.py, saves auto-reload)
hud dev env:env -w env.py

# In another terminal, test interactively
hud eval tasks.py claude --task-ids my-task

3. Deploy to Platform

# Deploy (builds remotely, takes 2-5 min first time)
hud deploy

# Verify deployment
hud analyze my-env

4. Sync Tasks

# Push tasks to platform taskset
hud sync tasks my-taskset

# Re-sync after changes
hud sync tasks

5. Run Remote Evaluations

# Run on HUD infrastructure
hud eval "my-taskset" claude --full --remote

# Monitor at hud.ai/jobs
# View results on taskset leaderboard

6. Analyze & Iterate

# Check traces on platform (hud.ai/evalsets)
# Identify failing tasks
# Update scenario/grading logic
# hud deploy (if env code changed)
# hud eval --remote (re-run)

7. Train Models (Optional)

# On platform: hud.ai/models
# Fork a base model
# Click "Train Model"
# Select your taskset as training data
# Use trained model: hud eval tasks.py your-model-id --full

Common Gotchas

• Forgot to deploy before syncing tasks — You must hud deploy first. The platform needs a container image to run evaluations. Sync tasks after deployment.

• Task slug changed — Renaming a slug creates a new task on the platform (old one remains). Choose slugs carefully and don't rename them.

• Local eval passes, remote fails — Check the trace logs (LOGS and DEBUG tabs). Common causes: missing dependencies in Dockerfile, grading logic diverged, environment state not isolated. Verify your latest build matches .hud/deploy.json.

• Scenario doesn't yield twice — Every scenario must have exactly two yields: first sends prompt, second returns reward (0.0–1.0). Missing the second yield causes silent failures.

• Tools not showing up in agent — Check tool docstrings (used as descriptions) and type hints (used as parameter schema). Both are required. Use hud analyze to inspect what the agent sees.

• Docker build fails — Use hud debug my-env:latest to see which phase failed. Check Dockerfile syntax, missing dependencies, and port conflicts.

• Grading logic uses external state — Scenarios must be deterministic and isolated. Don't rely on shared databases or files. Each evaluation spins up fresh.

• Agent gets stuck in loop — Set --max-steps to prevent infinite loops. Default is 10 (single task) or 100 (--full).

• Columns not syncing — Column types are inferred from values across all tasks. If a column has mixed types, sync may fail. Keep column values consistent.

• Remote eval shows 0.0 reward — Check: (1) Did agent attempt the task? (2) Are logs showing errors? (3) Does it pass locally? If local passes but remote fails, the deployed environment diverged.

Verification Checklist

Before submitting work:

• Scenario structure — Two yields (prompt → reward), reward is 0.0–1.0
• Tools documented — Docstrings and type hints present
• Tasks defined — Each task has a unique slug
• Local test passes — hud eval tasks.py claude --full succeeds
• Dockerfile valid — hud build . completes without errors
• Deployed — hud deploy succeeded and environment appears on platform
• Tasks synced — hud sync tasks shows expected tasks
• Remote eval runs — hud eval "taskset" claude --full --remote completes
• Traces inspectable — Click a task on leaderboard, trace viewer opens
• Grading deterministic — Same agent answer always gets same reward

Resources

Comprehensive navigation: https://docs.hud.ai/llms.txt

Critical pages:

• [Scaffolding](/building/scaffolding) — Create environments, define tools and scenarios
• [Tasks & Evaluation](/building/tasks-and-evaluation) — Define tasks, test locally, iterate
• [Deploy & Go Remote](/building/running-at-scale) — Deploy, sync, run at scale

For additional documentation and navigation, see: https://docs.hud.ai/llms.txt