Installation

$ npx skills add ukgovernmentbeis/inspect_evals --skill check-trajectories-workflow

Summary

Use Inspect Scout to analyze agent trajectories from evaluation log files.
Runs default and custom scanners to detect external failures, formatting issues, reward hacking, and ethical refusals.
Use when user asks to check/analyze agent trajectories.
Trigger when the user asks you to run the "Check Agent Trajectories" workflow.

Agent compatibility

Declared targets from SKILL.md / docs. Unmarked agents are not listed — the skill may still install via the CLI.

Claude Code Not declared

Cursor Not declared

Codex Not declared

GitHub Copilot Not declared

Windsurf Not declared

Gemini CLI Not declared

Cline Not declared

OpenCode Not declared

Repository health

Stars 571

License LICENSE

Default branch main

Open issues 128

Status Archived

Package contents

Files included with this skill beyond the listing page.

skill md SKILL.md 4,836 B

docs SUMMARY.md 363 B

History

Jun 24, 2026 First seen on skills.sh

Sep 9, 2026 First recorded snapshot · 3 installs

SKILL.md

Check Agent Trajectories

This workflow uses Inspect Scout to automatically analyze agent trajectories. It's faster than manual analysis but may miss nuanced issues.

Workflow Steps

If the user does not give any indication of what log file to use, ask them for which log file or files they want the workflow performed on. You can show them the most recent log file in logs/ and ask them if that's what they want.

Determine the name of the evaluation the log file is used in, and create a folder titled agentartefacts/trajectoryanalysis/<eval_name>. Add a number to the end if the folder already exists.

Read agentartefacts/trajectoryanalysis/inspect_scout/scanners.py to see what the default scanners check for:

1. outcomesummary: Brief summary of why the agent succeeded or failed. 2. externalfailure: Failed due to CAPTCHAs, rate limiting, network issues, missing dependencies. 3. formattingfailure: Failed due to incorrect answer formatting despite correct answer. 4. rewardhackingsuccess: Succeeded through reward hacking or unintended means. 5. ethicalrefusal: Failed because the agent refused on ethical or safety grounds.

Additionally, if agentartefacts/trajectoryanalysis/<eval_name><version> exists already, check for any scanners contained in the latest version and include those.

Tell the user what will be checked by default and ask if they want to check for anything else.

If the user wants additional checks, create an evalscanners.py file under agentartefacts/trajectoryanalysis/<evalname>/, and add Inspect Scout scanners that check for their requirements. Use the existing scanners in agentartefacts/trajectoryanalysis/inspectscout/scanners.py and the Inspect Scout documentation as references. Copy any scanners found in the eval_name folder at the end of Step 2 across as well.

Each custom scanner should be wrapped in an InspectEvalScanner object and added to a SCANNERS list:

```python from scanners import InspectEvalScanner

SCANNERS = [ InspectEvalScanner( name="myscanner", scannerfactory=myscannerfunction, invalidatessuccess=False, # Set True if flagging invalidates a success invalidatesfailure=True, # Set True if flagging invalidates a failure ), ] ```

Ask the user whether each scanner should invalidate successes, failures, both, or neither.

Check how many samples are in the log file with uv run python agentartefacts/trajectoryanalysis/inspectscout/runallscanners.py <logfile> --dry-run. If more than 100 samples would be analysed, ask the user if they want to run all samples or a subset. Tell them that Inspect Evals guidance is to run at least 100 samples, and ask them if they want to run any more than that.

Provide the user the command they can use to run the scanners. If the user asks you to run it, you can do so.

1. If no custom scanners: uv run python agentartefacts/trajectoryanalysis/inspectscout/runallscanners.py <logfile> -o agentartefacts/trajectoryanalysis/<evalname>/scoutresults with an optional --limit <num> if a subset is run. 2. If custom scanners were created in step 4, add -n <evalname> like so: uv run python agentartefacts/trajectoryanalysis/inspectscout/runallscanners.py <logfile> -n <evalname> -o agentartefacts/trajectoryanalysis/<evalname>/scoutresults

The -n <evalname> option automatically loads scanners from agentartefacts/trajectoryanalysis/<evalname>/eval_scanners.py. Duplicate scanner names are detected and skipped with a warning.

Once the command is done, extract the results: uv run python agentartefacts/trajectoryanalysis/inspectscout/extractresults.py agentartefacts/trajectoryanalysis/<evalname>/scoutresults

Analyze sample validity: uv run python agentartefacts/trajectoryanalysis/inspectscout/analyzevalidity.py agentartefacts/trajectoryanalysis/<evalname>/scoutresults

Review the extracted results and create a summary in <evalname><modelname>ANALYSIS.md. If the file already exists, check to see if the user wants it overwritten or a new file generated by date. If the user wants it generated by date, use <evalname><modelname><date>_ANALYSIS.md.

Tell the user the task is done.

check-trajectories-workflow