Feedback-Driven Skill Refinement

Collects PostToolUse feedback, attributes outcomes to skills semantically, and surfaces actionable insights for improving skills.

Quick Start

# Set up feedback collection (one time)
voyager feedback setup

# Use Claude Code normally - feedback is collected automatically

# View insights
voyager feedback insights

# View insights for a specific skill
voyager feedback insights --skill session-brain --errors

CLIs

feedback-setup / voyager feedback setup

Initialize feedback collection by:

1. Creating the feedback database at .claude/voyager/feedback.db
2. Installing a PostToolUse hook at .claude/hooks/posttooluse_feedback.py
3. Updating .claude/settings.local.json with hook configuration

Options:

• --dry-run / -n: Show what would be done without making changes
• --reset: Delete existing feedback data and start fresh
• --db PATH: Use a custom database path

skill-insights / voyager feedback insights

Analyze collected feedback and generate improvement recommendations.

Options:

• --skill SKILL / -s SKILL: Filter insights for a specific skill
• --errors / -e: Show common errors
• --json: Output results as JSON
• --db PATH: Use a custom database path

How Skill Attribution Works

The system uses a cascade of strategies to attribute tool executions to skills without hardcoded mappings:

1. Transcript Context (most accurate)

- Checks if Claude read a SKILL.md file in this session - If yes, attributes subsequent tool uses to that skill

1. Learned Associations (fast)

- Looks up similar tool+context patterns from past sessions - Improves over time as more feedback is collected

1. ColBERT Index Query (semantic, if available)

- Queries the skill retrieval index with tool context - Works when find-skill command is available

1. LLM Inference (comprehensive, disabled by default in hooks)

- Asks an LLM to identify the skill from context - Slowest but most comprehensive fallback

Storage

• Feedback Database: .claude/voyager/feedback.db (SQLite)
• Hook Script: .claude/hooks/posttooluse_feedback.py

Database Schema

tool_executions: Per-tool execution logs

• sessionid, toolname, toolinput, toolresponse
• success, errormessage, durationms
• skill_used (attributed skill)
• timestamp

session_summaries: Per-session aggregates

• toolsused, skillsdetected
• total/successful/failed calls
• taskcompleted, completionfeedback

learned_associations: Tool context → skill mappings

• context_key (tool|extension|command)
• skillid, confidence, hitcount

Insights Output

The insights command shows:

1. Summary: Total executions, sessions, skills detected
2. Skill Performance: Success rate and error counts per skill
3. Tool Usage: Which tools are used most, failure rates
4. Common Errors: Recurring error patterns
5. Recommendations: Actionable suggestions like:

- "Low success rate - update SKILL.md with better guidance" - "Recurring error (5x): file not found..." - "Low usage - add more trigger phrases"

Workflow for Improving Skills

1. Run voyager feedback insights --errors to see problem areas
2. Check specific skill with voyager feedback insights --skill NAME
3. Review the recommendations
4. Update SKILL.md or reference.md based on observed failures
5. Re-run insights periodically to track improvement

See Also

• reference.md - Technical reference for implementation details
• skills/skill-retrieval/ - Skill indexing for semantic attribution
• skills/skill-factory/ - Creating new skills from observed patterns