SKILL.md
Prompt Engineer — Agent Prompts & Evals
When to Use
- Write or refactor system/developer prompts for agents with tools
- Design tool descriptions and parameter schemas the model must choose correctly
- Build golden datasets and scenario suites (multi-turn, failures, edge cases)
- Implement offline eval harnesses and CI regression for prompts
- Define LLM-as-judge rubrics and human calibration samples
- Set release gates when prompts or tool lists change
- Debug wrong tool, looping, or format break in agent traces
When NOT to Use
- General prompt patterns without agent/eval focus →
prompt-engineer - End-to-end RAG indexing, retrieval, serving →
ai-engineer - Jailbreak and abuse red-team engagements →
ai-redteam - Org-wide model rollout and incident ops →
ai-lead-ops - Token cost program across product →
ai-token-improvement-plan-engineer - Vertical squad management and launch PM →
engineering-manager-vertical-ai-products - Prompt/eval team management and governance →
engineering-manager-agent-prompts-evals
Related skills
| Need | Skill |
|---|---|
| Broad prompt design and production guardrails | prompt-engineer |
| RAG, agents in production code | ai-engineer |
| Launch eval gates (manager view) | engineering-manager-vertical-ai-products |
| Risk tier and policy | ai-risk-governance |
| Adversarial testing | ai-redteam |
Core Workflows
1. Agent prompt structure
System vs developer messages, tools block, constraints, handoffs.
See references/agentsystemprompts.md.
2. Eval datasets
Golden sets, coverage matrix, synthetic and SME-labeled examples.
See references/evaldatasetdesign.md.
3. Harness and metrics
Offline runs, pass/fail, tool accuracy, trajectory checks.
See references/evalharnesspatterns.md.
4. Judges and rubrics
LLM judge design, bias controls, human agreement.
See references/llmjudgerubrics.md.
5. Versioning and regression
Prompt semver, baselines, CI gates, rollback.
See references/promptversioningregression.md.
6. Scenario catalog
Multi-turn, tool error, refusal, escalation cases.
See references/agentevalscenarios.md.
Output standards
- Every eval case has input, expected behavior (not always exact text), tags
- Tool schema changes include eval delta in PR description
- Judge rubrics published with scoring scale and known failure modes
- No production prompt change without baseline comparison on golden set
When to load references
- Prompts →
references/agentsystemprompts.md - Data →
references/evaldatasetdesign.md - Harness →
references/evalharnesspatterns.md - Judges →
references/llmjudgerubrics.md - CI →
references/promptversioningregression.md - Scenarios →
references/agentevalscenarios.md