Distill Session — Extract What You Can't Yet Explain

The most valuable knowledge in a top-tier model's output is the knowledge
it applied without mentioning. This skill makes that invisible reasoning
visible, testable, and transferable.

The Core Insight

When a weaker model fails, the gap is almost never "it didn't try hard enough." The gap is that the top-tier model executed steps it doesn't consciously report — constraint checks it ran silently, orderings it chose without deliberation, failure modes it avoided by reflex. These implicit behaviors are the extraction target.

The hard part is not getting the top-tier model to succeed. It's getting it to notice what it did that was non-obvious.

When to Use

• A weaker model produced wrong or shallow output on a task you handle well
• You completed a complex task and want to mine it for transferable rules
• A skill (coding-workflow, debug-workflow, task-router) needs strengthening
• You notice yourself doing something you can't fully articulate yet

Distill immediately after completing a complex task — don't wait for a failure to trigger this. Your reasoning is still warm.

Phase 1 — CAPTURE

Goal: Preserve exact failure and success before analysis distorts them.

Save to .omc/distill-sessions/YYYY-MM-DD-[short-title].md:

# Distill: [short title]
Date: YYYY-MM-DD | Task type: code-change / bug-fix / refactor / review

## The Task
[Exact prompt — verbatim]

## Weak Model Output
[Exact output — do NOT paraphrase. Preserve reasoning chain if visible.]

## What Went Wrong (Observable)
[Concrete defects: wrong file, missing call site, incorrect assumption.]

## Top-Tier Model Output
[Your actual output.]

Critical: Paste the weak model's FULL output. Phase 3 requires comparing actual text, not your memory of it.

Phase 2 — THE EXTRACTION (Core of This Skill)

Goal: Surface the implicit reasoning that made the difference.

This is where most distillation fails. Generic "what steps did you take?" gets cleaned-up narratives, not actual reasoning. The questions below bypass post-hoc rationalization to reach real decision points.

The Extraction Prompt

Answer with ruthless honesty. Do not clean up your reasoning or make it sound more systematic than it was.

<extraction>

## 1. What did I look at before I started doing anything?
[Specific files, symbols, docs you read BEFORE any change. Include
things you glanced at and decided weren't relevant. Weaker models
start acting before this step is complete.]

## 2. What did I almost do wrong, then correct?
[Any moment you began one approach, noticed a problem, pivoted.
These near-misses reveal constraints you check implicitly. A weaker
model would have followed through on the wrong approach.]

## 3. What information was I holding in my head simultaneously?
[Facts, constraints, relationships tracked at the same time. Example:
"function X called from 3 places, return type used as discriminated
union downstream, one call site in a hot path." Weaker models handle
these serially or drop some.]

## 4. Where did I choose to NOT do something?
[Restraint is invisible. Skipped refactors? Edge cases not worth
handling? Suppressed "while I'm here" changes? Weaker models often
fail by doing too much, not too little.]

## 5. What ordering did I choose and why?
[Dependency reason? Safety reason? Would a different order cause
problems? Weaker models edit in prompt-order rather than
dependency-order.]

## 6. What would break if I removed any single step?
[Steps where removal causes failure are essential. Steps where
removal doesn't matter were optional.]

## 7. What is the simplest incorrect version of my output?
[The most plausible wrong answer — ~80% understanding of the task.
Usually what the weak model produced. Name the specific 20% gap.]

## 8. Write the instruction set (3-7 rules)
[Each rule must be:
- Actionable in a single step (no "think carefully about X")
- Verifiable by a third party (someone can check if it was done)
- Triggered by an observable condition (not "when appropriate")]

</extraction>

Why These Questions Work

Phase 3 — DELTA ANALYSIS (Most Error-Prone Phase)

Goal: Identify which behaviors CAUSED the quality difference.

The common mistake: attributing success to the most impressive-looking step rather than the step that actually mattered.

The Attribution Test

For each behavior from Phase 2:

1. If the weak model did THIS ONE THING, would it have succeeded?

Yes → candidate root cause. No → contributing factor at best.

1. Did the weak model lack ABILITY or PROMPTING?

Ability gap → needs a tool, not a rule. Prompt gap → extractable.

1. Specific to this task, or generalizable?

Specific → useful as example. General → candidate pattern.

Common Mistakes

Output

## Delta Analysis
Root cause: the weak model failed because it ___
Minimum fix: [single most impactful rule]
Contributing factors: 1. [behavior] — [impact] 2. [behavior] — [impact]
Non-essential steps I took: [prevents rule bloat]

Phase 4 — DISTILL INTO RULES

Every rule must pass ALL FIVE criteria:

The causality check: "If the weak model followed ONLY this rule and changed nothing else, would the output improve?" Yes → keep. No → delete. Maybe → make more specific.

Save to .omc/distill-sessions/patterns/[task-type].md:

### Rule [N]: [short name]
**Trigger**: [Observable condition]
**Action**: [Single concrete instruction]
**Because**: [What goes wrong without it]
**Test**: [How to verify it was followed]

Phase 5 — INTEGRATE

Propagate rules into the skills that govern the behavior:

Bump target skill version (patch: 1.0.0 → 1.0.1).

Phase 6 — VALIDATE

The real question: Does this rule produce CORRECT output, or just BETTER-LOOKING output?

"Explain your reasoning before acting" makes outputs look thoughtful without improving correctness. "Grep for all call sites before modifying a function signature" produces measurably different output. Target the latter.

1. Re-run EXACT original task with weaker model + updated skill
2. Compare on the specific dimension that failed, not overall quality

## Validation
Original failure: [what was wrong]
After distillation: [is that specific thing correct now?]
VERDICT: FIXED / BETTER BUT STILL WRONG / NO CHANGE / REGRESSION

If not FIXED — was Phase 3 wrong? Re-examine root cause.

BETTER BUT STILL WRONG → rule addresses symptom, not cause. Redo Phase 3. NO CHANGE → rule not triggered, or action beyond model capability.

Quick Reference: Copy-Paste Extraction Prompt

The single most important artifact. Use immediately after completing a task.

You just completed a task. Before moving on, fill in this extraction
honestly — especially the parts that feel embarrassing or obvious.

<extraction>
## What I looked at before acting
## What I almost did wrong, then corrected
## What I was holding in my head simultaneously
## What I chose NOT to do
## What ordering I chose and why it mattered
## What would break if I skipped any single step
## The simplest plausible wrong answer to this task
## Rules for a weaker model (3-7, each one sentence, each verifiable)
</extraction>