skill-distiller

Skill Distiller

Transform skills authored for high-capability models (Opus) into deterministic workflows that execute reliably on lower-cost models (Sonnet, Haiku). The core insight from EvoSkills: skills encode reusable task structure, not model-specific artifacts. A skill evolved on Opus transfers with +35-45pp gains to other models — but only when the instructions are sufficiently deterministic that lower-capability models can follow them without improvising.

Reference Files

File	Contents	Load When
references/distillation-patterns.md	Pattern catalog for converting reasoning to rules	Always

Prerequisites

• The source skill must exist and satisfy the current package-evaluator 2.0.0 contract: actual report-v2 static_conformance.status: PASS AND no unresolved CRITICAL or HIGH authored findings. Keep machine evidence and authored findings separate; live_effectiveness.status remains NOT_MEASURED.
• Access to both the source model (Opus) and target model (Haiku/Sonnet) for validation
• The surrogate-verifier skill for cross-model assertion checking
• If the evaluator or required authored review is unavailable, report the missing prerequisite and stop before trace collection; do not fabricate a score or skip this gate. This prerequisite does not establish cross-model behavioral performance.

Workflow

Phase 1: Complexity Analysis

Score each section of the source SKILL.md for reasoning difficulty:

Complexity Signal	Score	Distillation Action
Decision tree with 3+ branches	HIGH	Convert to explicit if/then lookup table
"Use judgment" or "consider context"	HIGH	Replace with concrete heuristic rules
Multi-step inference chain	HIGH	Break into numbered atomic steps
Reference to domain expertise	MED	Add explicit reference file with knowledge
Clear enumerated steps	LOW	Keep as-is
Concrete examples with expected output	LOW	Keep as-is

Produce a complexity map: section name -> complexity score -> planned action.

Phase 2: Trace Collection

Execute the source skill with Opus on 5 representative tasks:

1. Select tasks from evals/cases.yaml (positive cases) or generate new ones
2. For each task, capture the full execution trace:
   • Tool calls made (which tools, in what order)
   • Intermediate reasoning visible in output
   • Final output structure and content
   • Time taken and token usage
3. Store traces as structured data for pattern extraction

Phase 3: Pattern Extraction

From the collected traces, extract deterministic patterns:

1. Decision paths — For each HIGH-complexity section, find the actual decisions Opus made across the 5 tasks. If Opus chose the same path in 4/5 cases, that path becomes the default rule
2. Lookup tables — Where Opus applied domain knowledge, build explicit lookup tables (e.g., "if input contains SQL, use these patterns; if input contains Python, use those")
3. Concrete examples — Extract representative input/output pairs from traces to serve as few-shot examples in the distilled skill
4. Tool sequences — Identify the common tool invocation pattern and make it explicit ("Step 1: Read the file. Step 2: Grep for pattern X. Step 3: Write output.")

Phase 4: Distilled Rewrite

Rewrite the SKILL.md applying all distillation actions from Phase 1:

Source Pattern	Distilled Replacement
"Analyze the code and determine..."	"Check for these 5 specific patterns: [list]"
"Use appropriate formatting"	"Output as a markdown table with columns: [A, B, C]"
"Consider the context to decide..."	"If [condition A]: do X. If [condition B]: do Y. Default: Z"
"Apply best practices for..."	Reference file with explicit best practices enumerated
Multi-paragraph reasoning instruction	Numbered step list with single-sentence steps

Rules for the rewrite:

• Every instruction must be actionable by a model with no domain expertise
• No step should require inference — each step's input and output must be explicit
• Replace all "consider", "analyze", "determine" verbs with "check", "count", "list", "output"
• Add concrete examples for any step that could be ambiguous
• Keep the SKILL.md under 500 lines (distillation should reduce, not expand)

Phase 5: Target Model Validation

Run the distilled skill on the target model (Haiku or Sonnet):

1. Execute the same 5 tasks from Phase 2 with the distilled skill loaded
2. Use the surrogate-verifier to generate assertions for each task output
3. Compare pass rates:

Metric	Source (Opus + original)	Target (Haiku + distilled)	Delta
Assertions passed	N/M	N/M	±
Weighted score	X.XX	X.XX	±
Output completeness	%	%	±
Format compliance	%	%	±

4. If target model score < 80% of source model score, iterate:
   • Identify which assertions the target model fails
   • Add more explicit instructions for those specific failure points
   • Re-run validation (max 3 iterations)

Phase 6: Cross-Model Report

Produce the final comparison:

# Skill Distillation Report: <skill-name>

## Complexity Reduction
- Sections distilled: N/M (HIGH → LOW)
- Instruction word count: original X → distilled Y (Z% reduction)
- Decision points replaced with lookup tables: N

## Cross-Model Performance
| Model   | Assertions Passed | Weighted Score | Format Compliance |
|---------|-------------------|----------------|-------------------|
| Opus    | 7/7               | 1.00           | 100%              |
| Sonnet  | 6/7               | 0.92           | 100%              |
| Haiku   | 5/7               | 0.85           | 85%               |

## Changes Made
1. [Section] "Analyze complexity" → explicit 5-item checklist
2. [Section] "Apply formatting" → fixed markdown table template
...

## Recommendation
[SHIP | ITERATE | MANUAL_REVIEW_NEEDED]

Error Handling

Error	Resolution
Source static conformance FAIL or unresolved CRITICAL/HIGH authored findings	Refuse distillation; report each evidence source separately and recommend source repair via test-engineer
Source evaluator or authored review unavailable	Report the exact missing prerequisite; withhold gate completion and trace collection
No execution traces available	Generate synthetic tasks and collect traces before proceeding
Target model fails all assertions	Skill may be too complex for target model; report with detail
Distilled skill longer than source	Review distillation; patterns may need consolidation

Limitations

• Cannot distill skills that rely on open-ended adaptive reasoning at many decision points or multi-turn reasoning
• Visual/interactive skills (HTML generation, browser automation) may not distill well
• Distillation optimizes for determinism, not creativity — skills requiring open-ended generation (writing, brainstorming) are poor candidates
• Trace collection requires actual model execution, incurring API costs