evals-clarify

evals-clarify

What this skill does

Conducts axial coding following EDD Principles III & IX to cluster related failure patterns, refine evaluation criteria, generate adversarial examples, and accept validated drafts into the published goldset.

Output:

1. Clustered Criteria - Related patterns grouped into coherent evaluation themes
2. Adversarial Examples - Generated attack scenarios and edge cases for robustness
3. Published Goldset - Accepted criteria in evals/{system}/goldset.md with full documentation
4. Holdout Dataset - Reserved test set (20%) for unbiased evaluation validation
5. JSON Configuration - Auto-generated goldset.json for system consumption
6. Auto-handoff to /evals-implement for grader generation

Key EDD Principles Applied:

• Principle III: Error Analysis & Pattern Discovery - Axial coding → theoretical relationships
• Principle IX: Test Data as Code - Adversarial generation, holdout splits, version control
• Principle II: Binary Pass/Fail - Maintain strict binary evaluation throughout
• Principle I: Spec-Driven Contracts - Criteria validate spec compliance

When to use

• After /evals-specify: Refine and accept draft criteria into goldset
• Dataset maintenance: Balance pass/fail examples or add adversarial cases
• Adding holdout split: Isolate validation data from training data

When NOT to use

• No draft criteria exist: Run /evals-specify to discover patterns first
• Grader generation: Use /evals-implement to convert accepted goldset into code

Process

User Input

$ARGUMENTS

• --accept IDS — Accept specific draft IDs (e.g., "EVAL-001,EVAL-003")
• --merge IDS — Merge related criteria (e.g., "EVAL-001+EVAL-002")
• --split ID — Split complex criterion into multiple focused criteria
• --holdout-ratio RATIO — Holdout percentage (default: 0.2, range: 0.1-0.3)

Execution Steps

Step 0: Detect Lightweight Draft Format

Check if the draft being reviewed uses the lightweight draft template (indicated by presence of type, evidence, source, revisit-when fields in frontmatter and ## Rejected Alternatives / ## Reason body sections without the full formal template sections).

If lightweight:

1. Read the draft's captured fields (Context, Decision, Rejected Alternatives, Reason)
2. Transform to full formal template:
   • ADR: full MADR format with Decision Drivers, Considered Options, Pros/Cons, Constitution Alignment, Related ADRs
   • PDR: full PDR format with Market Forces, Consequences, Alternatives Considered, Links
   • ChDR: full ChDR format with Issue Links, Commits, Consequences, Evidence
   • CDR: full CDR format with Context Type, Target Module, Descriptor, Evidence
   • EVAL: full eval format with Error Analysis, Pass/Fail Examples, Implementation Notes
3. Enrich from session context (add details the lightweight draft may have omitted)
4. Present the enriched draft for review

If already full format, proceed with normal review.

Phase 1: Axial Coding & Clustering

• Group related draft patterns into coherent themes.
• Resolve any overlaps or duplicate criteria.

Phase 2: Refinement & Adversarial Generation

• Generate 3-5 adversarial (attack) examples per criterion to test robustness.
• Balance pass/fail examples (~50/50 ratio).

Phase 3: Holdout Isolation

• Isolate exactly 20% of examples as a reserved holdout set (saved to .adlc/memory/evals/holdout.json — machine data stays under .adlc/ per the shared layout).
• Ensure holdout set is never used in implementation or training.

Phase 4: Publish Goldset

• Copy accepted drafts to docs/adlc/memory/evals/ (the shared layout memory root; holdout.json alone stays at .adlc/memory/evals/holdout.json) and update status to accepted.
• Compile published goldset to evals/{system}/goldset.md (human-readable) and evals/{system}/goldset.json (machine-readable).

Phase 5: Auto-Handoff

Trigger /evals-implement to generate code.

Verification

• Accepted drafts stored in docs/adlc/memory/evals/EVAL-*.md
• evals/{system}/goldset.md and goldset.json exist
• Holdout set .adlc/memory/evals/holdout.json isolated and populated
• All criteria are strictly binary (no confidence scores or Likert scales)
• Handover summary lists accepted criteria and adversarial counts