mcpField
S
skill-eval-scaffolder

skill-eval-scaffolder

github.com/wyattowalshTesting & quality535 installsUpdated 8w ago
Repository

About skill-eval-scaffolder

skill-eval-scaffolder creates minimal, schema-valid evaluation manifests for repository skills. It is for maintainers preparing skill tests, with safeguards against unapproved overwrites and live behavioral evaluation.

Key features

  1. 01Scaffolds evaluation manifests
  2. 02Checks existing manifest validity
  3. 03Supports preview and apply modes
  4. 04Requires explicit-invocation coverage

Use cases

  1. 01Create an initial skill eval manifest
  2. 02Validate a skill's existing eval file
  3. 03Preview a scaffold before writing it

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/wyattowalsh/agents --skill skill-eval-scaffolder

After installing, type/skill-eval-scaffolderin Claude Code to confirm it loaded.

Bundled files

References and scripts installed along with SKILL.md.

evals/
evals.json
scripts/
check.py
scaffold_evals.py
scripts/asset_toolkit/
validate_hooks.py

Details

From the project README

Skill Eval Scaffolder

Create minimal, schema-valid eval manifests for repository skills.

Scope: Eval scaffolding only. Do not run live behavioral evals unless the maintainer explicitly approves.

Dispatch

$ARGUMENTSAction
<name>Preview scaffold for skills/<name>/evals/evals.json
<name> --applyWrite scaffold when file is missing or user approved overwrite
check <name>Validate an existing eval manifest
EmptyShow usage gallery and manifest schema summary

Workflow

  1. Confirm skills/<name>/SKILL.md exists.
  2. Run the bundled scaffold script:
uv run python skills/skill-eval-scaffolder/scripts/scaffold_evals.py <name>
  1. Add explicit, implicit, and negative-control cases before claiming adequate coverage.
  2. Validate with the bundled check script.

Manifest Shape

Each evals/evals.json uses:

  • skill_name — kebab-case directory name
  • evals — non-empty list of cases with id, prompt, expected_output, optional files, assertions

Validation Contract

uv run python skills/skill-eval-scaffolder/scripts/check.py

Critical Rules

  1. Never overwrite an existing eval manifest without explicit user approval.
  2. Include at least one explicit-invocation case in new scaffolds.
  3. Run validate_evals.py before declaring the scaffold complete.
  4. Route live eval execution to skill-creator eval mode or maintainer CI workflows.
  5. Keep prompts grounded in the skill dispatch table and NOT-for boundaries.

Comments

Markdown supported