mcpField
E
eval-triggers

eval-triggers

github.com/datocmsTesting & quality3118 installsUpdated 6d ago
Repository

About eval-triggers

A skill for evaluating when skills are triggered by user requests, using separate Claude Code and Codex tracks. It is intended for users explicitly running trigger evaluations and reports per-skill precision, recall, F1, gate status, and relevant warnings.

Key features

  1. 01Claude Code、Codex 或双轨评估
  2. 02支持 frontmatter、metadata 和 combined 来源
  3. 03生成跨技能评估汇总
  4. 04报告精确率、召回率与 F1
  5. 05可选比较基线与候选结果

Use cases

  1. 01评估技能触发描述是否准确
  2. 02检查技能的精确率、召回率和 F1
  3. 03汇总技能评估门槛与缺失结果
  4. 04对比当前结果与基线摘要

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/datocms/agent-skills --skill eval-triggers

After installing, type/eval-triggersin Claude Code to confirm it loaded.

Details

From the project README

IMPORTANT: This skill is expensive (makes many LLM API calls). Only run when the user explicitly asks for it. Never run proactively.

Before running, ask the user which track to run unless they already specified it in $ARGUMENTS:

  • Claude Code only — --track claude
  • Codex only — --track codex
  • Both — run both tracks sequentially

Use --source frontmatter (the default) unless the user asks for metadata or combined. Pass --model and --effort when the user names them (the repo baseline for Codex is --model gpt-6-luna --effort medium); without --model, Codex results record no model, and without --effort a Codex run uses the model's own default effort. Every step below takes the same --track and --source.

Step 1 — Classify:

The runner writes to the canonical layout at evals/results/trigger/<skill>/<track>/<source>/results.json. You do not pass an output directory.

python3 evals/scripts/run_trigger_eval.py --track <track> --source <source> [--model <model>] [--effort <effort>]

Step 2 — Analyze:

python3 evals/scripts/analyze_trigger_results.py --track <track> --source <source>

This writes the cross-skill summary to evals/results/trigger/_summary/<track>/<source>/summary.{json,md}. Report:

  • The gate verdict: it passes only when every skill has results at or above F1 0.90. List the skills below the floor and the skills without results.
  • Per-skill precision, recall, F1.
  • Any warnings (different models, results older than the current description) and noteworthy false negatives / positives.

Step 3 — Compare (optional):

If the user provides a baseline summary or you have one from a previous run:

python3 evals/scripts/compare_trigger_runs.py \
  --baseline <baseline-summary>.json \
  --candidate evals/results/trigger/_summary/<track>/<source>/summary.json \
  --output-markdown local/comparison.md

Summarize regressions and improvements, and say so when the two runs used different or unrecorded models. For ad-hoc baselines that should not be committed, store them under local/ (gitignored).

Comments

Markdown supported