mcpField
E
eval

eval

github.com/alirezarezvaniAgent workflows28.9k1.4k installsUpdated 2d ago
Repository

About eval

eval is an AgentHub skill that ranks agent results for a session. It can run configured metrics, use an LLM judge to compare diffs, or combine both methods.

Key features

  1. 01Rank results by command-based metrics
  2. 02Compare agent diffs with an LLM judge
  3. 03Combine metric and qualitative evaluation
  4. 04Show rankings with justifications
  5. 05Update session state after evaluation

Use cases

  1. 01Rank agent work by measured performance
  2. 02Compare code changes for correctness and quality
  3. 03Break close metric results with qualitative review

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/alirezarezvani/claude-skills --skill eval

After installing, type/evalin Claude Code to confirm it loaded.

Details

From the project README

/hub:eval — Evaluate Agent Results

Rank all agent results for a session. Supports metric-based evaluation (run a command), LLM judge (compare diffs), or hybrid.

Usage

/hub:eval                           # Eval latest session using configured criteria
/hub:eval 20260317-143022           # Eval specific session
/hub:eval --judge                   # Force LLM judge mode (ignore metric config)

What It Does

Metric Mode (eval command configured)

Run the evaluation command in each agent's worktree:

python {skill_path}/scripts/result_ranker.py \
  --session {session-id} \
  --eval-cmd "{eval_cmd}" \
  --metric {metric} --direction {direction}

Output:

RANK  AGENT       METRIC      DELTA      FILES
1     agent-2     142ms       -38ms      2
2     agent-1     165ms       -15ms      3
3     agent-3     190ms       +10ms      1

Winner: agent-2 (142ms)
LLM Judge Mode (no eval command, or --judge flag)

For each agent:

  1. Get the diff: git diff {base_branch}...{agent_branch}
  2. Read the agent's result post from .agenthub/board/results/agent-{i}-result.md
  3. Compare all diffs and rank by:
    • Correctness — Does it solve the task?
    • Simplicity — Fewer lines changed is better (when equal correctness)
    • Quality — Clean execution, good structure, no regressions

Present rankings with justification.

Example LLM judge output for a content task:

RANK  AGENT    VERDICT                               WORD COUNT
1     agent-1  Strong narrative, clear CTA            1480
2     agent-3  Good data points, weak intro           1520
3     agent-2  Generic tone, no differentiation       1350

Winner: agent-1 (strongest narrative arc and call-to-action)
Hybrid Mode
  1. Run metric evaluation first
  2. If top agents are within 10% of each other, use LLM judge to break ties
  3. Present both metric and qualitative rankings

After Eval

  1. Update session state:
python {skill_path}/scripts/session_manager.py --update {session-id} --state evaluating
  1. Tell the user:
    • Ranked results with winner highlighted
    • Next step: /hub:merge to merge the winner
    • Or /hub:merge {session-id} --agent {winner} to be explicit

Comments

Markdown supported