mcpField
E
evaluations

evaluations

github.com/langwatchTesting & quality3138 installsUpdated 1d ago
Repository

About evaluations

evaluations is a compatibility skill that routes evaluation requests to the appropriate Langwatch skill. It distinguishes dataset-based tests from evaluation of live production traffic and asks for clarification when the request is ambiguous.

Key features

  1. 01Routes requests to experiments or online-evaluations
  2. 02Distinguishes dataset tests from live traffic
  3. 03Asks when evaluation intent is unclear
  4. 04Handles retry for specific validation errors

Use cases

  1. 01Directing benchmark requests to experiments
  2. 02Routing production monitoring to online-evaluations
  3. 03Clarifying an unspecified evaluation request

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/langwatch/skills --skill evaluations

After installing, type/evaluationsin Claude Code to confirm it loaded.

Details

From the project README

Route an Evaluation Request

This is a compatibility skill. Do not build an experiment, monitor, or guardrail from this skill.

Classify the user's intent:

IntentCorrect skill
Batch test a dataset, compare prompts or models, benchmark, create a CI quality gateexperiments
Score live traces or threads, monitor production quality, create a guardrailonline-evaluations

If the request remains ambiguous after inspecting context (a bare "make me an eval" that names neither a dataset nor live traffic), do not create anything yet. This choice picks what gets tested, so it is the user's to make, not a default's. Ask it as a question card and stop; the answer arrives as the next message.

In Langy, ask it with the question tool and stop; the options are the concrete alternatives, for example "A dataset, before deployment" and "Live production traffic". Elsewhere, ask the same question as one short line of prose.

A rejected field value is not this kind of choice. If a create later fails with a validation_error whose reason names the field and an expected list, correct that exact field from the list and retry once. Never turn a fixable slug into a question for the user.

Then hand off:

  1. If the correct companion skill is available, load it and follow it instead of continuing here.

  2. If experiments is missing, tell the user to install it with:

    npx [email protected] add langwatch/skills/experiments

  3. If online-evaluations is missing, tell the user to install it with:

    npx [email protected] add langwatch/skills/online-evaluations

Do not recreate the companion skill's instructions from memory. Load the focused skill so its current workflow, safety checks, and verification steps are used.

Comments

Markdown supported