mcpField
B
behavioral-evals

behavioral-evals

github.com/google-geminiTesting & quality108.6k287 installsUpdated today
Repository

About behavioral-evals

behavioral-evals is a guide to evaluating agent decisions, such as tool selection, rather than only functional outcomes. It covers test selection, assertions, verification, and companion procedures for creating, fixing, and promoting evaluations.

Key features

  1. 01Evaluate agent decisions and tool choices
  2. 02Choose a test rig based on interaction needs
  3. 03Assert through breakpoints or tool logs
  4. 04Verify tests locally with Vitest
  5. 05Guides for creating, fixing, and promoting evals

Use cases

  1. 01Validate prompt or tool changes
  2. 02Investigate regressions in agent behavior
  3. 03Choose an evaluation approach for interaction-heavy cases

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/google-gemini/gemini-cli --skill behavioral-evals

After installing, type/behavioral-evalsin Claude Code to confirm it loaded.

Bundled files

References and scripts installed along with SKILL.md.

assets/
interactive_eval.ts.txt
standard_eval.ts.txt
references/
creating.md
fixing.md
promoting.md
running.md

Details

From the project README

Behavioral Evals

Overview

Behavioral evaluations (evals) are tests that validate the agent's decision-making (e.g., tool choice) rather than pure functionality. They are critical for verifying prompt changes, debugging steerability, and preventing regressions.

Note

Single Source of Truth: For core concepts, policies, running tests, and general best practices, always refer to evals/README.md.

🔄 Workflow Decision Tree

  1. Does a prompt/tool change need validation?
    • No -> Normal integration tests.
    • Yes -> Continue below.
  2. Is it UI/Interaction heavy?
  3. Is it a new test?
    • Yes -> Set policy to USUALLY_PASSES.
    • No -> ALWAYS_PASSES (locks in regression).
  4. Are you fixing a failure or promoting a test?

📋 Quick Checklist

1. Setup Workspace

Seed the workspace with necessary files using the files object to simulate a realistic scenario (e.g., NodeJS project with package.json).

2. Write Assertions

Audit agent decisions using rig.setBreakpoint() (AppRig only) or index verification on rig.readToolLogs().

3. Verify

Run single tests locally with Vitest. Confirm stability locally before relying on CI workflows.

📦 Bundled Resources

Detailed procedural guides:

  • creating.md: Assertion strategies, Rig selection, Mock MCPs.
  • fixing.md: Step-by-step automated investigation, architecture diagnosis guidelines.
  • promoting.md: Candidate identification criteria and threshold guidelines.

Comments

Markdown supported