About modelwatch
modelwatch-mcp connects MCP-compatible clients to ModelWatch, a service for monitoring behavioral drift in LLM applications. It lets teams manage behavioral specs and inspect runs, health, and drift events from a chat interface.
Key features
- 01Manage monitored LLM endpoints
- 02Create behavioral specs and schedules
- 03Run specs on demand
- 04Reset baselines after model changes
- 05Review drift events and spec health
Use cases
- 01Check whether an endpoint’s behavior has changed
- 02Run a behavioral test from a chat client
- 03Inspect a spec’s history and latest drift status
MCP support
Exposes ModelWatch operations as tools to MCP-compatible clients, including endpoint and spec management, test runs, baseline resets, and drift reporting.
Configuration
Local server config is the same for every client. Copy it into your client’s MCP config file.
npx -y modelwatch-mcpMODELWATCH_API_KEYSecretAPI key from https://modelwatch.app (free tier: 5 specs, 500 runs/mo).MODELWATCH_API_BASEOverride the API base URL for self-hosted deployments.Local servers need a runtime on your machine (e.g. Node.js 18+). Config file locations differ by client; see your client’s docs.
Tools
Local servers expose tools only once running on your machine, and the maintainer has not provided a tool list.
Details
From the project READMEmodelwatch-mcp
MCP server for ModelWatch — continuous behavioral drift monitoring for LLM-powered applications.
Lets Claude Desktop / Claude Code (or any MCP-compatible client) create drift specs, run them, and surface drift events without leaving the chat.
Install
npm install -g modelwatch-mcp
Configure (Claude Desktop)
Add to ~/Library/Application Support/Claude/claude_desktop_config.json:
{
"mcpServers": {
"modelwatch": {
"command": "npx",
"args": ["-y", "modelwatch-mcp"],
"env": {
"MODELWATCH_API_KEY": "mw_PASTE_FROM_EMAIL"
}
}
}
}
Get a free key at https://modelwatch.app/#signup (5 specs, 500 runs/mo, no card). Then restart Claude Desktop.
The example configuration stores a plaintext key in a local config file. Restrict that file to your user account, do not commit it, and rotate the key if it is exposed. The key never belongs in a command-line argument.
Tools
| Tool | What it does |
|---|---|
list_endpoints | Show monitored LLM endpoints |
create_endpoint | Add an endpoint (provider + model) |
list_specs | Show behavioral specs |
create_spec | Add a spec (prompt + schedule + optional semantic similarity threshold) |
run_spec | Run a spec on demand; baseline if first run |
reset_baseline | Re-baseline after an intentional model swap |
get_drift_events | Recent drift events across the workspace |
get_spec_history | Run history for one spec |
get_health | Per-spec health status, last drift score and last run time |
Example session
You: Create a refusal canary against the OpenAI endpoint, asking about chest pain. Run it now and tell me the baseline.
Claude calls list_endpoints → create_spec → run_spec and reports the
score. The same spec then runs daily forever and emails you when drift is
detected.
Environment variables
| Var | Required | Default |
|---|---|---|
MODELWATCH_API_KEY | Yes | — |
MODELWATCH_API_BASE | No | https://api.modelwatch.app |
Self-hosted? Point MODELWATCH_API_BASE at an HTTPS origin. HTTP is accepted
only for loopback development; paths, query strings, URL credentials and
redirects are rejected before a key is sent.
Source
https://github.com/bch1212/modelwatch — MIT licensed.
Comments
New to MCP?
From what the protocol is to installing your first server, these answers cover most first questions.
An open protocol that defines how AI applications connect to external tools and data in a uniform way. Clients talk to the model; MCP servers provide tools, resources and prompt templates. They exchange standard messages, so one server can be reused by many clients.