mcp-scorecard

mcp-scorecard

Agent-readiness scorecard for any MCP server.
Probes a target over stdio, runs 10 checks, outputs a 0-100 score with itemized findings.

---

Quick start

# Audit a published npm package
npx -y mcp-scorecard whoop-mcp-unofficial

# Audit a GitHub repo (auto-resolves to the published npm package, or local dist)
npx -y mcp-scorecard https://github.com/davidmosiah/whoop-mcp

# Audit a local build (entry file or package directory)
npx -y mcp-scorecard /Users/you/Desktop/my-mcp/dist/index.js
npx -y mcp-scorecard ./my-mcp          # resolves package.json bin/main
npx -y mcp-scorecard ./dist/index.js   # relative paths work offline

# CI gate: fail the build if the score drops
npx -y mcp-scorecard my-mcp --min-score 80

# Structured JSON for piping into your own tooling
npx -y mcp-scorecard my-mcp --json

# Run scorecard itself as an MCP server (stdio default)
npx -y mcp-scorecard serve

# Same server on Streamable HTTP — v2 stateless, loopback only
npx -y mcp-scorecard serve --http
# GET  http://127.0.0.1:3000/health
# POST http://127.0.0.1:3000/mcp   (sessionless)

Env for HTTP: MCP_SCORECARD_HOST, MCP_SCORECARD_PORT, MCP_SCORECARD_TRANSPORT=http.

Demo

Real captured run auditing the official MCP reference server @modelcontextprotocol/server-everything — nothing here is hand-edited, this is exactly what the CLI printed:

$ npx -y mcp-scorecard @modelcontextprotocol/server-everything

# mcp-scorecard - @modelcontextprotocol/server-everything @2026.1.26

**Agent-readiness score:** 44/100

- [PASS] Schema validity              (13/13 tools have valid input schema)
- [FAIL] Tool naming convention       (12/13 tools violate snake_case)
- [FAIL] Privacy modes documented     (only 1 tool(s) mention privacy modes)
- [PASS] Mutation gating              (no write tools — n/a)
- [FAIL] Agent manifest               (no agent_manifest tool)
- [FAIL] Smoke test                   (no smoke script and no test script)
- [PASS] Resources advertised         (7 resources registered)
- [PASS] Tool descriptions            (avg 88 chars across 13 tools)
- [FAIL] Annotations                  (0/13 read tools annotated)
- [FAIL] Manifest discoverability     (no discovery tools)

## Details
### Tool naming convention
- Non-snake_case names: get-annotated-message, get-env, get-resource-links, get-resource-reference, get-structured-content, get-sum, get-tiny-image, gzip-file-as-resource, toggle-simulated-logging, toggle-subscriber-updates
### Annotations
- Missing readOnlyHint: echo, get-annotated-message, get-env, get-resource-links, get-resource-reference, get-structured-content, get-sum, get-tiny-image, gzip-file-as-resource, toggle-simulated-logging

## Suggested fixes
- Rename tools to lowercase snake_case (a-z, 0-9, _).
- Add a `privacy_mode` parameter (summary | structured | raw) on read tools so agents can request only what they need.
- Expose a `<prefix>_agent_manifest` tool that returns { recommended_first_calls, standard_tools, ... } so agents can self-onboard.
- Add `scripts/smoke-tools.mjs` that boots the server via StdioClientTransport and asserts the tool list.
- Add `annotations: { readOnlyHint: true, openWorldHint: false }` to every read tool definition.
- Expose discovery tools so agents can self-onboard: `*_agent_manifest`, `*_data_inventory`, `*_capabilities`, `*_connection_status`.

The reference server is a feature showcase, not a production integration — a 44 is expected and is exactly why the conventions checks exist. Agent-oriented servers that adopt snake_case naming, a manifest tool, and read-only annotations land in the 80s and 90s.

What it checks

Ten quality dimensions, each scored 0-10. Final score is the sum, capped at 100.

1. Schema validity - boots the server, lists tools, and validates each tool's inputSchema with ajv. Missing or non-object schemas score zero per tool.
2. Tool naming convention - rewards a shared prefix and snake_case (e.g. whoop_get_sleep). Mixed case, hyphens, and missing prefixes lose points.
3. Privacy modes documented - looks for a privacy_mode parameter on any tool, or descriptions that mention summary | structured | raw.
4. Mutation gating - any tool name matching (set|update|delete|create| pause|resume|enable|disable|cancel|publish|send) must document a gate in its description (Gated by ALLOW_MUTATIONS, requires explicit user intent, dry-run, confirm).
5. Agent manifest - calls <prefix>_agent_manifest and checks the response object has recommended_first_calls (non-empty array) AND standard_tools (non-empty array). The probe NEVER persists the payload
   • only field names and lengths are recorded.
6. Smoke test present - looks for scripts/smoke*.{mjs,js,ts} in the package, or a real test script in package.json (not the npm default echo-and-fail).
7. Resources advertised - counts what listResources() returns. Zero scores zero; one or two scores 5; three or more scores 10.
8. Tool descriptions - average description length across all tools. Below 30 chars scores 0; 30-60 scores 5; 60+ scores 10.
9. Annotations - counts what fraction of read tools (any non-mutation tool) carry annotations.readOnlyHint = true. Score scales linearly.
10. Manifest discoverability - has any of *_agent_manifest, *_data_inventory, *_capabilities, *_connection_status. Two or more scores 10; exactly one scores 7; none scores 0.

Output format

Markdown (default)

# mcp-scorecard - whoop-mcp-unofficial @0.4.3

**Agent-readiness score:** 88/100

- [PASS] Schema validity              (28/28 tools have valid input schema)
- [PASS] Tool naming convention       (consistent `whoop_` prefix, snake_case)
- [PASS] Privacy modes documented     (privacy_mode parameter on 6 tool(s))
- [PASS] Mutation gating              (no write tools - n/a)
- [PASS] Agent manifest               (recommended_first_calls present, 5 entries)
- [PASS] Smoke test                   (scripts/smoke-tools.mjs found)
- [PASS] Resources advertised         (8 resources registered)
- [PASS] Tool descriptions            (avg 142 chars across 28 tools)
- [WARN] Annotations                  (20/28 read tools annotated)
- [PASS] Manifest discoverability     (4/4 discovery tools present (agent_manifest, data_inventory, capabilities, connection_status))

## Suggested fixes
- Add `annotations: { readOnlyHint: true, openWorldHint: false }` to every read tool definition.

_Generated by mcp-scorecard v0.1.0 at 2026-05-23T15:42:11.000Z_

JSON (--json)

{
  "target": {
    "displayName": "whoop-mcp-unofficial",
    "version": "0.4.3",
    "serverName": "whoop-mcp",
    "serverVersion": "0.4.3"
  },
  "totalScore": 88,
  "checks": [
    {
      "id": "schema_validity",
      "label": "Schema validity",
      "score": 10,
      "status": "pass",
      "summary": "28/28 tools have valid input schema",
      "details": [],
      "fixes": []
    }
  ],
  "generatedAt": "2026-05-23T15:42:11.000Z",
  "scorecardVersion": "0.1.0"
}

How it probes

The scorecard launches the target MCP server over stdio with MCP_PROBE=1 set on the child's env. Only the MCP SDK's platform-specific runtime environment allowlist (such as PATH and HOME on Unix) is inherited; shell API keys, auth sockets, proxy settings, and code-loading settings such as NODE_OPTIONS are not forwarded. Author hook: if your MCP needs OAuth or other credentials to even list tools, detect this env var and return your tool/resource/prompt manifests anyway. The scorecard expects to be able to read your contract without making any auth-requiring API calls.

Stdio auditing executes the target's code as the current OS user and may call an *_agent_manifest tool. It is not a read-only or idempotent operation and does not sandbox filesystem access, network access, or the target's effects. Audit trusted targets; use a separate isolated environment for untrusted code. MCP_PROBE=1 asks the target to cooperate and cannot enforce these limits.

Privacy model

• The probe response is never persisted. Only counts and field names are recorded into the in-memory snapshot used by checks.
• Any string going into the report is run through a redaction pass that replaces values for customer_id, email, phone, access_token, refresh_token, client_secret, developer_token, and api_key with [REDACTED].
• The scorecard sends no telemetry. Resolving npm/GitHub targets may use the network, hosted URL audits make HTTP requests, and executed stdio targets can make their own network calls. The probe's environment allowlist does not prevent access to credentials stored in files or OS services.

Use cases

• CI gate - block a PR if the score drops below a threshold: npx -y mcp-scorecard my-mcp --min-score 85.
• Registry curation - run the audit across a list of MCPs to pick the best-documented and most agent-friendly options for a directory or catalog.
• Pre-publish self-check - add it to prepublishOnly so you never ship a regressed contract by accident.
• Comparative review - score two MCPs that ostensibly cover the same API and see which is friendlier to agents.

Limitations

• Cannot probe MCPs that REQUIRE auth before listTools() returns. If your server is one of these, support the MCP_PROBE env hook so the scorecard can still read your contract.
• Cannot grade logic correctness - the scorecard only inspects shape, metadata, and discoverability. A server can score 100/100 and still return wrong data.
• Mutation detection is name-based - if you call a write tool prepare_thing instead of set_thing, the check will miss it. This is intentional: we reward clear naming.

Roadmap

• v0.2 - rule pack for agent-rules-of-thumb (output shape stability, pagination patterns, error envelope consistency), a perf benchmark (median listTools() latency), and an OAuth probe with a mock token.

Support

• Issues: https://github.com/davidmosiah/mcp-scorecard/issues
• Email: [email protected]

Disclaimer

This is a quality audit tool. It does not certify security, privacy, or correctness; it only measures whether a server follows agent-friendly conventions. Always do your own review before plugging an MCP into a production agent.

License

MIT - see LICENSE. Code of Conduct.

---

New in v0.3.0 — differentiators

• Run it AS an MCP server so agents can call it mid-task:

  { "mcpServers": { "mcp-scorecard": { "command": "npx", "args": ["-y", "mcp-scorecard", "serve"] } } }
  

  Then your agent can audit("some-mcp-server") before installing it. An MCP that scores MCPs.
• Audit hosted/remote servers by URL (12 security + agent-readiness checks): npx -y mcp-scorecard https://your-server
• Badge for your README: npx -y mcp-scorecard my-mcp --badge → a shields.io markdown badge.
• GitHub Action:

  - uses: davidmosiah/[email protected]
    with:
      target: dist/index.js
      min-score: 80
  

v0.4.0

• mcp-scorecard compare a b c — side-by-side ranking.
• --profile security|quality|agent-ready — score one category. --baseline old.json — regression diff. --html — shareable scorecard.
• Checks explainer: https://mcp-scorecard.vercel.app/checks.html