mcp-failure-lab

MCP Failure Lab

Reproduce MCP timeouts, cancellation races, transport loss, and invalid responses with repeatable tests and CI reports.

Documentation · Compatibility · Project page

Example output (duration varies):

$ npx mcp-failure-lab demo
MCP Failure Lab — Demo
Running a real 500ms delay scenario...

Scenario: Deterministic delay demo
Outcome: success
Duration: ~500 ms
Assertions: passed

Why this exists

Real MCP clients behave differently when things break. A timeout may leave a connection usable; an interrupted response may close it. A malformed reply may be rejected by one client and accepted by another.

Failure Lab makes those cases repeatable so you can check both the failed call and what happens next.

Real-world findings

• Python SDK #3522: Python mcp 2.2.0 stayed closed after an interrupted HTTP response, rejected the next request, and raised an ExceptionGroup during cleanup.
• Rust SDK #1283: rmcp 3.4.0 accepted an invalid response containing both result and error over stdio and HTTP.
• Ruby SDK #589: mcp 1.6.1 accepted responses missing jsonrpc or declaring jsonrpc: "1.0" over stdio and HTTP in all three repeats. The Ruby report records all 60 executions; the results and upstream finding are included in the Observatory export.
• Java SDK 2.0.1 accepted jsonrpc: "1.0" over both transports (#1156). Over stdio, responses missing jsonrpc or containing both result and error left the next ping timing out (#1157). See the Java report.
• Duplicate-response comparison: all five tested SDKs completed the next ping. TypeScript reported the duplicate through its error callback; the other harnesses surfaced no call-level duplicate error.

The SDK comparisons describe specific tested versions, not every release. See the versioned reports for reproduction steps and limitations.

Quick start

Requires Node.js 22.23.4 or newer and npm.

npx mcp-failure-lab demo

The demo needs no API key, external server, or global installation. To test your own MCP client, connect it to Failure Lab over stdio or local Streamable HTTP:

npx mcp-failure-lab serve
# Or:
npx mcp-failure-lab serve --transport http

The stdio process waits for a client; it is not an interactive terminal command. Configure your client to launch it, or follow the getting started guide. Press Ctrl+C to stop a manually started server.

HTTP defaults to http://127.0.0.1:3000/mcp. It provides neither authentication nor TLS; do not expose it to an untrusted network.

Test your own MCP server

External targets support HTTP and stdio. After the repository setup, install the official GitHub MCP server with its executable on PATH and set GITHUB_PERSONAL_ACCESS_TOKEN in your environment. Then run:

npm run dev -- run examples/scenarios/github-get-me.json \
  --target examples/targets/github-stdio.json

The example calls GitHub's read-only get_me tool. Its target configuration uses envFrom to pass the token from your environment rather than storing it in JSON. For your own scenario and target configuration, use:

npx mcp-failure-lab run scenario.json --target target.json

External runs check tool results, deadlines, and adapter setup and cleanup. They do not inject faults into another server; Failure Lab is not a proxy.

See the external targets guide for prerequisites, credential handling, and configuration.

What Failure Lab can break

The built-in server exposes these tools for testing client behavior:

Tool	Behavior
ping	Returns a deterministic health response
protocol_ping_liveness	Sends a bounded protocol ping during its in-flight call
delay	Waits for a bounded duration before returning
hang	Remains pending until the client cancels
disconnect	Interrupts the active transport while a request is in flight
malformed_message	Violates one selected JSON-RPC response rule exactly once
duplicate_response	Sends the same JSON-RPC response twice for one request
response_after_cancellation	Sends one late response for a cancelled stdio request
session_loss	Invalidates the caller's legacy HTTP session
oversized_response	Returns a deterministic text payload up to 256 KiB
invalid_progress	Sends an unknown, regressing, or late progress notification

Protocol ping is distinct from the ping tool. Built-in scenarios default to MCP 2026-07-28; protocol liveness requires scenario protocolVersion: "2025-11-25". Both public transports accept legacy clients.

See the fault tools reference for timing bounds, activation, cancellation, cleanup, and transport limits.

oversized_response accepts sizeBytes from 0 through 262,144. It generates only ASCII x characters, so the text size is exact in UTF-8 bytes and contains no external data. The cap applies on both stdio and HTTP. Choose a size below your client's configured response limit to check success, then a size above that limit (but no more than 262,144) to check client rejection. The included 64 KiB scenario checks the exact returned text size and a follow-up request. The JSON-RPC and transport framing add bytes around the text, so a client's whole-message limit may trigger before sizeBytes reaches its threshold. The integration test confirms that a stdio client with a 4 KiB message limit accepts a 1 KiB payload and rejects an 8 KiB payload.

invalid_progress requires a client progress token and accepts unknown-token, regressing-progress, or after-completion. These are protocol notifications associated with one tool call; they are not tool result content. The unknown-token variant sends one notification with a different token. The regressing variant sends progress 2 followed by 1. The late variant sends one notification after the matching response on stdio or in the HTTP response stream. These violate request-token correlation, monotonic progress, and the completed-request lifecycle, respectively. Each variant is bounded and consumed once. See the regressing, unknown-token, and after-completion scenarios. The built-in runner can assert callback values with expect.progressValues; external target adapters do not report progress notifications and cannot use that assertion.

How recovery testing works

Trigger a fault, record its outcome, then make a second request on the same connection. A detected fault does not prove recovery; the follow-up must succeed too.

Scenario observe calls run after the primary call, including errors and timeouts. For example, duplicate-response.json calls duplicate_response, then uses ping to check that the connection remains usable:

# From a repository checkout
npm run dev -- run examples/scenarios/duplicate-response.json

This checks post-fault behavior, not an automatic recovery policy. A stdio disconnect terminates the server process and requires a new process and connection.

See Scenarios for outcome, duration, result, and observer assertions.

SDK / transport evidence

MCP Failure Observatory collects SDK and transport evidence. The repository reports preserve tested versions, methods, and limitations.

Results vary by SDK, version, transport, and protocol. A passing run is evidence for that combination, not a guarantee for every client.

CI usage

Save a scenario file with explicit expectations and timeouts, then produce a JUnit report:

npx mcp-failure-lab run scenario.json --report junit > junit.xml

Pin the Failure Lab package version in CI so upgrades do not change the test environment unexpectedly.

Exit codes: 0 means expectations passed, 1 means the scenario could not be loaded or executed, and 2 means an assertion failed. An expected timeout can pass; an unexpected success can fail.

See Reporting for JSON, JUnit, and lifecycle diagnostics.

Documentation

Detailed guides live at mcplab.dev: Getting started · CLI · Architecture · Troubleshooting.

Contributing

See CONTRIBUTING.md for setup, tests, and the contribution workflow. Planned work is tracked in GitHub Issues.

If Failure Lab helps you test an MCP integration, consider starring the repository.

License

MIT