NVSentinel Lab

Name: NVSentinel Lab
Author: omerfsen

Local

github.com/omerfsenOther0Updated 1d ago

Repository

About NVSentinel Lab

NVSentinel Lab is an MCP server for testing NVIDIA NVSentinel’s GPU fault detection and node handling in a kind lab cluster. It reads status, injects simulated faults, and checks cordoning and draining; it does not install NVSentinel.

Key features

01Read node, pod, and workload status
02Inspect scenarios, logs, and node events
03Inject and clear simulated GPU faults
04Run end-to-end fault test scenarios
05Keep lab-changing tools disabled by default

Use cases

01Verify XID 95 detection and node cordoning
02Test DCGM outage reporting and recovery
03Check workload eviction in a kind cluster
04Inspect NVSentinel reports through MCP

MCP support

The server exposes MCP tools over stdio for clients to inspect lab state, logs, and events, and run simulated fault tests. Tools that change the cluster are disabled by default and, when enabled, are restricted to the kind lab.

Configuration

Local server config is the same for every client. Copy it into your client’s MCP config file.

mcp.json · Universal config

{
  "mcpServers": {
    "omerfsen-nvidia-nvsentinel": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "-e",
        "NVSENTINEL_MCP_KUBECONFIG",
        "-e",
        "NVSENTINEL_MCP_ALLOW_CHANGES",
        "-e",
        "NVSENTINEL_MCP_LAB_CONTEXT",
        "docker.io/omerfsen/nvidia-nvsentinel-mcp:0.5.1"
      ],
      "env": {
        "NVSENTINEL_MCP_KUBECONFIG": "<NVSENTINEL_MCP_KUBECONFIG>",
        "NVSENTINEL_MCP_ALLOW_CHANGES": "<NVSENTINEL_MCP_ALLOW_CHANGES>",
        "NVSENTINEL_MCP_LAB_CONTEXT": "<NVSENTINEL_MCP_LAB_CONTEXT>"
      }
    }
  }
}

Run command

docker run -i --rm -e NVSENTINEL_MCP_KUBECONFIG -e NVSENTINEL_MCP_ALLOW_CHANGES -e NVSENTINEL_MCP_LAB_CONTEXT docker.io/omerfsen/nvidia-nvsentinel-mcp:0.5.1

Environment variables to set

NVSENTINEL_MCP_KUBECONFIGThe one kubeconfig it uses, private to you. Default in the image: /kube/config.

NVSENTINEL_MCP_ALLOW_CHANGESExactly true allows the change tools, on a kind cluster only. Off when unset.

NVSENTINEL_MCP_LAB_CONTEXTThe kubeconfig context changes need. Must start with kind-. Default kind-nvsentinel-lab.

Local servers need a runtime on your machine (e.g. Node.js 18+). Config file locations differ by client; see your client’s docs.

Packages

ocidocker.io/omerfsen/nvidia-nvsentinel-mcp:0.5.1stdio

Tools

No tool list yet

Local servers expose tools only once running on your machine, and the maintainer has not provided a tool list.

Details

From the project README

nvidia-nvsentinel-mcp

An MCP server for testing NVIDIA NVSentinel (GPU fault detection and remediation on Kubernetes) on a lab cluster, written in Go like NVSentinel itself. An MCP client such as Claude Code can read what NVSentinel reports, inject a fault into a simulated GPU, and check that NVSentinel cordons and drains the node.

This repo holds the server and example test results. It does not install NVSentinel: that is omerfsen.aicr, on a cluster from omerfsen.nvke or on kind. It is meant to go upstream one day: see docs/upstream.md.

Test results

Real output from NVSentinel v1.30.0 on kind, through this server over MCP: docs/test-results. These prove NVSentinel's pipeline against simulated DCGM, not real GPU health.

Standards

This server follows these standards. Each requirement, its status, the test that proves it and the one known gap are in docs/mcp-standards.md, with links to check for newer versions.

Standard	Version
Model Context Protocol	2026-07-28, and back to 2024-11-05
MCP Registry `server.json`	2025-12-11, valid and published (0.5.1)
AGENTS.md	AGENTS.md, imported by `CLAUDE.md`
Agent Skills	skills/nvsentinel-lab

It has not been run through the official conformance suite, which only tests HTTP servers. MCP has no standard for written docs, so docs/tools.md is generated from the server, the common practice.

Tools

Tool	Changes the lab	What it does
`lab_status`	no	Nodes (cordon, NVSentinel state, failing GPU checks), pods, the workload
`list_scenarios`	no	Scenarios, stages, allowed XIDs, log sources
`wait_for_stage`	no	Polls until a stage holds, such as `cordoned`
`component_logs`	no	The last lines of a NVSentinel component's logs
`node_events`	no	Recent events on the GPU node
`inject_fault`	yes	Injects XID 95 into the simulated GPU
`clear_fault`	yes	Restarts the simulated hostengine, which drops the fault
`reset_lab`	yes	Undoes a DCGM outage and an injected fault
`run_test_scenario`	yes	Runs a scenario end to end and reports each step

Scenario	What it checks
`xid95`	Inject XID 95: detected, cordoned, workload evicted, drain succeeded. Clear it: healthy, uncordoned, workload back
`dcgm_outage`	Take the hostengine away: the monitor reports it lost DCGM. Restore it: the condition clears

Tools that change anything are off unless NVSENTINEL_MCP_ALLOW_CHANGES=true, and then work only on a kind cluster. Details: docs/security.md.

Every tool's inputs, outputs and annotations: docs/tools.md, generated from what the server returns for tools/list, as is the raw docs/tools.json. A test fails if they no longer match the server; make docs regenerates them.

Install

The server needs a kubeconfig. For the kind lab, make lab-token writes one (see Run the tests on kind). It speaks MCP over stdio, so any MCP client can start it. All the commands below leave changes off; add -e NVSENTINEL_MCP_ALLOW_CHANGES=true only for a client that should change the lab.

1. Install the server. Download the binary for your platform from the latest release:

Platform	File
Linux, x86-64	`nvidia-nvsentinel-mcp-linux-amd64`
Linux, Arm	`nvidia-nvsentinel-mcp-linux-arm64`
macOS, Intel	`nvidia-nvsentinel-mcp-darwin-amd64`
macOS, Apple silicon	`nvidia-nvsentinel-mcp-darwin-arm64`
Windows, x86-64	`nvidia-nvsentinel-mcp-windows-amd64.exe`
Windows, Arm	`nvidia-nvsentinel-mcp-windows-arm64.exe`

On Linux and macOS, set FILE to yours:

FILE=nvidia-nvsentinel-mcp-linux-amd64
BASE=https://github.com/omerfsen/nvidia-nvsentinel-mcp/releases/latest/download
curl -fsSLO "$BASE/$FILE"
curl -fsSL "$BASE/checksums.txt" | grep " $FILE\$" | sha256sum -c -   # macOS: shasum -a 256 -c -
gh attestation verify "$FILE" --repo omerfsen/nvidia-nvsentinel-mcp    # optional: who built it
install -m 755 "$FILE" ~/.local/bin/nvidia-nvsentinel-mcp

On Windows, in PowerShell, keep the kubeconfig where only you can read it, as the server cannot check that there:

$File = "nvidia-nvsentinel-mcp-windows-amd64.exe"
$Base = "https://github.com/omerfsen/nvidia-nvsentinel-mcp/releases/latest/download"
Invoke-WebRequest "$Base/$File" -OutFile "$HOME\bin\nvidia-nvsentinel-mcp.exe"

Each release also has checksums.txt and an SBOM, nvidia-nvsentinel-mcp.sbom.cdx.json. Only Linux on x86-64 has been run; the others are built but untested. Or install with Go 1.27: go install github.com/omerfsen/nvidia-nvsentinel-mcp@latest. Or use the image, docker.io/omerfsen/nvidia-nvsentinel-mcp, which never runs as root and is about 32 MB.

Check it with nvidia-nvsentinel-mcp --version. Run with no flags, it logs one line to stderr and then waits for an MCP client on stdin, so it looks idle: that is normal, and Ctrl-C stops it. --help lists the flags, and --debug logs each tool call and cluster detail to stderr.

2. Add it to your client. For Claude Code, with the binary on your PATH:

claude mcp add nvsentinel-lab -- nvidia-nvsentinel-mcp

Or give the full path to the binary, such as ~/.local/bin/nvidia-nvsentinel-mcp.

With the image:

claude mcp add nvsentinel-lab -- docker run --rm -i --read-only --cap-drop=ALL \
  --security-opt=no-new-privileges --user "$(id -u):$(id -g)" --memory=128m --cpus=1 \
  --pids-limit=64 --network=kind \
  --volume "$HOME/.kube/nvsentinel-lab-mcp-docker.yaml:/kube/config:ro" \
  docker.io/omerfsen/nvidia-nvsentinel-mcp@sha256:cb6fba848870580f77d347d476a2a7003cd336cd6f4807a72928013538d7aed9

Other clients take the same command in their MCP config. Use the full path if the binary is not on the PATH your client sees:

{ "mcpServers": { "nvsentinel-lab": { "command": "nvidia-nvsentinel-mcp" } } }

3. Optional: add the skill, which teaches an agent how to use the tools. It holds only instructions, so step 1 and 2 are still needed. Either command works:

gh skill install omerfsen/nvidia-nvsentinel-mcp nvsentinel-lab
npx skills add omerfsen/nvidia-nvsentinel-mcp --skill nvsentinel-lab

The GitHub CLI has no gh mcp add as of v2.103.2: gh skill installs skills, and MCP servers go in through the client. It is listed on the official MCP Registry as io.github.omerfsen/nvidia-nvsentinel-mcp.

Run the tests on kind

Needs Go, Docker, kubectl, helm, yq, about 10 GiB of free memory, and a checkout of omerfsen/aicr with make dev done (until https://github.com/omerfsen/aicr/pull/8 merges, its dev/osen/nvsentinel-latest branch).

make lab-up           # the kind cluster, with its own kubeconfig
make lab-install      # NVSentinel v1.30.0 through omerfsen.aicr (AICR_DIR=<checkout>)
make lab-fixture      # simulated DCGM, a labelled node, a workload, the server's RBAC
make lab-token        # a kubeconfig for the server: its own identity, valid 8 hours
make e2e              # the tests, through the real binary over MCP; writes docs/test-results
make image e2e-image  # the same, with the server in its container
make lab-down

Tests with no cluster: make lint test.

Use it from Claude Code

From a checkout, for working on the server itself: make build lab-token, then restart Claude Code in this directory and approve the server from .mcp.json. That one allows changes.

Settings

Variable	Default	What it does
`NVSENTINEL_MCP_KUBECONFIG`	`~/.kube/nvsentinel-lab-mcp.yaml` (image: `/kube/config`)	The one kubeconfig it uses. Must be private (`chmod 600`)
`NVSENTINEL_MCP_ALLOW_CHANGES`	off	Exactly `true` allows the change tools, on the lab only
`NVSENTINEL_MCP_LAB_CONTEXT`	`kind-nvsentinel-lab`	The context changes need. Must start with `kind-`

What is not tested

The binaries for anything but Linux on x86-64: they are cross-compiled and not run.
A real GPU. The RTX 4080 SUPER is not on NVSentinel's validated list, and real DCGM health is still to be checked.
AICR's install on NVKE, as opposed to kind.
Remediation. The kind install stops at cordon and drain: AICR's remediation mixin needs metadata-collector, which needs NVML.
The syslog monitor: it reads kernel-transport journal entries, which a kind node cannot safely produce, so it is off on kind.
The image's cosign signature and SBOM attestation by hand: they are attached, and only its build provenance was verified (gh attestation verify).

Every step, failure and dead end is in docs/build-log.md.

License

Apache-2.0. lab/gpu-spec.yaml and lab/fake-dcgm.yaml come from NVSentinel, under the same license.

Comments

Markdown supported

New to MCP?

From what the protocol is to installing your first server, these answers cover most first questions.

Read the getting-started guide

An open protocol that defines how AI applications connect to external tools and data in a uniform way. Clients talk to the model; MCP servers provide tools, resources and prompt templates. They exchange standard messages, so one server can be reused by many clients.

NVSentinel Lab

About NVSentinel Lab

Key features

Use cases

MCP support

Configuration

Tools

Details

nvidia-nvsentinel-mcp

Test results

Standards

Tools

Install

Run the tests on kind

Use it from Claude Code

Settings

What is not tested

License

Comments

New to MCP?

What is MCP (Model Context Protocol)?

How do MCP servers differ from clients?

Should I use a local or a remote server?

How do I install a server in Claude, Cursor and others?

Are listed projects reviewed? What does “Verified” mean?

My project is listed. How do I claim it?