metrillm

Name: metrillm
Author: MetriLLM

Local

github.com/MetriLLMAI & Agents8Updated 21w ago

Repository

About metrillm

MetriLLM is an MCP server for developers evaluating local LLMs through Ollama or LM Studio. It lists models, runs performance and quality benchmarks, retrieves saved results, and can upload results to a public leaderboard.

Key features

01List locally available models
02Benchmark speed and quality
03Run performance-only benchmarks
04Retrieve saved benchmark results
05Upload results to public leaderboard

Use cases

01Compare local models before selecting one
02Check model performance on available hardware
03Review or share benchmark results

MCP support

Exposes tools to MCP clients for listing local models, running benchmarks, retrieving results, and uploading results to the public leaderboard.

Configuration

Local server config is the same for every client. Copy it into your client’s MCP config file.

mcp.json · Universal config

{
  "mcpServers": {
    "metrillm-metrillm": {
      "command": "npx",
      "args": [
        "-y",
        "metrillm-mcp"
      ]
    }
  }
}

Run commandnpx -y metrillm-mcp

Local servers need a runtime on your machine (e.g. Node.js 18+). Config file locations differ by client; see your client’s docs.

Packages

npmmetrillm-mcp0.2.6stdio

Tools

Saved tool list · updated 2w ago

Parameter	Type	Required	Description
runtime	enum	No	Inference runtime to use (ollama \| lm-studio).ollama \| lm-studio

Details

From the project README

MetriLLM MCP Server

MCP (Model Context Protocol) server for MetriLLM — benchmark local LLMs directly from Claude Code, Cursor, Windsurf, Continue.dev, or any MCP-compatible client.

Quick Start

Claude Code

claude mcp add metrillm -- npx metrillm-mcp@latest

Claude Desktop

Add to claude_desktop_config.json:

{
  "mcpServers": {
    "metrillm": {
      "command": "npx",
      "args": ["metrillm-mcp@latest"]
    }
  }
}

Cursor / Windsurf / Continue.dev

Add to your editor's MCP configuration:

{
  "mcpServers": {
    "metrillm": {
      "command": "npx",
      "args": ["metrillm-mcp@latest"]
    }
  }
}

Prerequisites

Node.js >= 20
One local runtime available:
- Ollama (ollama serve)
- LM Studio (Local Server enabled)
At least one model available on the selected runtime

Available Tools

`list_models`

List all locally available LLM models.

Param	Type	Default	Description
`runtime`	`"ollama" \| "lm-studio"`	`"ollama"`	Inference runtime

Example response:

{
  "models": [
    { "name": "llama3.2:3b", "size": 2019393189, "parameterSize": "3.2B", "quantization": "Q4_K_M", "family": "llama" }
  ],
  "count": 1
}

`run_benchmark`

Run a full benchmark (performance + quality) on a local model.

Param	Type	Default	Description
`model`	`string`	(required)	Model name (e.g. `"llama3.2:3b"`)
`runtime`	`"ollama" \| "lm-studio"`	`"ollama"`	Inference runtime
`perfOnly`	`boolean`	`false`	If `true`, measure performance only (skip quality)

Example response:

{
  "success": true,
  "model": "llama3.2:3b",
  "verdict": "GOOD",
  "globalScore": 65,
  "performance": {
    "tokensPerSecond": 42.5,
    "ttftMs": 120,
    "memoryUsedGB": 2.1,
    "memoryPercent": 13
  },
  "interpretation": "This model runs well on your hardware."
}

`get_results`

Retrieve previous benchmark results stored locally.

Param	Type	Default	Description
`model`	`string`	(optional)	Filter by model name (substring match)
`runtime`	`"ollama" \| "lm-studio"`	`"ollama"`	Inference runtime

`share_result`

Upload a result to the public MetriLLM leaderboard.

Param	Type	Description
`resultFile`	`string`	Absolute path to the result JSON file

Uses official MetriLLM upload defaults out of the box.

Optional overrides for self-hosted/staging deployments:

METRILLM_SUPABASE_URL
METRILLM_SUPABASE_ANON_KEY
METRILLM_PUBLIC_RESULT_BASE_URL

Architecture

The MCP server is a thin wrapper around the existing MetriLLM CLI logic:

mcp/src/index.ts  → MCP entry point (stdio transport)
mcp/src/tools.ts  → Tool definitions + calls to CLI modules
    ↓
../src/core/      → CLI logic reused directly
../src/commands/  → CLI commands (bench, list)

No code duplication — the MCP server imports CLI modules directly.

Supported Runtimes

Runtime	Status
Ollama	Supported
LM Studio	Supported
MLX	Planned
llama.cpp	Planned
vLLM	Planned

The runtime parameter is present on every tool to prepare for multi-runtime support. Unimplemented runtimes return a clear error.

License

Apache License 2.0

Comments

Markdown supported

New to MCP?

From what the protocol is to installing your first server, these answers cover most first questions.

Read the getting-started guide

An open protocol that defines how AI applications connect to external tools and data in a uniform way. Clients talk to the model; MCP servers provide tools, resources and prompt templates. They exchange standard messages, so one server can be reused by many clients.