mcpField
R
ragmetric-mcp

ragmetric-mcp

Local
github.com/MukundaKattaMemory & Knowledge0Updated 20w ago
Repository

About ragmetric-mcp

ragmetric-mcp is an MCP server for evaluating RAG retrieval with standard information-retrieval metrics. It calculates individual metrics or averages them across batches of queries with known relevant document IDs.

Key features

  1. 01Calculate Recall@k and Hit@k
  2. 02Calculate MRR and NDCG@k
  3. 03Evaluate batches across queries

Use cases

  1. 01Measure retrieval quality against known relevant documents
  2. 02Compare RAG retriever iterations across a query set

MCP support

Exposes retrieval metric and batch-evaluation tools to MCP clients.

Configuration

Local server config is the same for every client. Copy it into your client’s MCP config file.

mcp.json · Universal config
{
  "mcpServers": {
    "mukundakatta-ragmetric": {
      "command": "npx",
      "args": [
        "-y",
        "@mukundakatta/ragmetric-mcp"
      ]
    }
  }
}
Run commandnpx -y @mukundakatta/ragmetric-mcp

Local servers need a runtime on your machine (e.g. Node.js 18+). Config file locations differ by client; see your client’s docs.

Packages
npm@mukundakatta/ragmetric-mcp0.1.1

Tools

5

Saved tool list · updated 2w ago

ParameterTypeRequiredDescription
kintegerYes
relevantarrayYesGround-truth relevant doc IDs (binary relevance).
retrievedarrayYesDoc IDs returned by retriever, in rank order.

Details

From the project README

@mukundakatta/ragmetric-mcp

MCP server for RAG retrieval evaluation. Lets an LLM compute IR metrics on demand: Recall@k, Hit@k, MRR, NDCG@k, plus an evaluate_batch convenience that runs all four across many queries.

Tools

ToolReturns
recall_at_k(retrieved, relevant, k)Fraction of relevant items in top k
hit_at_k(retrieved, relevant, k)1.0 if any relevant in top k else 0
mrr(retrieved, relevant)1 / rank of first relevant, 0 if none
ndcg_at_k(retrieved, relevant, k)NDCG@k with binary relevance (log2 discount)
evaluate_batch(queries, k)Mean of each metric across many queries

retrieved and relevant are arrays of doc IDs (strings).

Install — Claude Desktop

{
  "mcpServers": {
    "ragmetric": {
      "command": "npx",
      "args": ["-y", "@mukundakatta/ragmetric-mcp"]
    }
  }
}

Install — Cursor / Cline / Windsurf / Zed

Same shape per tool. npx -y @mukundakatta/ragmetric-mcp over stdio.

Use case

You're iterating on a RAG retriever. Hand the LLM a JSONL of queries with ground-truth doc IDs, ask it to call evaluate_batch, and it returns the mean metrics in one call. No notebook, no boto3, no scikit-learn.

License

MIT OR Apache-2.0.

Comments

Markdown supported

New to MCP?

From what the protocol is to installing your first server, these answers cover most first questions.

An open protocol that defines how AI applications connect to external tools and data in a uniform way. Clients talk to the model; MCP servers provide tools, resources and prompt templates. They exchange standard messages, so one server can be reused by many clients.