mcpField
T
token-report

token-report

github.com/codewithmukeshAgent workflows829387 installsUpdated 1w ago
Repository

About token-report

A skill for measuring token savings from the Roslyn MCP server, using an in-session tool and a cross-repository statistics CLI. It notes that telemetry is opt-in and that reported estimates have limitations.

Key features

  1. 01Summarize token savings from MCP tools
  2. 02Show calls and savings by tool
  3. 03Aggregate repositories and export multiple formats
  4. 04Calibrate the token estimate divisor
  5. 05Report collection status and measurement caveats

Use cases

  1. 01Assess whether an MCP server justifies its context cost
  2. 02Find tools that receive no calls
  3. 03Support token-efficiency claims with checkable data
  4. 04Compare tool adoption across repositories

MCP support

In a Roslyn MCP server session, call get_token_savings for repository-level statistics; the CLI aggregates across repositories. Telemetry is off by default and, when enabled, records are stored locally.

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/codewithmukesh/dotnet-claude-kit --skill token-report

After installing, type/token-reportin Claude Code to confirm it loaded.

Details

From the project README

/token-report — Roslyn Navigator token savings

What

Reports what the Roslyn MCP server has actually saved, measured rather than asserted. Two surfaces:

ScopeSurfaceUse when
This repository, mid-sessionget_token_savings MCP tool"how are we doing here?"
All repositories, aggregateCWM.RoslynNavigator.Stats CLIproducing a public number

When

  • Deciding whether the MCP server earns its context cost
  • Finding tools nobody calls — a description problem, or a tool that should not exist
  • Backing a claim about token efficiency with something checkable

Prerequisite

Collection is opt-in and off by default. Nothing was recorded unless the server was started with it enabled:

CWM_ROSLYN_TELEMETRY=on cwm-roslyn-navigator --solution MySolution.sln

Nothing is transmitted. Records are written under %LOCALAPPDATA%/cwm-roslyn-navigator/ (or the XDG equivalent), outside the repository. Delete that directory to erase everything.

Adoption rate is read from Claude Code transcripts and needs no opt-in, so it is available even when telemetry was never enabled.

How

1 — This repository, mid-session. Call the get_token_savings MCP tool. Returns totals, a per-tool leaderboard, separate zero-result / not-ready / error counts, and the methodology. If it reports Enabled: false, telemetry was never turned on — say so rather than reporting zero as if it were a measurement.

2 — All repositories.

cd mcp/CWM.RoslynNavigator
dotnet run --project stats/CWM.RoslynNavigator.Stats.csproj            # table
dotnet run --project stats/CWM.RoslynNavigator.Stats.csproj -- --markdown
dotnet run --project stats/CWM.RoslynNavigator.Stats.csproj -- --json
dotnet run --project stats/CWM.RoslynNavigator.Stats.csproj -- --repo myapp

3 — Interpret it honestly. Three things must be said whenever a number is quoted:

  • The divisor is uncalibrated. 3.5 bytes/token is an estimate, and it scales every figure linearly. If DivisorCalibrated is false, the number is indicative, not precise.
  • The baseline understates. It counts only the files an answer came from, ignoring the wrong files a text search would have opened first.
  • Adoption is biased upward. Its denominator counts only Read/Grep/Glob calls that demonstrably targeted C#; ambiguous searches are excluded.

4 — Act on the leaderboard. A tool with no calls is the actionable finding. Low adoption in a configured repository usually means a tool description is not attracting the work it should — more useful than the savings figure itself.

Calibrating the divisor

Turns the one estimated input into a measurement. Free, manual, and never run by the server:

dotnet run --project stats/CWM.RoslynNavigator.Stats.csproj -- calibrate \
  --api-key $ANTHROPIC_API_KEY --path /path/to/dotnet/repo --sample 200

Samples C# files, gets exact counts from the Anthropic count_tokens endpoint, and regresses bytes against tokens. Apply the result to TokenEstimator in the server and record the sample size and R² next to the constant so the figure stays checkable.

Example

  Repositories        3  (1 with telemetry)
  Tool calls          412
  Tokens saved        1,284,000
  Adoption            68%

  Tool leaderboard
  tool                  calls      saved   p50 ms  0-result
  find_references         180    612,000       84        4%
  find_symbol             150    431,000       22        2%

  Never called (2): get_nuget_packages, resolve_stack_trace
  Divisor: 3.5 bytes/token — UNCALIBRATED estimate.

When the server was never invoked, the report says so plainly instead of printing zeros — "no calls" and "no savings" are different findings.

Related

  • /verify — quality gates before a PR
  • /health-check — codebase health grades

Comments

Markdown supported