tokennuke

TokenNuke

Intelligent code indexing MCP server. 15 tools, 10 languages, tree-sitter AST extraction, hybrid search (FTS5 + vector), call graphs, remote repo indexing, incremental indexing.

Save 99% of tokens — get exact function source via byte-offset seek instead of reading entire files.

Formerly codemunch-pro. Same code, better name.

Install

pip install tokennuke

Quick Start

Claude Desktop / Cline

Add to your MCP client config:

{
  "mcpServers": {
    "tokennuke": {
      "command": "tokennuke"
    }
  }
}

HTTP Server

tokennuke --transport streamable-http --port 5002

15 MCP Tools

Tool	Description
index_folder	Index a local directory (incremental, SHA-256 based)
index_repo	Index a GitHub/GitLab repo (tarball download, no git needed)
list_repos	List all indexed repositories with stats
invalidate_cache	Force re-index a repository
file_tree	Get directory tree with file counts
file_outline	List symbols in a single file
repo_outline	List all symbols in repo (summary)
get_symbol	Get full source of one symbol (O(1) byte seek)
get_symbols	Batch get multiple symbols
search_symbols	Hybrid search (FTS5 + vector RRF)
search_text	Full-text search in file contents
get_callees	What does this function call?
get_callers	Who calls this function?
diff_symbols	What changed since last index? (PR review)
dependency_map	What does this file depend on? What depends on it?

10 Languages

Python, JavaScript, TypeScript, Go, Rust, Java, C, C++, C#, Ruby

All via tree-sitter-language-pack — zero compilation, pre-built binaries.

Key Features

O(1) Symbol Retrieval

Every symbol stores its byte offset and length. get_symbol seeks directly to the function source — no reading entire files. A 200-byte function from a 40KB file = 99.5% token savings.

Incremental Indexing

Files are hashed (SHA-256). Only changed files are re-parsed. Re-indexing a 10K file repo after changing one file takes milliseconds.

Hybrid Search (FTS5 + Vector)

Combines BM25 keyword matching with semantic vector similarity using Reciprocal Rank Fusion. Search "authentication middleware" and find auth_middleware, verify_token, and login_handler.

Call Graphs

Traces function calls through the AST. get_callees("main") shows what main calls. get_callers("authenticate") shows who calls authenticate. Supports depth traversal.

Remote Repo Indexing

Index any public GitHub or GitLab repo by URL — no git binary needed. Downloads the tarball via API, extracts, and indexes. Cached locally with SHA-based freshness checks. Supports private repos with auth tokens and sparse paths.

Full-Text Content Search

Search raw file contents — string literals, TODO comments, config values, error messages. Not just symbol names.

How It Works

1. Parse — tree-sitter builds an AST for each source file
2. Extract — Walk AST to find functions, classes, methods, types, interfaces
3. Store — SQLite database per repo with FTS5 virtual tables
4. Embed — FastEmbed (ONNX, CPU-only) generates 384-dim vectors for semantic search
5. Graph — Call expressions extracted from function bodies, edges stored and resolved
6. Serve — FastMCP exposes 15 tools via stdio or HTTP

Architecture

~/.tokennuke/
├── myproject_a1b2c3d4e5f6.db    # Per-repo SQLite database
├── otherproject_7890abcdef.db
└── ...

Each DB contains:
├── files          # Indexed files with SHA-256 hashes
├── symbols        # Functions, classes, methods, types
├── symbols_fts    # FTS5 full-text search index
├── symbols_vec    # sqlite-vec 384-dim vector index
├── call_edges     # Call graph (caller → callee)
└── file_content_fts  # Raw file content search

Use Cases

• AI Coding Agents: Give your agent surgical access to codebases without burning context
• Code Review: Find all callers of a function before changing its signature
• Onboarding: Search symbols semantically — "where is error handling?" finds relevant code
• Refactoring: Map call graphs before moving functions between modules
• Documentation: Extract all public APIs with signatures and docstrings

License

MIT