Darwin RAG

Darwin RAG

Search your own documents with any AI agent, privately, on your own machine.

Darwin is a local-first RAG engine that ingests documents, indexes them with BM25 + dense embeddings, and exposes search through an MCP server. Point Claude, Cursor, or any MCP client at it and your agent can search your PDFs, notes, spreadsheets, and web pages without sending them to a cloud index.

One-time internet is needed to download models during setup. After that, search and indexing run fully offline.

• Ingestion: PDF, Markdown, HTML, images (OCR), CSV, Excel, ODS, URLs
• Indexing: BM25 keyword + dense embedding hybrid index with configurable chunking strategies
• Search: Hybrid, semantic, or keyword retrieval with reranking, diversity rerank, and structural penalties
• Generation: LLM-backed answer synthesis via LiteLLM (OpenAI, Anthropic, Gemini, etc.)
• Observability: Structured logging with per-session history, queryable via MCP
• Isolation: Multiple independent stores for tenant/project separation
• Deployment: stdio (AI agent subprocess), SSE, or Streamable HTTP; Docker-ready
• Local-first: Everything runs locally, fully offline-capable after setup

Built by BrightDotDev.

---

Why Darwin?

• Private by default. Your documents stay on your machine.
• Works with your agent. Standard MCP server: Claude Desktop, Cursor, and other MCP clients.
• Search that fits the question. Keyword for exact terms, semantic for meaning, hybrid when unsure.
• Light to start. The required preset downloads just one embedding model.

---

Quick Start

1. Install

pip install darwin-rag

2. Set up models

# Interactive: detects hardware, pick your models
darwin-admin setup interactive

# Or one-shot (embedding-only, no prompts)
darwin-admin setup --preset required

3. Start the MCP server & connect

# stdio mode: for AI agent subprocess (Claude Desktop, Cursor, etc.)
darwin mcp

# Or HTTP mode: for remote clients
darwin mcp --http --port 8765

Configure your MCP client:

{
  "mcpServers": {
    "darwin": {
      "command": "darwin",
      "args": ["mcp"],
      "env": {
        "OPENAI_API_KEY": "sk-..."  // At least one LLM provider key
      }
    }
  }
}

Or generate config automatically:

darwin config claude          # Claude Desktop config
darwin config cursor          # Cursor config
darwin config all --copy      # All clients + copy to clipboard

4. Try it

Ask your agent: "Ingest ./my-notes with Darwin, then find everything I wrote about ."

---

Use it as an agent skill

Darwin ships with a skill (skills/darwin-rag/SKILL.md) that tells agents when and how to use the MCP tools: check what's indexed, ingest, pick a search mode, and answer from sources. Copy the darwin-rag folder into your agent's skills directory (for Claude Code: ~/.claude/skills/).

---

MCP Tools

Tool	Description
search_darwin	Query the knowledge base with hybrid/semantic/keyword search
search_lists	Search structured data (CSVs, JSON arrays) by field values
get_search_results	List saved search results
get_search_result_by_id	Load a saved search result by filename
get_schema	Inspect schemas for structured files (keys, types, record counts)
run_pipeline	Ingest + index documents from a path or URL
purge_artifacts	Delete pipeline artifacts for specific files
create_store	Create a new isolated data store
list_files	List all tracked files with pipeline status
file_status	Detailed status for a single file across all stages
get_logs	Query session logs (oldest first, INFO excluded)

Full documentation: docs/mcp.md

---

Remote / HTTP Mode

Start the server on a network-accessible endpoint:

# SSE transport (legacy)
darwin mcp --sse --host 0.0.0.0 --port 8765

# Streamable HTTP transport (recommended for production)
darwin mcp --http --host 0.0.0.0 --port 8765

Configure your MCP client with the URL:

{
  "mcpServers": {
    "darwin": {
      "url": "http://your-host:8765/mcp"  // or /sse for SSE mode
    }
  }
}

---

Environment Variables

Variable	Required	Description
OPENAI_API_KEY	No*	OpenAI provider key
ANTHROPIC_API_KEY	No*	Anthropic provider key
GEMINI_API_KEY	No*	Google Gemini provider key
MISTRAL_API_KEY	No*	Mistral AI provider key
GROQ_API_KEY	No*	Groq provider key
COHERE_API_KEY	No*	Cohere provider key
TOGETHER_API_KEY	No*	Together AI provider key
OPENROUTER_API_KEY	No*	OpenRouter provider key
DEEPSEEK_API_KEY	No*	DeepSeek provider key
DARWIN_BASE_DIR	No	Override the base data directory
NO_COLOR	No	Set to any value to disable ANSI color output

* At least one LLM provider key is required for answer generation. Search/indexing works without any.

---

Setup Details

Command	What it does
darwin-admin setup interactive	Guided setup: detect hardware, choose models
darwin-admin setup --preset required	Download embedding model only (fastest)
darwin-admin setup --preset recommended	Embedding + reranker + OCR models
darwin-admin setup logging	Reconfigure logging only
darwin-admin setup validate	Validate current setup

See docs/setup.md for the full walkthrough including Docker, from-source install, and API key configuration.

---

CLI Reference

darwin: User CLI

Command	Description
darwin mcp	Start MCP server (stdio, --sse or --http for network)
darwin config [client]	Generate MCP client config

darwin-admin: Power-user CLI

Command	Description
darwin-admin setup	Setup models, logging, and configuration
darwin-admin status	System status overview
darwin-admin models	Model registry: list, install, switch, keys
darwin-admin store	Data store: status, files, audit, health, repair
darwin-admin pipeline	Ingestion pipeline: run, ingest, index, purge
darwin-admin search	Interactive search
darwin-admin logs	Structured log viewer and management
darwin-admin system	System information
darwin-admin uninstall	Remove Darwin data and configuration

See docs/admin.md for the full command reference.

---

Python API

For embedding darwin-rag as a library in your own app:

from core import Darwin

d = Darwin()
d.ingest("./papers", recursive=True)
results = d.search("what is this paper about")
records = d.search_records(filters={"status": "active"})

Full reference: docs/api.md

---

Documentation

Doc	What
setup.md	Full setup walkthrough
mcp.md	MCP server, tools, resources, transports
admin.md	Admin CLI reference
architecture.md	For developers and contributors
pipeline.md	Ingestion & indexing
retrieval.md	Search engine
storage.md	DarwinStore
models.md	Model registry & inference
logger.md	Structured logging
orchestrators.md	High-level business logic
api.md	Python API (Darwin class)

---

Contributing

Found a bug? Want to add something? You're welcome here.

• Issues: open one at github.com/BrightDotDev/DARWIN/issues
• Code: fork, branch, PR. Keep it minimal.
• AI-generated code is fine, but you own what you ship. Test it before submitting.

Read CONTRIBUTING.md for the full guidelines.

---

License

MIT. See LICENSE.

Core architecture and implementation by BrightDotDev.