win-mind

WinMind

A Windows MCP server that lets an AI see and drive any app through the accessibility tree - no screenshots, no mouse, no pixel guessing - plus 200+ native Windows tools.

Most computer-use agents look at the screen: take a screenshot, send thousands of image tokens to a model, guess coordinates, click, take another screenshot. WinMind reads what Windows already knows. Every app publishes its buttons, text boxes, lists and text to UI Automation, the accessibility layer screen readers like Narrator and NVDA use. WinMind turns that into a compact, numbered tree an AI can read in one call and act on in the next - even when the app is behind other windows.

WhatsApp  (WhatsApp.Root.exe, window)
├─ [6] banner
│  ├─ [7] button "Chats" (42) checked [toggle]
│  ├─ [8] button "Calls" (4) unchecked [toggle]
├─ [20] textbox "Search or start a new chat" [set_value]
├─ [21] tabs "chat-list-filters" (4)
│  ├─ [22] tab "All" selected [select]
│  ├─ [23] tab "Unread 42" [select]
├─ [26] grid "Chat list" (9)
│  ├─ [29] row "Alex Yesterday Photo" [click]
│  ├─ [30] row "Study Group 3 unread messages Yesterday" [click]
└─ [37] button "Send document" [click]

ui_act(ref=30) opens that chat. ui_act(name="Play", title_contains="spotify") plays music. No screenshot was taken.

Why it's different

	Screenshot agents	WinMind
What the model gets	images (thousands of tokens each)	a text tree with real names, states and actions
Speed per look	seconds (capture + vision model)	0.1-0.3 s for a whole app (one cross-process call)
Clicking	pixel coordinates, moves your mouse	the app's own accessibility actions - your mouse never moves
Apps behind other windows	must bring them to front	reads and acts in the background
Text in editors/documents	OCR	exact text from the app (TextPattern)

Things WinMind handles that trip up simpler UI Automation tools (details in docs/ARCHITECTURE.md):

• Chromium-based apps behind other windows (Discord, Spotify, Steam, Teams, VS Code) expose no content while covered - Chromium's occlusion tracking pauses them. WinMind wakes them properly once; the tree then stays readable in the background. It never pulls a window over a full-screen game.
• WinUI 3 apps that host WebView2 (WhatsApp) loop back into their own web content - naive scans fetch the same elements thousands of times. WinMind splits and de-duplicates: 7.3 s -> 0.25 s.
• Web clicks that silently do nothing: pressing a list item whose click handler sits on an inner link. WinMind performs the default action of the element that really owns the click - without moving the mouse.
• Buttons and text in one tree: editor/document content comes through next to the controls.
• Finds apps by process, not just title (Spotify's title is the song playing), including apps hidden in the tray.
• Token budget: 222 tool definitions cost ~32k tokens (143 per tool); long outputs are trimmed with a note saying what was cut; errors are one actionable line, not tracebacks.

What's in the box (222 tools)

Area	Examples
UI automation	ui_tree, ui_map, ui_act, ui_read, ui_find_text, ui_element_at, ui_focused
Health & diagnostics	full_health_scan, why_is_my_pc_slow, smart_disk_health, battery_health, reliability_records
Hardware & gaming	CPU/GPU metrics, temperatures, fans, FPS / frame times / stutter / GPU-bottleneck (PresentMon)
Live OS tracing	ETW process / DNS / file / network events
Network	why_no_internet, port_process_map, whats_on_port, DNS cache, Wi-Fi scan
Privacy & security	whats_using_my_webcam / _mic, Defender, BitLocker, USB history, startup entries
Windows & desktops	windows, virtual desktops, saved workspaces
Files, processes, services, registry	with guards against system folders, critical keys and core OS processes
Display, audio, media, speech	per-app volume, now-playing + media control, brightness, night light, TTS
Screen capture (explicit only)	capture_window_image renders one window even when it's covered

Full list with one line per tool: TOOLS.md. An AI can also call tool_guide for a categorized index.

Quick start

One-click: Claude Desktop

Download winmind-mcp-<version>.mcpb from Releases and open it - Claude Desktop installs it like a browser extension (it needs uv, which sets up the Python dependencies for you). The bundle includes the native helpers.

Any MCP client (Claude Code, Cursor, VS Code, local models...)

Requirements: Windows 10/11, Python 3.10+.

git clone https://github.com/jhonpork1233-beep/win-mind
cd win-mind
pip install -e .

Add it to your MCP client. Use the full path to Python so a changed PATH can never break it:

// Claude Desktop / Claude Code (~/.claude.json "mcpServers") / any MCP client
{
  "winmind": {
    "command": "C:\\Path\\To\\Python\\python.exe",
    "args": ["-m", "winmind"]
  }
}

Claude Code shortcut: claude mcp add winmind -- "C:\Path\To\Python\python.exe" -m winmind

Then ask things like "what's in my WhatsApp unread chats?", "play my liked songs on Spotify", "why is my PC slow?", "screenshot Notepad without switching windows".

Optional native helpers

The core (UI engine, files, processes, registry, health, network...) is pure Python. Some tools use small native helpers - live event hooks, ETW, window streaming, virtual desktops, the WinRT broker (media, notifications), HUD overlays, PresentMon. Build them from source in native/:

powershell -ExecutionPolicy Bypass -File scripts\build_all.ps1

(Visual Studio 2022 Build Tools with the C++ workload and Windows SDK; MinGW g++ for winmind_native.dll.) Tools whose helper isn't built say so and name the script to run.

Using it efficiently (for AI clients and prompt writers)

• Start with ui_tree (structure) or ui_read (just the text) - not screenshots.
• Act by name when it's unambiguous: ui_act(name="Search", title_contains="spotify"); by ref otherwise.
• Prefer summary tools (full_health_scan, system_health_summary, windows_summary) over raw listings (appx_packages, service_list, port_process_map).
• Every tool accepts detail=true (no trimming), compact=true (short keys).
• Clients with deferred tool loading (e.g. Claude Code's tool search) only load the definitions they use.

Safety

WinMind runs with your user's rights and can change your system - that's its purpose. Read SECURITY.md.

• Every tool carries MCP risk annotations (readOnlyHint, destructiveHint) so clients can ask before risky calls; 16 tools are marked destructive.
• Built-in guards: file tools refuse Windows/System32/Program Files and drive roots; registry writes/deletes refuse critical keys; process tools refuse core OS processes.
• The UI engine never takes screenshots and never moves the mouse. Screen capture exists only as explicit tools.
• No telemetry. Everything WinMind writes stays in %LOCALAPPDATA%\WinMind (override with WINMIND_HOME).
• Run it non-elevated unless you need admin-only tools.

Testing

pip install -e .[dev]
python tests\audit_tools.py tests\audit.json     # calls every safe tool, sandbox-tests the write tools
python tests\audit_summary.py tests\audit.json

The audit only writes to a temp folder, the throwaway registry key HKCU\Software\WinMindAudit, a process it starts itself and its own Notepad window. Current results: 102/109 read-only tools OK (the rest need admin rights, a running session or an open Office document), 37/37 sandbox and guard tests pass.

Roadmap

• Java Access Bridge (Java/Swing apps), IAccessible2 text for Firefox/LibreOffice
• Stable selectors and recorded workflows
• Live UI event stream (tree deltas instead of re-reading)
• UI Automation Remote Operations for even faster trees
• One-click installs: Claude Desktop extension, Windows agent connector (On-Device Registry)
• Public head-to-head benchmark against other Windows MCP servers

License

Apache-2.0. Third-party components: THIRD_PARTY_NOTICES.md.