mcpField
S
ScreenPilot

ScreenPilot

Local
github.com/MtehabsimOther55Updated 16w ago
Repository

About ScreenPilot

ScreenPilot is an MCP server that lets an LLM control a device's graphical interface through a screen automation toolkit. It covers screen capture, mouse and keyboard input, and element detection, and is intended for automation, teaching and experimentation.

Key features

  1. 01Screen capture and analysis
  2. 02Mouse clicking and positioning
  3. 03Keyboard typing, key presses and hotkeys
  4. 04Scrolling and element detection
  5. 05Multi-step action sequences

Use cases

  1. 01Automating repetitive graphical interface tasks
  2. 02Demonstrating interface interaction in teaching
  3. 03Waiting for on-screen elements before continuing actions
#automation#mcp-server

Configuration

Local server config is the same for every client. Copy it into your client’s MCP config file.

No install method listed yet

We haven’t collected a package or remote endpoint for this server. Follow the instructions in the project README.

Repository

Tools

No tool list yet

Local servers expose tools only once running on your machine, and the maintainer has not provided a tool list.

Details

From the project README

MseeP.ai Security Assessment Badge

Trust Score

ScreenPilot

MCP server to let LLM take full control on your device by providing screen automation toolkit for controlling and interacting with graphical user interfaces. Good for automation, education and having fun.

Main Features

  • 📷 Screen capture and analysis
  • 🖱️ Mouse control (clicking, positioning)
  • ⌨️ Keyboard input (typing, key presses, hotkeys)

watch demo

https://github.com/user-attachments/assets/c18380c0-b3dd-4b7c-925d-28ef205ca11f

Installation

  1. Install python 3.12
  2. Clone the repository: 
    git clone https://github.com/Mtehabsim/ScreenPilot.git
  3. create virtiual environment

python -m venv venv
  1. activate the env
venv\Scripts\activate
  1. Install the required packages: 
    pip install -r requirements.txt
  2. Open Claude AI desktop
  3. file -> settings -> developer -> edit config
  4. open config file and paste this
{
    "mcpServers": {
        "device-controll": {
            "command": "pathToEnv\\venv\\Scripts\\python.exe",
            "args": [
                "pathToProject\\ScreenPilot\\main.py"
            ]
        }
    }
}

  1. Replace     "pathToEnv\venv\Scripts\python.exe" → with the full path to your python.exe     "pathToProject\ScreenPilot\main.py" → with the full path to your main.py file

  2. Save the config file.

  3. Open Claude AI Desktop.

  4. Go to File → Exit

  5. You can now open Claude AI Desktop and enjoy ScreenPilot.

Available Tools
  • Screen Capture: Take screenshots and get screen information
  • Mouse Control: Move the mouse and perform clicks
  • Keyboard Actions: Type text, press keys, and use hotkey combinations
  • Scrolling: Scroll in different directions and to specific positions
  • Element Detection: Check if elements exist on screen and wait for them to appear
  • Action Sequences: Perform multiple actions in sequence

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

Comments

Markdown supported

New to MCP?

From what the protocol is to installing your first server, these answers cover most first questions.

An open protocol that defines how AI applications connect to external tools and data in a uniform way. Clients talk to the model; MCP servers provide tools, resources and prompt templates. They exchange standard messages, so one server can be reused by many clients.