mcpField
A
agent-desktop

agent-desktop

github.com/lahfirAgent workflows1.8k186 installsUpdated 1d ago
Repository

About agent-desktop

agent-desktop is a CLI skill for AI agents to inspect and operate desktop apps through the operating system's accessibility tree. It provides structured JSON results and an observe, act, and verify workflow, with headless semantic actions by default.

Key features

  1. 01Inspect apps through accessibility trees
  2. 02Find and operate UI elements by reference
  3. 03Verify changes with fresh snapshots
  4. 04Return JSON envelopes for commands
  5. 05Use headless semantic actions by default

Use cases

  1. 01Automate desktop workflows through an AI agent
  2. 02Inspect a dialog before interacting with it
  3. 03Recover safely when a UI action result is uncertain

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/lahfir/agent-desktop --skill agent-desktop

After installing, type/agent-desktopin Claude Code to confirm it loaded.

Bundled files

References and scripts installed along with SKILL.md.

references/
commands-interaction.md
commands-observation.md
commands-system.md
workflows.md

Details

From the project README

agent-desktop

agent-desktop is a CLI that lets an AI agent observe and operate desktop apps through the OS accessibility tree. It is a tool, not an agent. Every command prints one JSON envelope (ok, data or error). The observe, act and verify loop runs in you.

Shell syntax

Examples use a POSIX shell. In PowerShell, quote every ref ('@s8f3k2p9:e1'), because a bare @token is the splatting operator. Use $env:NAME = 'value' instead of export, and | ConvertFrom-Json instead of $(...) and jq. The Windows skill has the PowerShell forms.

First run

  1. Run agent-desktop status. It reports the platform, permissions and supported_surfaces.
  2. Run agent-desktop skills get platform. It loads the skill for the OS the binary runs on (macOS, Windows or Linux).
  3. For flags and syntax, run agent-desktop <command> --help. This skill does not repeat flag lists.

The loop

Use find when you know the role or exact name. Otherwise map the app first and drill into one region.

agent-desktop snapshot --skeleton --app "Notes" -i --compact   # shallow map, children_count on cut branches
agent-desktop snapshot --root @s8f3k2p9:e3 -i --compact        # drill into one region
agent-desktop click @s8f3k2p9:e12                              # act on a qualified ref
agent-desktop snapshot --root @s8f3k2p9:e3 -i --compact        # verify: re-read the same region
  • A small app needs no skeleton. Take a full snapshot -i.
  • Never skeleton a surface (--surface menu, sheet, alert). A surface is already focused.
  • An action that opens a menu, sheet or alert lists it in data.surfaces. Target that overlay next.
  • After an async step, use wait (see commands-system.md). A fixed sleep is a race.

Refs

  • Output refs are qualified: @<snapshot_id>:e<N>, for example @s8f3k2p9:e1. Pass them to later commands as they are.
  • Refs are valid for one snapshot. After any UI change, re-drill the affected region or take a new snapshot.
  • Only addressable elements get refs. Static text and inert containers do not.
  • STALE_REF: the ref no longer matches the live UI. Take a fresh snapshot and use the new ref.
  • AMBIGUOUS_TARGET: more than one live element matches. Narrow the target with find --role --name --exact or --root, then use the new ref.
  • SNAPSHOT_NOT_FOUND: the snapshot id expired. Snapshot again.

Delivery and retry

Action results carry data.disposition and errors carry error.disposition, each with delivery and retry. Observation commands such as snapshot have none on success; their errors, and every INVALID_ARGS refusal, carry not_delivered and retry: safe.

  • Retry or substitute an action only when disposition.retry is safe. Only not_delivered is safe.
  • Any other delivery value means the action may have landed. A repeat can click, type or submit twice, and toggle can undo itself.
  • On a result that is not safe, read the live state (get, is, or a fresh snapshot) and decide from that. details.post_state helps when present.
  • Branch on error.code and error.disposition. Message and suggestion text can change.
  • details.retryable: false means waiting cannot change the outcome. Try a different approach.

Headless and headed

Ref actions are headless by default. They use semantic accessibility calls and do not take focus, move the cursor, synthesize keys or use the clipboard. If no semantic path exists, the action fails closed. Use the global --headed flag only when the user explicitly permits physical input, such as double-click, hover, drag, or focus. Never switch to headed mode to recover from a headless failure. Raw coordinates (--xy) never imply focus. Which actions need --headed differs by OS: read the platform skill.

References

Load one only when you need it: agent-desktop skills get desktop <reference>.

ReferenceOpen it when
commands-observationYou need snapshot, find, get, is, screenshot, list-surfaces semantics, partial trees, or the --root rules
commands-interactionYou choose between click, type, set-value, select, toggle, scroll, press and mouse commands, or you debug an action error
commands-systemYou launch or close apps, manage windows, use the clipboard, wait, batch, sessions, multi-agent cursors, tracing, or load skills
workflowsYou want worked patterns (forms, menus, dialogs, waiting for a new window, scroll-find, drag) and the anti-patterns list

Comments

Markdown supported