mcpField
A
agent-device

agent-device

github.com/callstackAgent workflows5.2k19.2k installsUpdated today
Repository

About agent-device

agent-device is a skill for driving app interfaces through the agent-device command line. It guides agents through interactive snapshots, actions, and verification, with selector and screenshot-based fallback guidance.

Key features

  1. 01Start sessions with an interactive snapshot
  2. 02Use refs and selectors for actions
  3. 03Verify outcomes with text and state checks
  4. 04Scroll directly to off-screen targets
  5. 05Use screenshots when accessibility data is unavailable

Use cases

  1. 01Navigate an app interface
  2. 02Enter and submit form data
  3. 03Find off-screen controls
  4. 04Recover when accessibility selectors are unavailable

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/callstack/agent-device --skill agent-device

After installing, type/agent-devicein Claude Code to confirm it loaded.

Details

From the project README

agent-device

For a normal app-driving task, start immediately. Do not probe first with --help, --version, devices, appstate, snapshot, or screenshot:

agent-device open <app> --foreground

That starts the session and returns the initial interactive snapshot with @refs.

Loop: act with press|click|fill|longpress <target> ... --settle, scroll <direction> --settle, or back --settle; continue from the printed diff, verify the named expectation (wait text "...", is, get, or find), then run agent-device close.

Reaching an off-screen target is one command, not a scroll-and-check loop: scroll down --until <selector> scrolls until that element is on screen, and scroll bottom runs to the end of the content. Repeated bare scroll down calls are the slow way to find something.

Copy refs byte-for-byte: @e12, @e12~s4 — keep the @ and any ~sN. Prefer current refs, then id/label/role selectors; coordinates are a last resort. If snapshot reports sparse/AX-unavailable, its refs and selectors are invalid: run agent-device screenshot, inspect the image, use coordinates, then retry snapshot -i after navigating. Otherwise run snapshot -i only when the diff lacks the next target.

Error output includes corrective hints; follow them instead of re-planning. Only when the task is specialized (for example gestures, scripting, TV, macOS, remote, or debugging) or a command shape is unclear, run agent-device help <topic>. agent-device --help lists topics, but is not a startup step.

Comments

Markdown supported