flutter-mcp-toolkit-control

When to use

Use this skill when you need to drive a running Flutter app as a user would:

• Tap buttons, icons, list items, or any interactive widget.
• Type text into fields, submit forms, clear inputs.
• Scroll or swipe to reveal off-screen content.
• Navigate between routes programmatically (push, pop, popUntil).
• Dismiss dialogs and bottom sheets.
• Give a widget keyboard focus, then press keys (Enter, Escape, Tab, arrows, ASCII chars).
• Hot-reload or hot-restart after editing Dart source files.
• Combine reload + screenshot + semantics in one round-trip for fast iteration.

Selectors

Most interaction tools target a widget by ref — a short string like "s_0" returned by semantic_snapshot. For visible widgets, call semantic_snapshot, scan the returned nodes, find the right ref, then pass it. Narrow the node set with identifierPrefix or subtreeOf (a ref or an identifier), and cut every node down to the keys you read with fields; refs are those of the full tree either way. For off-screen targets with stable semantics text or identifier, use reveal_search; it performs a bounded snapshot → match → scroll loop and returns a fresh ref/snapshotId.

Snapshot node keys to scan:

Want to find	Scan field	Example value
By semantics identifier	identifier	"nav.tasks"
By visible label / text	label	"Login"
By value or hint	value / hint	"[email protected]"
By semantic role	type	"button"
By what it accepts	actions	["tap"]

Example — find the "Login" button ref:

semantic_snapshot()
→ nodes: [{ref:"s_0", label:"Login", actions:["tap"]}, ...]
tap_widget(ref: "s_0")

Pass snapshotId (from the snapshot response) to any interaction call. If the tree has changed, the call returns stale_snapshot with both IDs so you know to re-snapshot. Refs are only valid against the most recent snapshot.

Recipes

Tap a widget by text

semantic_snapshot()
→ find node where label == "Submit" → ref "s_3"
tap_widget(ref: "s_3", snapshotId: <id>)

Fill a login form

semantic_snapshot() → email ref "s_1", password ref "s_2"
fill_form(fields: [{ref:"s_1", text:"[email protected]"}, {ref:"s_2", text:"secret"}], snapshotId: <id>)
→ one round-trip; stops on first failure

Scroll to find an item

scroll(direction: "down", distance: 300)
semantic_snapshot() → item now visible → ref "s_5"
tap_widget(ref: "s_5")

Reveal an off-screen field by identifier

reveal_search(query: "greeting_input_field", matchBy: "identifier", direction: "down", maxAttempts: 4)
→ returns ref "s_14", snapshotId 2, match, attempts trace
enter_text(ref: "s_14", snapshotId: 2, text: "hello")

Wait for a widget to appear

wait_for(predicate: {kind: "text", text: "Welcome"}, timeoutMs: 8000)
→ returns fresh snapshot when text appears
tap_widget(ref: <ref from wait_for snapshot>)

Navigate to a route

navigate(action: "push", route: "/settings", arguments: {tab: "account"})
semantic_snapshot() → fresh refs in the new screen

Hot reload after a code change

hot_reload_and_capture()
→ screenshot + semantic snapshot + errors in one call

Press the back hardware button

press_key has no Back key. Use navigate(action: "pop") for Navigator pop; handle_dialog(action: "dismiss") for dialogs; press_key(key: "Escape") on desktop.

navigate(action: "pop")

Tool reference

tap_widget

Tap a widget by ref. ref • string • required. snapshotId • integer • optional. connection • object • optional.

{"name": "tap_widget", "arguments": {"ref": "s_3", "snapshotId": 7}}

Returns: {"via": "semantic_action", "ref": "s_3"} — Failures: stale_snapshot, ref_not_found

long_press

Long-press a widget by ref. ref • string • required. snapshotId • integer • optional. connection • object • optional.

{"name": "long_press", "arguments": {"ref": "s_2"}}

Returns: {"via": "semantic_action"} — Failures: stale_snapshot, ref_not_found

enter_text

Enter text into a text field; taps to focus before typing. ref • string • required. text • string • required. snapshotId • integer • optional. connection • object • optional.

{"name": "enter_text", "arguments": {"ref": "s_1", "text": "[email protected]"}}

Returns: {"via": "editable_state", "verified": true, "appliedText": "..."}. A write the field kept nothing of fails with text_not_applied, appliedText (empty) and restoredText (the value put back). Failures: stale_snapshot, ref_not_found, text_not_applied

reveal_search

Find a semantic target that may be off-screen. query • string • required. matchBy • string • optional (text|identifier|label|value|hint, default text). direction • string • optional (up|down|left|right, default down). maxAttempts • integer • optional • max 10. distance • number • optional. connection • object • optional.

{"name": "reveal_search", "arguments": {"query": "greeting_input_field", "matchBy": "identifier", "direction": "down", "maxAttempts": 4}}

Returns: {"ref": "s_14", "snapshotId": 2, "match": {...}, "attempts": [...]} — Failures: missing_query, target_not_found, scroll_blocked, target_not_actionable. identifier matches whole and case-sensitively; the other modes are case-insensitive substring tests. A miss by identifier carries identifiersSeen (how many the searched screens published) and nearIdentifiers — the closest of them, so panel.tab not found comes back with panel.tab.overview / panel.tab.jobs to pass instead.

fill_form

Batch text entry: fills multiple fields in one call. Stops on first failure. snapshotId validated on first field only. fields • array of {ref, text} • required. snapshotId • integer • optional. connection • object • optional.

{"name": "fill_form", "arguments": {"fields": [{"ref":"s_1","text":"user"},{"ref":"s_2","text":"pass"}], "snapshotId": 5}}

Returns: {"filled": 2} — Failures: stale_snapshot, ref_not_found

scroll

Scroll to reveal content. "down" reveals content below (finger swipes up). direction • string • required (up|down|left|right). ref • string • optional — the list to scroll, or any node inside it; without it the list under the screen centre scrolls. distance • number • optional • default 300, honoured exactly where the list takes an offset, otherwise rounded to a viewport page. snapshotId • integer • optional. connection • object • optional.

{"name": "scroll", "arguments": {"direction": "down", "ref": "s_0", "distance": 500}}

Returns: {"via": "semantic_action", "scrollBefore": 0.0, "scrollAfter": 500.0, "distance": 500.0}. scrollBefore/scrollAfter are forwarded whenever measurable; distance is present on exact-offset and pointer-scroll paths. A node that advertises the scroll action without a scroll position (custom Semantics(onScrollUp: ...)) takes it and returns success: true, verified: false, unmeasured: true. Failures: ref_not_found, stale_snapshot, no_scrollable_at_point, no_scroll_movement, unsupported_scroll_action, semantics_owner_unavailable

swipe

High-velocity fling. Same direction model as scroll. Always Tier 2 pointer events. direction • string • required. ref • string • optional. distance • number • optional • default 300. snapshotId • integer • optional. connection • object • optional.

{"name": "swipe", "arguments": {"direction": "left", "ref": "s_4"}}

Returns: {"via": "pointer_events", "scrollBefore": 0.0, "scrollAfter": 420.0} when movement is measurable. When finite scroll offsets are unavailable, a dispatched gesture returns success: true, verified: false, and measurementReason. Failures: ref_not_found, stale_snapshot, no_scroll_movement, web_gesture_not_supported

drag

Drag from one widget to another. Always Tier 2. fromRef • string • required. toRef • string • required. snapshotId • integer • optional. connection • object • optional.

{"name": "drag", "arguments": {"fromRef": "s_2", "toRef": "s_7"}}

Returns: {"via": "pointer_events"} — Failures: ref_not_found, web_gesture_not_supported

hover

Synthesize a mouse hover. Desktop/web only — no hover concept on mobile. ref • string • required. snapshotId • integer • optional. connection • object • optional.

{"name": "hover", "arguments": {"ref": "s_5"}}

Returns: {"via": "pointer_events"} — Failures: ref_not_found, platform error on mobile

focus_widget

Give keyboard focus to a widget, so the press_key that follows reaches it. Uses the node's semantic focus action when the snapshot lists one, otherwise the focusable widget inside the ref's bounds (the only route on iOS, where Flutter exposes no focus action). Leaves a parked hover in place. ref • string • required. snapshotId • integer • optional. connection • object • optional.

{"name": "focus_widget", "arguments": {"ref": "s_5"}}

Returns: {"via": "semantic_action" | "focus_node", "verified": true, "verifiedBy": "semantics_flag" | "focus_node", "focusMoved": bool} — success: true means the target reports focus after the framework caught up; focusMoved: false means it already held it. On the focus-node route, focusedNow names the control inside the ref that actually took focus. Failures: ref_not_found, stale_ref, stale_snapshot, target_disabled, focus_not_exposed (nothing focusable behind the ref), focus_refused (the request went through but the target does not report focus; focusedNow says where it sits)

press_key

Synthesize key press (down+up). Accepted: Enter Escape Tab Backspace Delete Space ArrowUp ArrowDown ArrowLeft ArrowRight plus single ASCII (a-z 0-9). key • string • required. ctrl/shift/alt/meta • boolean • optional • default false. connection • object • optional.

{"name": "press_key", "arguments": {"key": "Enter"}}

Returns: {"key": "Enter", "handled": bool} — handled says whether either dispatch phase (hardware keyboard handlers or focus chain) claimed the main key-down event; false does not describe the key-up or modifier events. clearedStaleModifiers / releasedStuckModifiers name any modifier the call had to release — a modifier left held by an earlier keystroke makes every plain shortcut stop matching, so the call normalizes it and says so. Failures: unknown_key

wait_for

Wait for a UI predicate; returns fresh semantic snapshot. Predicates: {kind:"text",text} | {kind:"noText",text} | {kind:"time",ms} | {kind:"stable",stableWindowMs}. stable samples once per frame and matches after the semantics tree has remained unchanged for the requested wall time; the match reports stableFor.sampledFrames and stableFor.elapsedMs. stableWindowMs must be less than timeoutMs (default 5000); an impossible budget fails immediately with invalid_predicate. predicate • object • required. timeoutMs • integer • optional • default 5000 • max 30000. connection • object • optional.

{"name": "wait_for", "arguments": {"predicate": {"kind": "text", "text": "Dashboard"}, "timeoutMs": 8000}}

Returns: fresh semantic snapshot — Failures: timeout, invalid_predicate

navigate

Drive the registered Navigator. Requires MCPToolkitBinding.instance.navigatorKey = key in the app. action • string • required (push|pop|popUntil). route • string • required for push/popUntil. arguments • object • optional (for push). connection • object • optional.

{"name": "navigate", "arguments": {"action": "push", "route": "/profile", "arguments": {"userId": "42"}}}

Returns: {"action": "push", "route": "/profile"}. A generated unnamed route returns success: true, verified: false, and via: "stack_changed_unnamed_route" because the stack changed but its requested name cannot be checked. pop reads success off the stack and reports handled from maybePop: a page under PopScope(canPop: false) comes back handled: true, success: false, nothing_popped. Failures: navigator_not_configured, route_not_found

handle_dialog

Dismiss the topmost popup/dialog route. Only action: "dismiss" supported. Requires navigatorKey = key on MCPToolkitBinding.instance in the app. action • string • required (must be "dismiss"). connection • object • optional.

{"name": "handle_dialog", "arguments": {"action": "dismiss"}}

Returns: {"success": true, "handled": true, "routeType": "DialogRoute<void>"}. success is read off the route stack; handled is what maybePop answered, so a dialog under PopScope(canPop: false) comes back handled: true, success: false, dialog_declined_pop. Failures: navigator_not_registered, no_popup_route, dialog_declined_pop

hot_reload_flutter

Hot reload the app. Preserves state. force • boolean • optional • default false (reload even without source changes). connection • object • optional.

{"name": "hot_reload_flutter", "arguments": {}}

Returns: "Hot reload completed" + report JSON — Failures: vm_not_connected, compilation_error

hot_restart_flutter

Full restart. App state not preserved. No required params. connection • object • optional.

{"name": "hot_restart_flutter", "arguments": {}}

Returns: {"report": {"type": "Success", "success": true}} — Failures: vm_not_connected

hot_reload_and_capture

Hot reload then capture screenshot + semantics + errors in one call. compress • boolean • default true. includeSemantics • boolean • default true. includeErrors • boolean • default true. errorsCount • integer • default 4. connection • object • optional.

{"name": "hot_reload_and_capture", "arguments": {"includeErrors": true}}

Returns: screenshot (base64) + semantic snapshot + errors — Failures: vm_not_connected, compilation_error

Patterns

Always wait_for before tap_widget after navigation

After navigate(action: "push") the new route's widgets are not in the tree yet. Use wait_for with a text predicate to confirm the destination has rendered, then snapshot and act.

navigate(action: "push", route: "/checkout")
wait_for(predicate: {kind: "text", text: "Order Summary"}, timeoutMs: 5000)
semantic_snapshot() → tap target widgets

Prefer fill_form over multiple enter_text calls

Each enter_text is a separate VM round-trip. fill_form sends all field/text pairs in one call; snapshotId is checked once (on the first field). For any form with 2+ fields, always prefer fill_form.

# Avoid: 2 round-trips
enter_text(ref: "s_1", text: "Alice")
enter_text(ref: "s_2", text: "secret")

# Prefer: 1 round-trip
fill_form(fields: [{ref: "s_1", text: "Alice"}, {ref: "s_2", text: "secret"}])

After hot_reload_*, wait for the new tree before continuing

Hot reload completes asynchronously. Use wait_for(predicate: {kind:"stable", stableWindowMs:300}) to confirm the tree has settled before calling semantic_snapshot. Or use hot_reload_and_capture which returns a post-reload snapshot directly.

hot_reload_flutter()
wait_for(predicate: {kind: "stable", stableWindowMs: 300})
semantic_snapshot() → interact with reloaded widgets

# Or in one call (preferred):
hot_reload_and_capture() → screenshot + semantics + errors already post-reload