mcpField
A
ai-vision

ai-vision

github.com/httprunnerAgent workflows288 installsUpdated 35w ago
Repository

About ai-vision

A standalone CLI skill for sending screenshots to multimodal models for UI queries, assertions, and single-step action planning. It is intended for agents that handle execution and multi-step control separately through device drivers.

Key features

  1. 01Query UI elements from screenshots
  2. 02Assert conditions visible on screen
  3. 03Plan a normalized next action
  4. 04Configure model endpoint and credentials
  5. 05Output structured JSON

Use cases

  1. 01Locate a control in a screenshot
  2. 02Check whether a UI condition is met
  3. 03Get coordinates for the next interaction
  4. 04Feed model output into an external device driver

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/httprunner/skills --skill ai-vision

After installing, type/ai-visionin Claude Code to confirm it loaded.

Bundled files

References and scripts installed along with SKILL.md.

package-lock.json
package.json
tsconfig.json
agents/
openai.yaml
references/
doubao-api.md
scripts/
ai_vision.ts

Details

From the project README

AI Vision

Overview

This skill provides a standalone CLI to call multimodal models for UI querying, assertion, and single-step planning. It does not depend on device type; you supply a screenshot and receive structured output (coordinates, decisions, or next actions). Execution and multi-step loops are handled externally by agents using adb/hdc or other drivers. Prefer storing screenshots in ~/.eval/screenshots/ and add timestamps to avoid overwriting.

Path Convention

Canonical install and execution directory: ~/.agents/skills/ai-vision/. Run commands from this directory:

cd ~/.agents/skills/ai-vision

One-off (safe in scripts/loops from any working directory):

(cd ~/.agents/skills/ai-vision && npx tsx scripts/ai_vision.ts --help)

Model Configuration

Default Doubao configuration via environment variables:

  • ARK_BASE_URL (e.g. https://ark.cn-beijing.volces.com/api/v3)
  • ARK_API_KEY
  • ARK_MODEL_NAME

For non-Doubao providers, pass explicit flags:

  • --base-url, --api-key, --model

Default model if none provided: doubao-seed-1-6-vision-250815.

Script

Path: scripts/ai_vision.ts

Run with:

npx tsx scripts/ai_vision.ts --help

Log level (for troubleshooting raw model response):

npx tsx scripts/ai_vision.ts --log-level debug <command> [flags]

Output formatting:

  • When --log-json is set, logs are emitted as JSON.
  • Otherwise, the final result is pretty-printed JSON, and logs are colorized when TTY is available.
AIQuery
npx tsx scripts/ai_vision.ts query \
  --screenshot ~/.eval/screenshots/ui_YYYYMMDD_HHMMSS.png \
  --prompt "请识别屏幕上的‘搜索’按钮，并返回其坐标"
AIAssert
npx tsx scripts/ai_vision.ts assert \
  --screenshot ~/.eval/screenshots/ui_YYYYMMDD_HHMMSS.png \
  --prompt "当前页面包含搜索框"
plan-next (single-step planning)
npx tsx scripts/ai_vision.ts plan-next \
  --screenshot ~/.eval/screenshots/ui_YYYYMMDD_HHMMSS.png \
  --prompt "点击放大镜图标进入搜索页"

Output Notes

  • plan-next returns a normalized next action with absolute pixel coordinates.
  • If the model outputs relative coordinates (1000x1000), the script scales to screen pixels.
  • Combine with adb/hdc actions (e.g., adb shell input tap X Y) for device control.
  • Use --log-level debug to print the raw model response for troubleshooting.

Default Models (Doubao)

  • doubao-seed-1-8-251228
  • doubao-seed-1-6-vision-250815

References

  • references/doubao-api.md

Comments

Markdown supported