just-scrape

使用 just-scrape 进行网页抓取

由 ScrapeGraph AI 提供 AI 驱动的网页抓取 CLI。前往 dashboard.scrapegraphai.com 获取 API 密钥。

设置

请始终安装或运行 @latest 版本，以确保使用最新功能和修复。

npm install -g just-scrape@latest           # npm
pnpm add -g just-scrape@latest              # pnpm
yarn global add just-scrape@latest          # yarn
bun add -g just-scrape@latest               # bun
npx just-scrape@latest --help               # run without installing
bunx just-scrape@latest --help              # run without installing (bun)

export SGAI_API_KEY="sgai-..."

API 密钥的查找顺序：SGAI_API_KEY 环境变量 → .env 文件 → ~/.scrapegraphai/config.json → 交互式提示（会保存到配置文件）。

命令选择

需求	命令
从已知 URL 提取结构化数据	smart-scraper
搜索网络并从结果中提取数据	search-scraper
将网页转换为干净的 Markdown	markdownify
抓取网站中的多个页面	crawl
获取原始 HTML	scrape
自动执行浏览器操作（登录、点击、填写）	agentic-scraper
根据描述生成 JSON 架构	generate-schema
获取站点地图中的所有 URL	sitemap
查看信用额度余额	credits
浏览以往请求	history
验证 API 密钥	validate

常用标志

所有命令都支持 --json，用于输出机器可读内容（隐藏横幅、加载动画和提示）。

抓取命令都支持以下可选标志：

• --stealth — 绕过反机器人检测（+4 个信用额度）
• --headers <json> — 以 JSON 字符串形式指定自定义 HTTP 标头
• --schema <json> — 强制使用输出 JSON 架构

命令

Smart Scraper

使用 AI 从任意 URL 提取结构化数据。

just-scrape smart-scraper <url> -p <prompt>
just-scrape smart-scraper <url> -p <prompt> --schema <json>
just-scrape smart-scraper <url> -p <prompt> --scrolls <n>     # infinite scroll (0-100)
just-scrape smart-scraper <url> -p <prompt> --pages <n>       # multi-page (1-100)
just-scrape smart-scraper <url> -p <prompt> --stealth         # anti-bot (+4 credits)
just-scrape smart-scraper <url> -p <prompt> --cookies <json> --headers <json>
just-scrape smart-scraper <url> -p <prompt> --plain-text

# E-commerce extraction
just-scrape smart-scraper https://store.example.com/shoes -p "Extract all product names, prices, and ratings"

# Strict schema + scrolling
just-scrape smart-scraper https://news.example.com -p "Get headlines and dates" \
  --schema '{"type":"object","properties":{"articles":{"type":"array","items":{"type":"object","properties":{"title":{"type":"string"},"date":{"type":"string"}}}}}}' \
  --scrolls 5

# JS-heavy SPA behind anti-bot
just-scrape smart-scraper https://app.example.com/dashboard -p "Extract user stats" \
  --stealth

Search Scraper

搜索网络并从结果中提取结构化数据。

just-scrape search-scraper <prompt>
just-scrape search-scraper <prompt> --num-results <n>     # sources to scrape (3-20, default 3)
just-scrape search-scraper <prompt> --no-extraction       # markdown only (2 credits vs 10)
just-scrape search-scraper <prompt> --schema <json>
just-scrape search-scraper <prompt> --stealth --headers <json>

# Research across sources
just-scrape search-scraper "Best Python web frameworks in 2025" --num-results 10

# Cheap markdown-only
just-scrape search-scraper "React vs Vue comparison" --no-extraction --num-results 5

# Structured output
just-scrape search-scraper "Top 5 cloud providers pricing" \
  --schema '{"type":"object","properties":{"providers":{"type":"array","items":{"type":"object","properties":{"name":{"type":"string"},"free_tier":{"type":"string"}}}}}}'

Markdownify

将任意网页转换为干净的 Markdown。

just-scrape markdownify <url>
just-scrape markdownify <url> --stealth         # +4 credits
just-scrape markdownify <url> --headers <json>

just-scrape markdownify https://blog.example.com/my-article
just-scrape markdownify https://protected.example.com --stealth
just-scrape markdownify https://docs.example.com/api --json | jq -r '.result' > api-docs.md

Crawl

抓取多个页面并从每个页面提取数据。

just-scrape crawl <url> -p <prompt>
just-scrape crawl <url> -p <prompt> --max-pages <n>      # default 10
just-scrape crawl <url> -p <prompt> --depth <n>           # default 1
just-scrape crawl <url> --no-extraction --max-pages <n>   # markdown only (2 credits/page)
just-scrape crawl <url> -p <prompt> --schema <json>
just-scrape crawl <url> -p <prompt> --rules <json>        # include_paths, same_domain
just-scrape crawl <url> -p <prompt> --no-sitemap
just-scrape crawl <url> -p <prompt> --stealth

# Crawl docs site
just-scrape crawl https://docs.example.com -p "Extract all code snippets" --max-pages 20 --depth 3

# Filter to blog pages only
just-scrape crawl https://example.com -p "Extract article titles" \
  --rules '{"include_paths":["/blog/*"],"same_domain":true}' --max-pages 50

# Raw markdown, no AI extraction (cheaper)
just-scrape crawl https://example.com --no-extraction --max-pages 10

Scrape

获取 URL 的原始 HTML 内容。

just-scrape scrape <url>
just-scrape scrape <url> --stealth          # +4 credits
just-scrape scrape <url> --branding         # extract logos/colors/fonts (+2 credits)
just-scrape scrape <url> --country-code <iso>

just-scrape scrape https://example.com
just-scrape scrape https://store.example.com --stealth --country-code DE
just-scrape scrape https://example.com --branding

Agentic Scraper

使用 AI 自动化浏览器操作——登录、点击、导航和填写表单。步骤以逗号分隔的字符串表示。

just-scrape agentic-scraper <url> -s <steps>
just-scrape agentic-scraper <url> -s <steps> --ai-extraction -p <prompt>
just-scrape agentic-scraper <url> -s <steps> --schema <json>
just-scrape agentic-scraper <url> -s <steps> --use-session   # persist browser session

# Login + extract dashboard
just-scrape agentic-scraper https://app.example.com/login \
  -s "Fill email with [email protected],Fill password with secret,Click Sign In" \
  --ai-extraction -p "Extract all dashboard metrics"

# Multi-step form
just-scrape agentic-scraper https://example.com/wizard \
  -s "Click Next,Select Premium plan,Fill name with John,Click Submit"

# Persistent session across runs
just-scrape agentic-scraper https://app.example.com \
  -s "Click Settings" --use-session

Generate Schema

根据自然语言描述生成 JSON 架构。

just-scrape generate-schema <prompt>
just-scrape generate-schema <prompt> --existing-schema <json>

just-scrape generate-schema "E-commerce product with name, price, ratings, and reviews array"

# Refine an existing schema
just-scrape generate-schema "Add an availability field" \
  --existing-schema '{"type":"object","properties":{"name":{"type":"string"},"price":{"type":"number"}}}'

Sitemap

获取网站站点地图中的所有 URL。

just-scrape sitemap <url>
just-scrape sitemap https://example.com --json | jq -r '.urls[]'

History

浏览请求历史记录。默认采用交互模式（使用方向键导航，选择条目查看详情）。

just-scrape history <service>                     # interactive browser
just-scrape history <service> <request-id>        # specific request
just-scrape history <service> --page <n>
just-scrape history <service> --page-size <n>     # max 100
just-scrape history <service> --json

服务：markdownify、smartscraper、searchscraper、scrape、crawl、agentic-scraper、sitemap

just-scrape history smartscraper
just-scrape history crawl --json --page-size 100 | jq '.requests[] | {id: .request_id, status}'

Credits & Validate

just-scrape credits
just-scrape credits --json | jq '.remaining_credits'
just-scrape validate

常用模式

先生成架构，再用它进行抓取

just-scrape generate-schema "Product with name, price, and reviews" --json | jq '.schema' > schema.json
just-scrape smart-scraper https://store.example.com -p "Extract products" --schema "$(cat schema.json)"

通过管道传递 JSON 以用于脚本

just-scrape sitemap https://example.com --json | jq -r '.urls[]' | while read url; do
  just-scrape smart-scraper "$url" -p "Extract title" --json >> results.jsonl
done

受保护的网站

# JS-heavy SPA behind Cloudflare
just-scrape smart-scraper https://protected.example.com -p "Extract data" --stealth

# With custom cookies/headers
just-scrape smart-scraper https://example.com -p "Extract data" \
  --cookies '{"session":"abc123"}' --headers '{"Authorization":"Bearer token"}'

信用额度消耗

功能	额外信用额度
--stealth	每次请求 +4
--branding（仅限 scrape）	+2
search-scraper 提取	每次请求 10
search-scraper --no-extraction	每次请求 2
crawl --no-extraction	每页 2

环境变量

SGAI_API_KEY=sgai-...              # API key
JUST_SCRAPE_TIMEOUT_S=300          # Request timeout in seconds (default 120)
JUST_SCRAPE_DEBUG=1                # Debug logging to stderr