firecrawl

Firecrawl 网页抓取与数据提取

安装

pip install firecrawl-py

环境设置

设置 Firecrawl API 密钥：

export FIRECRAWL_API_KEY="your-api-key-here"

脚本

注意：将 SKILL_ROOT 设置为此技能的基础目录。引用捆绑的脚本时，使用 python3 "$SKILL_ROOT/scripts/<script>.py" ...（不要使用相对于当前工作目录的路径）。

scrape.py - 单页抓取

功能最强大、最可靠的抓取工具。已明确知道哪个页面包含所需信息时使用。

# Basic scrape (returns markdown)
python3 "$SKILL_ROOT/scripts/scrape.py" "https://example.com"

# Get HTML format
python3 "$SKILL_ROOT/scripts/scrape.py" "https://example.com" --format html

# Extract only main content (removes headers, footers, etc.)
python3 "$SKILL_ROOT/scripts/scrape.py" "https://example.com" --only-main

# Combine options
python3 "$SKILL_ROOT/scripts/scrape.py" "https://docs.example.com/api" --format markdown --only-main

search.py - 网页搜索

不知道哪个网站包含所需信息时，搜索网页。

# Basic search
python3 "$SKILL_ROOT/scripts/search.py" "latest AI research papers 2024"

# Limit results
python3 "$SKILL_ROOT/scripts/search.py" "Python web scraping tutorials" --limit 5

# Search with scraping (get full content)
python3 "$SKILL_ROOT/scripts/search.py" "firecrawl documentation" --limit 3

map.py - URL 发现

发现网站上的所有 URL。在决定抓取哪些内容之前使用。

# Map a website
python3 "$SKILL_ROOT/scripts/map.py" "https://docs.example.com"

# Limit number of URLs
python3 "$SKILL_ROOT/scripts/map.py" "https://example.com" --limit 100

# Search within mapped URLs
python3 "$SKILL_ROOT/scripts/map.py" "https://docs.example.com" --search "authentication"

crawl.py - 多页爬取

从多个相关页面提取内容。警告：速度可能较慢，且会返回大量结果。

# Basic crawl
python3 "$SKILL_ROOT/scripts/crawl.py" "https://docs.example.com"

# Limit pages
python3 "$SKILL_ROOT/scripts/crawl.py" "https://docs.example.com" --limit 20

# Control crawl depth
python3 "$SKILL_ROOT/scripts/crawl.py" "https://docs.example.com" --limit 10 --depth 2

extract.py - 结构化数据提取

使用大模型能力提取特定的结构化数据。

# Extract with prompt
python3 "$SKILL_ROOT/scripts/extract.py" "https://example.com/pricing" \
  --prompt "Extract all pricing tiers with their features and prices"

# Extract with JSON schema
python3 "$SKILL_ROOT/scripts/extract.py" "https://example.com/team" \
  --prompt "Extract team member information" \
  --schema '{"type":"object","properties":{"members":{"type":"array","items":{"type":"object","properties":{"name":{"type":"string"},"role":{"type":"string"},"bio":{"type":"string"}}}}}}'

# Extract from multiple URLs
python3 "$SKILL_ROOT/scripts/extract.py" "https://example.com/page1" "https://example.com/page2" \
  --prompt "Extract product information"

agent.py - 自主数据收集

可自主搜索、浏览并从网络上的任意位置提取数据的 Agent。

# Simple research task
python3 "$SKILL_ROOT/scripts/agent.py" --prompt "Find the founders of Firecrawl and their backgrounds"

# Complex data gathering
python3 "$SKILL_ROOT/scripts/agent.py" --prompt "Find the top 5 AI startups founded in 2024 and their funding amounts"

# Focus on specific URLs
python3 "$SKILL_ROOT/scripts/agent.py" \
  --prompt "Compare the features and pricing" \
  --urls "https://example1.com,https://example2.com"

# With output schema
python3 "$SKILL_ROOT/scripts/agent.py" \
  --prompt "Find recent tech layoffs" \
  --schema '{"type":"object","properties":{"layoffs":{"type":"array","items":{"type":"object","properties":{"company":{"type":"string"},"count":{"type":"number"},"date":{"type":"string"}}}}}}'

输出格式

所有脚本都会将 JSON 输出到 stdout。错误会写入 stderr。

成功响应

{
  "success": true,
  "data": { ... }
}

错误响应

{
  "success": false,
  "error": "Error message"
}

提示

1. 性能：抓取单个页面时使用 scrape，借助缓存，速度可提升 500%
2. 发现：先使用 map 查找 URL，再使用 scrape 抓取特定页面
3. 大型网站：优先使用 map + scrape，以获得更好的控制，而不是使用 crawl
4. 结构化数据：使用 extract 配合 JSON 架构，确保输出一致
5. 研究：不知道在哪里查找数据时使用 agent