mcpField
B
BrightData

BrightData

github.com/danielmiesslerData & databases19.6k147 installsUpdated 1d ago
Repository

About BrightData

BrightData is a Claude skill for fetching a URL or crawling a website, returning content as Markdown. It escalates through four retrieval methods only when a page blocks the current method.

Key features

  1. 01Single-page URL retrieval
  2. 02Website crawling
  3. 03Four-tier access escalation
  4. 04Markdown output
  5. 05Cloudflare Markdown pre-check

Use cases

  1. 01Retrieving content from a public URL
  2. 02Fetching pages that require JavaScript rendering
  3. 03Crawling a documentation section or site

MCP support

For blocked pages, the fourth retrieval tier uses the Bright Data MCP tool `mcp__Brightdata__scrape_as_markdown`. Light Crawl also uses the MCP `scrape_batch` tool.

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/danielmiessler/lifeos --skill brightdata

After installing, type/brightdatain Claude Code to confirm it loaded.

Bundled filesWorkflows/Crawl.mdWorkflows/FourTierScrape.md

Details

From the project README

Customization

Before executing, check for user customizations at: ~/.claude/LIFEOS/USER/CUSTOMIZATIONS/SKILLS/BrightData/

If this directory exists, load and apply any PREFERENCES.md, configurations, or resources found there. These override default behavior. If the directory does not exist, proceed with skill defaults.

🚨 MANDATORY: Voice Notification (REQUIRED BEFORE ANY ACTION)

You MUST send this notification BEFORE doing anything else when this skill is invoked.

  1. Send voice notification:

    curl -s -X POST http://localhost:31337/notify \
  -H "Content-Type: application/json" \
  -d '{"message": "Running the WORKFLOWNAME workflow in the BrightData skill to ACTION"}' \
  > /dev/null 2>&1 &

  2. Output text notification:

    Running the **WorkflowName** workflow in the **BrightData** skill to ACTION...

This is not optional. Execute this curl command immediately upon skill invocation.

BrightData

Scrapes a single URL (FourTierScrape) or crawls a whole site (Crawl), escalating through four tiers only as far as each page needs. Output is always markdown. Start at Tier 1 and step up only when blocked — reaching for the heavy proxy every time wastes Tier-4 credits. A Cloudflare Accept: text/markdown pre-check runs before Tier 1 (recipe in FourTierScrape.md).

The four tiers (tool contract)

TierToolWins onCost / latency
1WebFetchpublic content, no bot detectionfree · ~2-5s
2curl + Chrome headersuser-agent / basic header checksfree · ~3-7s
3Interceptor (real Chrome)JavaScript-rendered / SPA pagesfree · ~10-20s
4Bright Data MCP mcp__Brightdata__scrape_as_markdownCAPTCHA, advanced fingerprinting, residential-IP needsBright Data credits · ~5-15s

Playwright is banned across LifeOS — Tier 3 is Interceptor. Skip-ahead: explicit "use Bright Data" → Tier 4; "use browser" → Tier 3; a domain that already failed Tier 1 → start at Tier 2. The exact curl header block, Cloudflare pre-check, and Interceptor commands live in Workflows/FourTierScrape.md.

Workflows

When routing, output: Running the **WorkflowName** workflow in the **BrightData** skill to ACTION...

WorkflowTriggerFile
FourTierScrape"scrape/fetch/pull/get/retrieve [URL]", "can't access this site", "site is blocking me", "use Bright Data to fetch"Workflows/FourTierScrape.md
Crawl"crawl this site", "spider this domain", "map this website", "get all pages from", "scrape the whole site", "crawl all pages under /docs"Workflows/Crawl.md

Crawl picks Light Crawl (MCP scrape_batch + link loop, ≤50 pages, ~$0.006/page) for a section, or Full Crawl (Bright Data Crawl API api.brightdata.com/datasets/v3/trigger, $1.50/1K pages) for whole sites.

Gotchas

  • 4-tier escalation: WebFetch → curl → Interceptor → Bright Data proxy. Always start at Tier 1 and escalate only when blocked. Playwright is banned across LifeOS.
  • Bright Data proxy has usage costs. Don't use Tier 4 for sites accessible via Tier 1-3.
  • CAPTCHA-solving introduces latency. Allow extra time for Tier 4 responses.
  • Credentials in ~/.claude/.env — BRIGHTDATA_API_KEY.

Execution Log

After completing any workflow, append a single JSONL entry:

echo '{"ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","skill":"BrightData","workflow":"WORKFLOW_USED","input":"8_WORD_SUMMARY","status":"ok|error","duration_s":SECONDS}' >> ~/.claude/LIFEOS/MEMORY/SKILLS/execution.jsonl

Replace WORKFLOW_USED with the workflow executed, 8_WORD_SUMMARY with a brief input description, and SECONDS with approximate wall-clock time. Log status: "error" if the workflow failed.

Comments

Markdown supported