mcpField
F
firecrawl-crawl

firecrawl-crawl

github.com/firecrawlData & databases252104 installsUpdated 1w ago
Repository

About firecrawl-crawl

A Firecrawl skill for bulk website extraction, following links within configured depth and page limits. It covers scoping crawls, waiting for completion, and checking saved results.

Key features

  1. 01Crawl linked pages in bulk
  2. 02Limit crawl depth and page count
  3. 03Scope crawls by URL paths
  4. 04Wait for jobs or poll asynchronously
  5. 05Save and inspect crawl results

Use cases

  1. 01Extracting a documentation section
  2. 02Collecting pages up to a depth limit
  3. 03Checking completion of an asynchronous crawl

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/firecrawl/firecrawl-claude-plugin --skill firecrawl-crawl

After installing, type/firecrawl-crawlin Claude Code to confirm it loaded.

Details

From the project README

firecrawl crawl

Bulk extract content from a website. Crawls pages following links up to a depth/limit.

Prerequisite: crawl requires authentication (no keyless free tier); without credentials the CLI prompts an interactive login.

Quick start

# Crawl a docs section
firecrawl crawl "<url>" --include-paths /docs --limit 50 --wait -o .firecrawl/crawl.json

# Full crawl with depth limit
firecrawl crawl "<url>" --max-depth 3 --wait --progress -o .firecrawl/crawl.json

# Check status of a running crawl
firecrawl crawl <job-id>

Run firecrawl crawl --help for the full option list.

Done when: the crawl reaches a terminal status and the saved output under .firecrawl/ contains the expected pages.

Tips

  • Use --wait when you need the results immediately. It has no default timeout; use --timeout <seconds> to bound polling. Without --wait, crawl returns a job ID for async polling.
  • Scope crawls with --include-paths whenever the request names a section — crawl only the pages you need.
  • Crawl consumes credits per page. Check firecrawl credit-usage before large crawls (credit-usage requires authentication).

See also

Comments

Markdown supported