mcpField
F
firecrawl-crawl

firecrawl-crawl

Official
firecrawlMarketing & growth67480.9k installsUpdated 2d ago
Repository

About firecrawl-crawl

firecrawl crawl is a skill in the Firecrawl CLI for bulk extracting content from a website by following links up to a depth or page limit. It requires authentication, and is aimed at users who need many pages from one site saved locally.

Key features

  1. 01Crawls pages by depth and page limit
  2. 02Scopes crawls with include-paths
  3. 03Waits for results with a wait flag
  4. 04Returns a job ID for asynchronous polling
  5. 05Saves output to local files

Use cases

  1. 01Extracting a documentation section in bulk
  2. 02Checking credit usage before large crawls
  3. 03Saving site content locally for later processing

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/firecrawl/cli --skill firecrawl-crawl

After installing, type/firecrawl-crawlin Claude Code to confirm it loaded.

Details

From the project README

firecrawl crawl

Bulk extract content from a website. Crawls pages following links up to a depth/limit.

Prerequisite: crawl requires authentication (no keyless free tier); without credentials the CLI prompts an interactive login.

Quick start

# Crawl a docs section
firecrawl crawl "<url>" --include-paths /docs --limit 50 --wait -o .firecrawl/crawl.json

# Full crawl with depth limit
firecrawl crawl "<url>" --max-depth 3 --wait --progress -o .firecrawl/crawl.json

# Check status of a running crawl
firecrawl crawl <job-id>

Run firecrawl crawl --help for the full option list.

Done when: the crawl reaches a terminal status and the saved output under .firecrawl/ contains the expected pages.

Tips

  • Use --wait when you need the results immediately. It has no default timeout; use --timeout <seconds> to bound polling. Without --wait, crawl returns a job ID for async polling.
  • Scope crawls with --include-paths whenever the request names a section — crawl only the pages you need.
  • Crawl consumes credits per page. Check firecrawl credit-usage before large crawls (credit-usage requires authentication).

See also

Comments

Markdown supported