mcpField
F
firecrawl-crawl

firecrawl-crawl

github.com/firecrawlData & databases1395.9k installsUpdated 6d ago
Repository

About firecrawl-crawl

firecrawl-crawl is a skill for bulk extraction of website content by following links within depth and page limits. It covers scoping crawls, waiting for results or polling jobs, and saving output.

Key features

  1. 01Follow links to extract site pages
  2. 02Set crawl depth and page limits
  3. 03Include selected URL paths
  4. 04Wait for completion or poll a job
  5. 05Save crawl output

Use cases

  1. 01Collect pages from a documentation section
  2. 02Crawl a site to a limited depth
  3. 03Restrict extraction to specified paths
  4. 04Retrieve results from an asynchronous crawl

MCP support

The input does not state an MCP connection or interface.

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/firecrawl/skills --skill firecrawl-crawl

After installing, type/firecrawl-crawlin Claude Code to confirm it loaded.

Details

From the project README

firecrawl crawl

Bulk extract content from a website. Crawls pages following links up to a depth/limit.

Prerequisite: crawl requires authentication (no keyless free tier); without credentials the CLI prompts an interactive login.

Quick start

# Crawl a docs section
firecrawl crawl "<url>" --include-paths /docs --limit 50 --wait -o .firecrawl/crawl.json

# Full crawl with depth limit
firecrawl crawl "<url>" --max-depth 3 --wait --progress -o .firecrawl/crawl.json

# Check status of a running crawl
firecrawl crawl <job-id>

Run firecrawl crawl --help for the full option list.

Done when: the crawl reaches a terminal status and the saved output under .firecrawl/ contains the expected pages.

Tips

  • Use --wait when you need the results immediately. It has no default timeout; use --timeout <seconds> to bound polling. Without --wait, crawl returns a job ID for async polling.
  • Scope crawls with --include-paths whenever the request names a section — crawl only the pages you need.
  • Crawl consumes credits per page. Check firecrawl credit-usage before large crawls (credit-usage requires authentication).

See also

Comments

Markdown supported