crawlport

crawlport

crawlport is a doorway between an agent and the readable web, backed by Firecrawl. Give it one page and it hands back clean markdown; give it a whole site and it will map the pages, crawl a section, or scrape a list in batch; give it a question and it can search, pull structured fields out of pages to a schema you describe, research across many sources, or draft the llms.txt a site would publish about itself. It runs as a CLI or as an MCP server, and it needs a Firecrawl API key.

Setup

Get a key from Firecrawl and export it:

export FIRECRAWL_API_KEY=<redacted>

Without the variable set, every command and tool call fails with an error that names FIRECRAWL_API_KEY — nothing is sent anywhere.

Install

npx @reneturcios/crawlport scrape https://example.com/some/page

Or run the container image:

docker run -i -e FIRECRAWL_API_KEY ghcr.io/cloudcorprecords/crawlport

The container defaults to the MCP server (see below). Pass CLI arguments to use it as a CLI instead.

CLI

# One page, as markdown (add --formats html,links etc. for more)
crawlport scrape https://example.com/article --only-main-content

# Discover a site's URLs, optionally filtered
crawlport map https://example.com --search pricing --limit 100

# Web search, with locale controls
crawlport search "standing desk reviews" --limit 5 --lang en --country US

# Start a crawl, then poll it with the returned job id
crawlport crawl https://example.com/docs --limit 50 --max-depth 2
crawlport crawl-status <job-id>

# Structured extraction and batch scraping
crawlport extract https://example.com/a https://example.com/b --prompt "pull the price"
crawlport batch-scrape https://example.com/a https://example.com/b
crawlport batch-status <job-id>

# Research and llms.txt
crawlport deep-research "home battery storage costs in 2026" --max-urls 40
crawlport llmstxt https://example.com --max-urls 25

crawlport --version

Successful calls print the API's JSON response. Crawl and batch jobs are asynchronous upstream: the start call returns a job id, and the matching status command reads its progress and results. Exit codes: 0 on success, 1 when the API call fails (including a missing key), 2 for usage errors.

MCP server

One bin serves both faces. The mcp subcommand starts a stdio MCP server:

npx -y @reneturcios/crawlport mcp

MCP client configuration:

{
  "mcpServers": {
    "crawlport": {
      "command": "npx",
      "args": ["-y", "@reneturcios/crawlport", "mcp"],
      "env": { "FIRECRAWL_API_KEY": "your-key-here" }
    }
  }
}

Tools

Tool	Input	What it does
firecrawl_scrape	url (required); formats, onlyMainContent, waitFor, timeout, actions, includeTags, excludeTags, blockAds, proxy (optional)	Scrape one page via POST /v1/scrape.
firecrawl_map	url (required); search, limit, ignoreSitemap	List a site's discovered URLs via POST /v1/map.
firecrawl_search	query (required); limit, lang, country, scrapeOptions	Web search via POST /v1/search, optionally scraping each result.
firecrawl_crawl	url (required); limit, maxDepth, allowBackwardLinks, allowExternalLinks, excludePaths, includePaths, scrapeOptions	Start a crawl via POST /v1/crawl; returns a job id.
firecrawl_check_crawl_status	id (required)	Poll a crawl job via GET /v1/crawl/{id}.
firecrawl_extract	urls (required); prompt, schema, enableWebSearch, allowExternalLinks	LLM extraction to a prompt/schema via POST /v1/extract.
firecrawl_batch_scrape	urls (required); formats, onlyMainContent	Start a batch scrape via POST /v1/batch/scrape; returns a job id.
firecrawl_check_batch_status	id (required)	Poll a batch job via GET /v1/batch/scrape/{id}.
firecrawl_deep_research	query (required); maxDepth, timeLimit, maxUrls	Multi-source research via POST /v1/deep-research.
firecrawl_generate_llmstxt	url (required); maxUrls, showFullText	Generate llms.txt content via POST /v1/llmstxt.

Tool results are the upstream JSON, pretty-printed as text. Optional arguments are forwarded only when provided, so the API's defaults apply otherwise. Upstream failures come back as tool errors carrying the HTTP status and the API's own message.

Development

npm install
npm run build   # tsc -> dist/
npm test        # node test runner over test/*.test.ts, no network (fake fetch fixtures)
npm run dev     # run the CLI from source via tsx

License

MIT — see LICENSE.