mcpField
F
firecrawl-knowledge-ingest

firecrawl-knowledge-ingest

github.com/firecrawlData & databases21231.4k installsUpdated 1w ago
Repository

About firecrawl-knowledge-ingest

firecrawl-knowledge-ingest is a skill for extracting documentation from portals that may require browser navigation, authentication, pagination, or JavaScript rendering. It collects article content and metadata, tracks extraction progress and failures, and can produce structured JSON or Markdown output.

Key features

  1. 01Browser-based portal navigation
  2. 02Support for authentication and pagination
  3. 03Article content extraction as Markdown
  4. 04Collection of article metadata
  5. 05Progress and failure tracking
  6. 06JSON or Markdown output

Use cases

  1. 01Ingesting a documentation portal
  2. 02Extracting content from authenticated sections
  3. 03Collecting articles across paginated pages
  4. 04Preserving code examples and tables

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/firecrawl/firecrawl-workflows --skill firecrawl-knowledge-ingest

After installing, type/firecrawl-knowledge-ingestin Claude Code to confirm it loaded.

Details

From the project README

Firecrawl Knowledge Ingest

Use this when a docs portal needs browser navigation, auth, pagination, or JS rendering.

Onboarding Interview

Infer the portal URL, output format, auth needs, and page limit from context. If the portal is clear, proceed immediately.

Ask at most 1-3 concise questions only if blocked, such as the portal URL, whether authentication is required, or the desired output format.

Firecrawl Collection Plan

Use Firecrawl browser to:

  • open the portal and inspect navigation
  • identify sections, categories, sidebar links, and article URLs
  • follow sidebar navigation, next links, pagination, load-more controls, or search
  • scrape article content as markdown
  • extract metadata such as title, section, last updated date, author, and tags

Try Firecrawl map as a supplement for public URLs, but use browser navigation for auth-gated or JS-heavy content.

Final Deliverable

# Knowledge Ingest: [Portal]

## Summary
[Pages extracted, sections covered, limitations]

## Output
[JSON/markdown/merged file path or content]

## Sections
[Section names and article counts]

## Failed Or Restricted Pages
[Any access/loading issues]

## Sources
[URLs extracted]

## Rerun Inputs
workflow: firecrawl-knowledge-ingest
url: [portal url]
format: [json/markdown/merged]
max_pages: [number]

JSON Shape

Use source, url, extractedAt, totalArticles, and sections[] with article title, url, section, content, and metadata.

Quality Bar

  • Preserve code examples, tables, and formatting.
  • Strip nav chrome, headers, and footers.
  • Track extraction progress and page failures.
  • Respect authentication boundaries.

Comments

Markdown supported