mcpField
J
jb-docs-scraper

jb-docs-scraper

github.com/bjesuiterDocuments036 installsUpdated 1w ago
Repository

About jb-docs-scraper

jb-docs-scraper is a skill that crawls documentation websites and saves their pages as local Markdown files. It is for people who need a local copy of docs, with URL filtering and controls for crawl depth and page count.

Key features

  1. 01Crawl documentation with breadth-first search
  2. 02Detect domain and URL pattern
  3. 03Filter pages by URL pattern
  4. 04Convert pages to Markdown
  5. 05Mirror URL paths in output folders
  6. 06Set crawl depth and page limits

Use cases

  1. 01Save a documentation site for local reference
  2. 02Limit a crawl to a documentation subsection
  3. 03Control the scope of a test crawl
  4. 04Refresh local Markdown copies of documentation

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/bjesuiter/skills --skill jb-docs-scraper

After installing, type/jb-docs-scraperin Claude Code to confirm it loaded.

Bundled filesreferences/scrape_docs.py

Details

From the project README

Documentation Scraper

Scrape any documentation website into local markdown files. Uses crawl4ai for async web crawling.

Quick Start

# Scrape any documentation URL
uv run --with crawl4ai python ./references/scrape_docs.py <URL>

# Examples
uv run --with crawl4ai python ./references/scrape_docs.py https://mediasoup.org/documentation/v3/
uv run --with crawl4ai python ./references/scrape_docs.py https://docs.rombo.co/tailwind

Output goes to ./docs/<auto-detected-name>/ by default.

Prerequisites (First Time Only)

uv run --with crawl4ai playwright install

Usage

uv run --with crawl4ai python ./references/scrape_docs.py <URL> [OPTIONS]
Options
OptionDescriptionDefault
-o, --output PATHOutput directory./docs/<auto-detected-name>
--max-depth NMaximum link depth6
--max-pages NMaximum pages to scrape500
--url-pattern PATTERNURL filter (glob)Auto-detected
-q, --quietSuppress verbose outputFalse
Examples
# Basic - scrape to ./docs/documentation_v3/
uv run --with crawl4ai python ./references/scrape_docs.py \
  https://mediasoup.org/documentation/v3/

# Custom output directory
uv run --with crawl4ai python ./references/scrape_docs.py \
  https://docs.rombo.co/tailwind \
  --output ./my-tailwind-docs

# Limit crawl scope
uv run --with crawl4ai python ./references/scrape_docs.py \
  https://tanstack.com/start/latest/docs/framework/react/overview \
  --max-pages 50 \
  --max-depth 3

# Custom URL pattern filter
uv run --with crawl4ai python ./references/scrape_docs.py \
  https://example.com/docs/api/v2/ \
  --url-pattern "*api/v2/*"

How It Works

  1. Auto-detects domain and URL pattern from the input URL
  2. Crawls using BFS (breadth-first search) strategy
  3. Filters to stay within the documentation section
  4. Converts pages to clean markdown
  5. Saves with directory structure mirroring the URL paths

Output Structure

docs/<name>/
  index.md           # Root page
  getting-started.md
  api/
    overview.md
    client.md
  guides/
    installation.md

Troubleshooting

IssueSolution
Playwright browser binaries are missingRun uv run --with crawl4ai playwright install
Empty outputCheck if URL pattern matches actual doc URLs. Try --url-pattern
Missing pagesIncrease --max-depth or --max-pages
Wrong pages scrapedUse stricter --url-pattern

Tips

  1. Test first - Use --max-pages 10 to verify config before full crawl
  2. Check output name - Script auto-detects from URL path segments
  3. Rerun safe - Files are overwritten, duplicates skipped

Comments

Markdown supported