mcpField
W
web-fetch

web-fetch

github.com/alleneubankProductivity50976 installsUpdated 18w ago
Repository

About web-fetch

web-fetch is a skill for extracting web content, trying Markdown endpoints, site-specific selectors, and a Bun script in that order. It is aimed at turning pages into readable text, with selector patterns for known sites.

Key features

  1. 01Prefer Markdown content endpoints
  2. 02Extract HTML with CSS selectors
  3. 03Use a Bun fallback script
  4. 04Support include and exclude selectors
  5. 05Provide troubleshooting guidance

Use cases

  1. 01Extract article text from documentation sites
  2. 02Select specific regions of a web page
  3. 03Run fallback parsing when selectors perform poorly

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/alleneubank/claude-code --skill web-fetch

After installing, type/web-fetchin Claude Code to confirm it loaded.

Bundled files

References and scripts installed along with SKILL.md.

.gitignore
fetch.ts
package.json

Details

From the project README

Web Content Fetching

Fetch web content in this order:

  1. Prefer markdown-native endpoints (content-type: text/markdown)
  2. Use selector-based HTML extraction for known sites
  3. Use the bundled Bun fallback script when selectors fail

Prerequisites

Verify required tools before extracting:

command -v curl >/dev/null || echo "curl is required"
command -v html2markdown >/dev/null || echo "html2markdown is required for HTML extraction"
command -v bun >/dev/null || echo "bun is required for fetch.ts fallback"

Install Bun dependencies for the bundled script:

cd ~/.claude/skills/web-fetch && bun install

Default Workflow

Use this as the default flow for any URL:

URL="<url>"
CONTENT_TYPE="$(curl -sIL "$URL" | awk -F': ' 'tolower($1)=="content-type"{print tolower($2)}' | tr -d '\r' | tail -1)"

if echo "$CONTENT_TYPE" | grep -q "markdown"; then
  curl -sL "$URL"
else
  curl -sL "$URL" \
    | html2markdown \
        --include-selector "article,main,[role=main]" \
        --exclude-selector "nav,header,footer,script,style"
fi

Known Site Selectors

SiteInclude SelectorExclude Selector
platform.claude.com#content-container-
docs.anthropic.com#content-container-
developer.mozilla.orgarticle-
github.com (docs)articlenav,.sidebar
Genericarticle,main,[role=main]nav,header,footer,script,style

Example:

curl -sL "<url>" \
  | html2markdown \
      --include-selector "#content-container" \
      --exclude-selector "nav,header,footer"

Finding the Right Selector

When a site isn't in the patterns list:

# Check what content containers exist
curl -s "<url>" | grep -o '<article[^>]*>\|<main[^>]*>\|id="[^"]*content[^"]*"' | head -10

# Test a selector
curl -sL "<url>" | html2markdown --include-selector "<selector>" | head -30

# Check line count
curl -sL "<url>" | html2markdown --include-selector "<selector>" | wc -l

Universal Fallback Script

When selectors produce poor output, run the bundled parser:

bun ~/.claude/skills/web-fetch/fetch.ts "<url>"

If already in the skill directory:

bun fetch.ts "<url>"

Options Reference

--include-selector "CSS"  # Keep only matching elements
--exclude-selector "CSS"  # Remove matching elements
--domain "https://..."    # Convert relative links to absolute

Troubleshooting

Empty output with selectors: The page might be markdown-native. Check headers first:

curl -sIL "<url>" | grep -i '^content-type:'

Wrong content selected: The site may have multiple article/main regions:

curl -s "<url>" | grep -o '<article[^>]*>'

html2markdown not found: Install it, then retry selector-based extraction.

bun or script deps missing: Run cd ~/.claude/skills/web-fetch && bun install.

Missing code blocks: Check if the site uses non-standard code formatting.

Client-rendered content: If HTML only has "Loading..." placeholders, the content is JS-rendered. Neither curl nor the Bun script can extract it; use browser-based tools.

Comments

Markdown supported