mcpField
J
jina-web-fetch

jina-web-fetch

github.com/xiaojiongqianProductivity357 installsUpdated 25w ago
Repository

About jina-web-fetch

jina-web-fetch is a skill for capturing web page content with a direct-fetch-first workflow and a jina.ai fallback when access fails or appears blocked. It saves raw captures for traceability and reports which fetch path was used.

Key features

  1. 01Try direct fetch before fallback
  2. 02Retry through jina.ai when blocked
  3. 03Save captured content to a file
  4. 04Report the selected source path
  5. 05Configure timeouts and fallback behavior

Use cases

  1. 01Capture content from pages that resist direct fetching
  2. 02Preserve raw page text for auditing
  3. 03Fetch pages that require JavaScript or show access barriers

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/xiaojiongqian/skills-hub --skill jina-web-fetch

After installing, type/jina-web-fetchin Claude Code to confirm it loaded.

Bundled filesagents/openai.yamlscripts/fetch_with_jina_fallback.sh

Details

From the project README

Jina Web Fetch

Use this skill to capture content from hard-to-fetch pages while keeping a deterministic workflow:

  1. Try direct fetch first.
  2. If direct fetch fails or looks blocked, retry via jina.ai.

Quick Start

bash "<path-to-skill>/scripts/fetch_with_jina_fallback.sh" "<url>" "<output_file>"

Example:

bash "<path-to-skill>/scripts/fetch_with_jina_fallback.sh" \
  "https://x.com/trq212/status/2027463795355095314" \
  "raw/x-status.txt"

The script prints source=direct or source=jina to stderr so you can see which path was used.

If output_file is omitted, content is printed to stdout.

Default Workflow

  1. Run the script with the target URL.
  2. Save raw output under a traceable path like raw/<slug>.txt.
  3. Parse extracted text/markdown for:
    • main body
    • media links (images/videos)
    • referenced URLs
  4. Keep original URL + raw capture together for auditability.

Blocking Heuristics

The script auto-falls back to jina.ai when direct content looks like:

  • login wall / sign-up prompt
  • JS-required page
  • anti-bot / captcha / access denied page
  • very small shell-like HTML page (default threshold < 800 bytes)

Environment Knobs

  • FETCH_TIMEOUT (default 25)
  • FETCH_CONNECT_TIMEOUT (default 10)
  • FETCH_MIN_BYTES (default 800)
  • JINA_FORCE=1 to skip direct fetch and always use jina.ai

URL Format Note

Fallback URL is built as: https://r.jina.ai/http://<original-host-and-path>

Comments

Markdown supported