scrapfly-scraper

Scrapfly Scraper

Use the Scrapfly Scraper API to collect web page data with proxy rotation, anti-bot bypass, JavaScript rendering, and JavaScript scenarios for browser control.

When to use

• Scraping web pages (HTML, JSON, text, markdown)
• Bypassing anti-bot protections (Cloudflare, DataDome, PerimeterX, Kasada, and more)
• Collecting data through rotating proxies with geo-targeting
• Rendering JavaScript-heavy pages with headless browsers
• Control the browser using JavaScript scenario for common actions (waiting for selectors, clicking, filling elements, etc.)
• Session-reuse for session-presited scraping
• Capturing browser XHR call data

Setup

pip install scrapfly-sdk

The API key must be provided via environment variable SCRAPFLY_API_KEY or passed directly to the client.

API Reference

Endpoint: https://api.scrapfly.io/scrape The HTTP method is forwarded to the upstream URL. To retrieve data from a web page or an API, use the GET method, which is the default method. If the API resource like an API requries other methods, like POST. You can use it via the method parameter of the ScrapeConfig

ScrapflyClient

from scrapfly import ScrapflyClient, ScrapeConfig, ScrapeApiResponse
import os

client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])

ScrapeConfig Parameters

Parameter	Type	Default	Description
url	str	required	Target URL to scrape
method	str	"GET"	HTTP method: GET, POST, PUT, PATCH, HEAD, OPTIONS
headers	dict	None	Custom HTTP headers
cookies	dict	None	Custom cookies (merged into headers["cookie"])
body	str	None	Raw request body for POST/PUT/PATCH; used when you pre-encode payload
data	dict	None	Form/JSON data for POST/PUT/PATCH; encoded according to Content-Type when body is not provided
timeout	int	None	Request timeout in ms (default ~150,000ms)
retry	bool	True	Auto-retry on network failures
country	str	None	Proxy country (ISO 3166-1 alpha-2, e.g. "us", "de")
proxy_pool	str	None	Proxy pool: "basic" (datacenter proxies, Unblocker off, 1 credit), "unblocker" (residential proxies and the Unblocker anti-bot bypass, 25 credits), "unblocker_ultra" (the same, and the only pool a target or an anti-bot protection with a price of its own accepts; that price is added) or "auto" (the API picks the pool before the request, from the target's profile: the Unblocker on datacenter proxies, or the pool the target needs; accepted on every target, the ones that require unblocker_ultra included, where their price is added; priced as the pool it runs on). Each pool sets the Unblocker itself. Unset, the SDK sends no proxy_pool. The migration guide covers the requests written before the proxy pools, the deprecated pool names "public_datacenter_pool" and "public_residential_pool" included
session	str	None	Session ID to persist cookies/fingerprint across requests
session_sticky_proxy	bool	True	Keep the same proxy IP for a given session; False is refused on "unblocker", "unblocker_ultra" and "auto"
unblocker	bool	False	Deprecated: use proxy_pool ("unblocker_ultra" for the Unblocker). The migration guide covers requests written before the proxy pools. asp is its alias; when both are supplied, asp wins
render_js	bool	False	Enable headless browser JavaScript rendering (+5 credits)
rendering_wait	int	None	Wait time in ms after page load (requires render_js=True)
rendering_stage	str	"complete"	Browser readiness stage: "complete" or "domcontentloaded" (requires render_js=True)
wait_for_selector	str	None	CSS/XPath selector to wait for (requires render_js=True)
js	str	None	JavaScript code to execute in the browser (auto-encoded)
js_scenario	list	None	List of browser actions (click, fill, scroll, wait, etc.)
auto_scroll	bool	None	Automatically scroll page during rendering for lazy-loaded content (requires render_js=True)
screenshots	dict	None	Capture screenshots: {"fullpage": "png"} or {"selector": ".element"} (requires render_js=True)
screenshot_flags	list[str]	None	Screenshot options: "load_images", "dark_mode", "block_banners", "high_quality", "print_media_format"
format	str	"raw"	Output format: "raw", "clean_html", "json", "markdown", "text"
format_options	list[str]	None	Format modifiers (markdown only): "no_images", "no_links", "only_content"
extract	dict	None	Raw extraction spec to apply on the response (encoded and sent as extract)
extraction_template	str	None	Name of a saved server-side extraction template
extraction_ephemeral_template	dict	None	Inline JSON extraction template used once (ephemeral:)
extraction_prompt	str	None	Natural language instructions for Extraction API
extraction_model	str	None	LLM model name to use for Extraction API
cache	bool	False	Enable response caching
cache_ttl	int	None	Cache time-to-live in seconds
cache_clear	bool	False	Clear any existing cached response when cache=True
dns	bool	False	Collect DNS records and timings in result.dns (slower)
ssl	bool	False	Collect SSL certificate details in result.ssl (slower)
debug	bool	False	Enable debug recording and extra metadata
raise_on_upstream_error	bool	True	Raise exceptions for upstream 4xx/5xx HTTP responses
correlation_id	str	None	Custom ID for request tracking across systems
tags	list[str]	None	Custom tags for request organization and analytics
lang	list[str]	None	Accept-Language values, e.g. ["en-US", "en"]
os	str	None	Override browser OS fingerprint, e.g. "windows", "macos"
webhook	str	None	Named webhook for async scrape completion callbacks
cost_budget	int	None	Max credits the Unblocker may spend on a request on the unblocker, unblocker_ultra and auto pools

ScrapeApiResponse

response = client.scrape(ScrapeConfig(url="https://httpbin.dev"))

response.content                      # Page content (HTML/JSON/text)
response.scrape_result                # Full result dict
response.status_code                  # Scrapfly API HTTP status code
response.scrape_result["status_code"] # Upsteam HTTP status code
response.headers                      # Response headers
response.context                      # Metadata about features used

Examples

Basic scrape

from scrapfly import ScrapflyClient, ScrapeConfig
import os

client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])

result = client.scrape(ScrapeConfig(
    url="https://httpbin.dev",
))
print(result.content)

Scrape with anti-bot bypass and geo-targeting

result = client.scrape(ScrapeConfig(
    url="https://httpbin.dev",
    proxy_pool="unblocker", # residential proxies and the Unblocker to bypass antibots
    country="us", # match the proxy with the domain country
))

Scrape with JavaScript rendering

result = client.scrape(ScrapeConfig(
    url="https://web-scraping.dev/products",
    render_js=True, # enable JavaScript rendering using a cloud browser
    rendering_wait=5000, # rendering wait for wait for
    wait_for_selector="div.product", # wait a for a selector to be present on the page
))

Scrape as markdown

result = client.scrape(ScrapeConfig(
    url="https://web-scraping.dev/products",
    format="markdown", # other supported formats are: json, text, clean_html 
))
print(result.content)

POST request with body

result = client.scrape(ScrapeConfig(
    url="https://httpbin.dev/anything",
    method="POST",
    headers={"Content-Type": "application/json"},
    body='{"query": "search term"}',
))
print(result.content)

JavaScript scenario (browser actions)

result = client.scrape(ScrapeConfig(
    url="https://web-scraping.dev/login",
    render_js=True,
    # browser control
    js_scenario=[
        {
            "fill":{
                "selector":"input[name='username']","clear":True,"value":"user123"
                }
            },{
            "fill":{
                "selector":"input[name='password']","clear":True,"value":"password"
                }
            },{
            "click":{
                "selector":"form > button[type='submit']"
                }
            },{
            "wait_for_navigation":{
                "timeout":5000
                }
            }
    ],
    # request headers
    headers={
        "cookie":"cookiesAccepted=true"
    }
))

# access the element under login
print(result.selector.css("div#secret-message::text").get())

Built-in Parsel selector

result = client.scrape(ScrapeConfig(
    url="https://web-scraping.dev/products",
    render_js=True
))

# access the built-in Parsel selector
selector = result.selector
for product in selector.css("div.product"):
    print("product name:", product.xpath(".//a/text()").get()) # using xpath
    print("product price:", product.css("div.price::text").get()) # using css

Concurrent scraping

import asyncio
from scrapfly import ScrapflyClient, ScrapeConfig

client = ScrapflyClient(key=os.environ["SCRAPFLY_API_KEY"])

urls = [
    "https://web-scraping.dev/products",
    "https://web-scraping.dev/products?page=2",
    "https://web-scraping.dev/products?page=3",
]

configs = [ScrapeConfig(url=url) for url in urls]

async def concurrent_scraping():
    results = []
    async for result in client.concurrent_scrape(configs):
        results.append(result)

    for result in results:
        print(result.content[:100])
        print("===========")

asyncio.run(concurrent_scraping())

Using sessions for stateful scraping

# First request: login
result = client.scrape(ScrapeConfig(
    url="https://web-scraping.dev/login",
    render_js=True,
    # browser control
    js_scenario=[
        {
            "fill":{
                "selector":"input[name='username']","clear":True,"value":"user123"
                }
            },{
            "fill":{
                "selector":"input[name='password']","clear":True,"value":"password"
                }
            },{
            "click":{
                "selector":"form > button[type='submit']"
                }
            },{
            "wait_for_navigation":{
                "timeout":5000
                }
            }
    ],
    # request headers
    headers={
        "cookie":"cookiesAccepted=true"
    },
    session="logged-in-session"
))

# Second request: access protected page with same session
result = client.scrape(ScrapeConfig(
    url="https://web-scraping.dev/login",
    session="logged-in-session"
))

# access the built-in selector
print(result.selector.css("div#secret-message::text").get())

Caching responses

result = client.scrape(ScrapeConfig(
    url="https://example.com",
    cache=True,
    cache_ttl=3600,  # Cache for 1 hour
))

Error Handling

from scrapfly import ScrapflyClient, ScrapeConfig
from scrapfly.errors import (
    ScrapflyError,
    UpstreamHttpClientError,
    UpstreamHttpServerError,
    ScrapflyProxyError,
    ScrapflyThrottleError,
)

try:
    result = client.scrape(ScrapeConfig(url="https://httpbin.dev", proxy_pool="unblocker"))
except ScrapflyThrottleError as e:
    print(f"Rate limited, retry after {e.retry_delay}s")
except UpstreamHttpClientError as e:
    print(f"Target returned 4xx: {e.message}")
except UpstreamHttpServerError as e:
    print(f"Target returned 5xx: {e.message}")
except ScrapflyProxyError as e:
    print(f"Proxy error: {e.message}")
except ScrapflyError as e:
    print(f"Scrapfly error: {e.message}")

A target or an anti-bot protection with a price of its own accepts proxy_pool="unblocker_ultra" and "auto", which add that price: basic and unblocker get ERR::PROXY::UNBLOCKER_ULTRA_REQUIRED (HTTP 422, not billed). Resend on unblocker_ultra:

try:
    result = client.scrape(ScrapeConfig(url=url, proxy_pool="unblocker"))
except ScrapflyError as e:
    if e.code != "ERR::PROXY::UNBLOCKER_ULTRA_REQUIRED":
        raise
    result = client.scrape(ScrapeConfig(url=url, proxy_pool="unblocker_ultra"))

Important Notes

• render_js=True is required for rendering_wait, wait_for_selector, js, js_scenario, and screenshots
• Pick the proxy pool a site needs: "basic" where datacenter proxies get through, "unblocker" where it needs residential proxies and the Unblocker, "unblocker_ultra" when the API answers ERR::PROXY::UNBLOCKER_ULTRA_REQUIRED, or "auto" to let the API pick it from the target's profile
• unblocker=True and its alias asp=True are deprecated: use proxy_pool ("unblocker_ultra" for the Unblocker). The migration guide covers requests written before the proxy pools
• Use format="markdown" for clean content accessible for LLMs
• Use session parameter to maintain state across multiple requests
• The concurrent_scrape method handles rate limiting automatically