scrapling

Scrapling

自适应、高性能的 Python 网页抓取库。速度比 BeautifulSoup 快 774 倍。网站结构变化时可自动重新定位元素。

主要功能： HTTP/浏览器抓取、绕过 Cloudflare、自适应选择器、Spider 爬取框架、代理轮换、CLI 工具、面向 AI Agent 的 MCP 服务。

v0.4 重大变更： 已移除 css_first()/xpath_first()——请改用 .css('.sel').first 或 .css('.sel').get()。css('::text') 和 css('::attr()') 现在返回 Selector 对象（而非 TextHandler）。Response.body 始终为 bytes。

快速开始

from scrapling.fetchers import Fetcher

page = Fetcher.get('https://example.com')
titles = page.css('.title::text')
links = page.css('a.link::attr(href)').getall()

# Response metadata
print(page.status, page.headers, page.cookies)

全局配置

Fetcher.configure(adaptive=True, encoding="utf-8", keep_comments=False)

抓取器选择

需求	使用	原因
静态 HTML	Fetcher	速度最快，支持 TLS 伪装
JavaScript 内容	DynamicFetcher	Playwright 浏览器
Cloudflare/反机器人	StealthyFetcher	Camoufox + stealth
多页面爬取	Spider	异步爬取框架

完整的抓取器选项请参见 references/fetchers.md。

常见模式

基本抓取

from scrapling.fetchers import Fetcher

page = Fetcher.get('https://quotes.example.com')

for quote in page.css('.quote'):
    text = quote.css('.text').first.text
    author = quote.css('.author').first.text
    print(f'{text} - {author}')

JavaScript 渲染的内容

from scrapling.fetchers import DynamicFetcher

page = DynamicFetcher.fetch(
    'https://spa-app.com',
    headless=True,
    network_idle=True,
    wait_selector='.content-loaded'
)

绕过 Cloudflare

from scrapling.fetchers import StealthyFetcher

page = StealthyFetcher.fetch(
    'https://protected-site.com',
    solve_cloudflare=True,
    humanize=True,
    headless=True
)

使用 Cookie 的会话

from scrapling.fetchers import FetcherSession

with FetcherSession(impersonate='chrome') as session:
    session.get('https://site.com/login')
    dashboard = session.get('https://site.com/dashboard')

多页面爬取（Spider）

from scrapling.spiders import Spider, Response

class MySpider(Spider):
    name = "demo"
    start_urls = ["https://example.com/"]

    async def parse(self, response: Response):
        for item in response.css('.product'):
            yield {"title": item.css('h2::text').get()}

MySpider().start()

完整的 Spider 框架请参见 references/spiders.md。

代理轮换

from scrapling.engines.toolbelt import ProxyRotator

rotator = ProxyRotator(["http://proxy1:8080", "http://proxy2:8080"])
page = Fetcher.get('https://example.com', proxy_rotator=rotator)

自适应选择器（自我修复）

from scrapling import Selector

# Save element properties on first visit (storage defaults to SQLite)
selector = Selector(html, url='https://example.com', adaptive=True)
products = selector.css('.product-card', auto_save=True)

# On subsequent runs, relocate even if structure changed
products = selector.css('.product-card', adaptive=True)

元素选择

# CSS (recommended)
page.css('.class')              # Multiple → Selectors list
page.css('.class').first        # Single (safe, returns None if missing)
page.css('.class').get()        # First as TextHandler (alias: extract_first)
page.css('.title::text')        # Text extraction → Selector objects
page.css('a::attr(href)')       # Attribute → Selector objects

# XPath
page.xpath('//div[@id="main"]')
page.xpath('//h1').first

# BeautifulSoup-style
page.find('div', class_='content')
page.find_all('a', attrs={'data-type': 'link'})

# Text search (no tag= param — filter by tag separately if needed)
page.find_by_text('Add to Cart')
page.css('button').filter(lambda el: el.text == 'Add to Cart').first

注意： .first 和 .last 是安全的访问器——如果无法访问，它们会返回 None，而不是引发 IndexError。

导航和高级选择请参见 references/selectors.md。

元素导航

el = page.css('.target').first
el.parent                # Parent element
el.children              # Child elements
el.next                  # Next sibling
el.previous              # Previous sibling
el.siblings              # All siblings
el.text                  # Inner text
el.text.clean()          # Whitespace-normalized text
el.attrib['href']        # Attribute access

解析现有 HTML

from scrapling.parser import Selector

html = '<div class="item">Content</div>'
page = Selector(html)
content = page.css('.item').first.text

CLI 用法

# Interactive shell
scrapling shell

# Extract from URL (output format by extension: .html, .md, .txt)
scrapling extract get 'https://example.com' output.md --css-selector '.content'

# Dynamic content with browser
scrapling extract fetch 'https://spa.com' out.html --network-idle

# Cloudflare bypass
scrapling extract stealthy-fetch 'https://site.com' out.html --solve-cloudflare

所有命令和选项请参见 references/cli.md。

错误处理

from scrapling.fetchers import Fetcher

try:
    page = Fetcher.get('https://example.com', timeout=10)  # seconds
    element = page.css('.target').first
    if element:
        print(element.text)
    else:
        print('Element not found')
except Exception as e:
    print(f'Request failed: {e}')

MCP 服务（AI 集成）

支持 AI Agent（Claude Desktop/Code）通过自然语言进行网页抓取：

pip install "scrapling[ai]"
scrapling install

工具：get、bulk_get、fetch、bulk_fetch、stealthy_fetch、bulk_stealthy_fetch

配置请参见 references/mcp.md。

资源

• references/fetchers.md - 完整的抓取器选项和配置
• references/selectors.md - 选择器方法和元素导航
• references/spiders.md - Spider 爬取框架
• references/cli.md - CLI 命令参考
• references/adaptive.md - 自我修复选择器系统
• references/mcp.md - 用于 AI 集成的 MCP 服务
• scripts/scrape_list.py - 用于将列表项提取到 JSON/CSV 的 CLI 工具