mcpField
X
x-scraper

x-scraper

github.com/aatroooxData & databases368 installsUpdated 2w ago
Repository

About x-scraper

x-scraper is a Python skill for extracting recent posts and engagement data from X.com user timelines. It uses authenticated cookies and saves results as JSON for analysis.

Key features

  1. 01Scrape user timelines, including replies
  2. 02Capture post text, links, and timestamps
  3. 03Collect views, likes, reposts, and replies
  4. 04Save results as JSON
  5. 05Convert Cookie-Editor data for Playwright

Use cases

  1. 01Gather recent posts from an X.com account
  2. 02Review a user's posts and engagement metrics
  3. 03Prepare timeline data for analysis

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/aatrooox/blog.zzao.club --skill x-scraper

After installing, type/x-scraperin Claude Code to confirm it loaded.

Bundled files

References and scripts installed along with SKILL.md.

references/
setup.md
troubleshooting.md
usage.md
scripts/
convert_cookies.py
scraper.py

Details

From the project README

X.com Post Scraper

Extracts recent posts from X.com users with full engagement data using authenticated cookies.

Quick Start

Basic command:

cd .opencode/skills/x-scraper/scripts
python3 scraper.py <username> [count]

Example:

python3 scraper.py example_user 15

Output: /tmp/x_{username}_posts.json

Prerequisites

Before first use, verify environment requirements:

  1. Python 3.11+: Check with python3 --version
  2. Playwright: Check with python3 -c "import playwright"
  3. Cookie file: Check with ls /tmp/x_cookies_pw.json

If any prerequisite is missing, see references/setup.md for detailed installation and configuration guide.

Common Workflows

First-time setup

See references/setup.md for complete environment configuration.

Daily scraping
python3 scraper.py <username> [count]
Custom cookie file
python3 scraper.py <username> [count] --cookie-file /path/to/cookies.json
Troubleshooting

If scraper fails, see references/troubleshooting.md for common issues and solutions.

Output Format

{
  "index": 1,
  "username": "example_user",
  "postId": "1234567890123456789",
  "publishTime": "2025-12-03T18:28:32.000Z",
  "postLink": "https://x.com/example_user/status/1234567890123456789",
  "textContent": "Post text content...",
  "views": "471K",
  "likes": "1.1K",
  "retweets": "153",
  "replies": "44"
}

Key fields:

  • postLink - Direct URL to post
  • publishTime - ISO 8601 timestamp
  • views/likes/retweets/replies - Abbreviated metrics (K, M)

When to Use This Skill

Trigger when user requests:

  • "整理 @某人 最近的发言"
  • "看看某人在X上说了什么"
  • "Scrape X.com posts from @username"
  • "Get latest tweets from user"
  • "Analyze X user's recent posts"

Available Scripts

scraper.py - Main scraper
python3 scraper.py <username> [count] [--cookie-file <path>]
  • Scrapes user timeline with replies
  • Default count: 10 posts
  • Default cookie: /tmp/x_cookies_pw.json
convert_cookies.py - Cookie converter
python3 convert_cookies.py <input-file> [output-file]
  • Converts Cookie-Editor JSON to Playwright format
  • Required before first scraping

Reference Documents

  • setup.md - Complete environment setup guide (Python, Playwright, cookies)
  • troubleshooting.md - Error diagnosis and solutions
  • usage.md - Detailed usage examples and advanced options

Limitations

  • Requires X.com authentication cookies
  • Cookies expire (~7 days), need re-export
  • Rate limits may apply
  • Cannot access private/protected accounts

Comments

Markdown supported