twitter-scraping

Twitter Scraping

Auth Setup

All scrapers require an auth_token cookie from x.com (DevTools -> Application -> Cookies -> copy auth_token value).

Node.js Scraper API

npm install xactions -- all functions from src/scrapers/index.js.

import { createBrowser, createPage, loginWithCookie, scrapeProfile } from 'xactions';

const browser = await createBrowser();
const page = await createPage(browser);
await loginWithCookie(page, AUTH_TOKEN);

const profile = await scrapeProfile(page, 'nichxbt');
await browser.close();

Function	Purpose
scrapeProfile(page, username)	Profile data (bio, followers, following)
scrapeFollowers(page, username, {limit})	Follower list
scrapeFollowing(page, username, {limit})	Following list
scrapeTweets(page, username, {limit})	User's tweets
searchTweets(page, query, {limit})	Search results
scrapeHashtag(page, tag, {limit})	Hashtag tweets
scrapeThread(page, tweetUrl)	Thread tweets
scrapeMedia(page, username, {limit})	Media URLs
exportToJSON(data, path) / exportToCSV(data, path)	File export

Browser Console Scripts

Standalone IIFEs -- paste into DevTools console on x.com. No dependencies.

Script	File	Navigate to
Download video	src/scrapers/videoDownloader.js	Tweet with video
Export bookmarks	src/scrapers/bookmarkExporter.js	x.com/i/bookmarks
Unroll thread	src/scrapers/threadUnroller.js	Any thread
Find viral tweets	src/scrapers/viralTweets.js	User profile
Scrape followers	scripts/scrapeFollowers.js	x.com/USER/followers
Scrape following	scripts/scrapeFollowing.js	x.com/USER/following
Scrape profile	scripts/scrapeProfile.js	User profile
Scrape search	scripts/scrapeSearch.js	Search results
Scrape hashtag	scripts/scrapeHashtag.js	x.com/hashtag/TAG
Scrape likes	scripts/scrapeLikes.js	x.com/USER/likes
Scrape likers	scripts/scrapeLikers.js	Any tweet
Scrape replies	scripts/scrapeReplies.js	Any tweet
Scrape media	scripts/scrapeMedia.js	User media tab
Scrape DMs	scripts/scrapeDMs.js	x.com/messages
Scrape notifications	scripts/scrapeNotifications.js	x.com/notifications
Scrape list	scripts/scrapeList.js	Any X List
Scrape bookmarks	scripts/scrapeBookmarks.js	x.com/i/bookmarks
Export to CSV	scripts/exportToCSV.js	After any scrape
Link scraper	scripts/linkScraper.js	Any timeline
Quote retweets	scripts/scrapeQuoteRetweets.js	Any tweet

Interactive browser toolbox (recommended for console scraping)

For paste-in-console scraping, prefer scripts/twitter/scraper-toolbox.js: an on-page control panel (start/pause/stop, live filters, user exclusions, keyword rules) that captures X's own GraphQL responses, so counts are exact integers and long posts are untruncated. Exports JSON/CSV/Markdown/TXT/HTML and copies to clipboard. Works on any timeline: profile, search, list, likes, bookmarks, home. Docs: scripts/twitter/README-scraper-toolbox.md.

MCP Scraping Tools

All scrapers available via the MCP server for AI agents:

Tool	Description
x_get_profile	Scrape user profile
x_get_followers	Get follower list
x_get_following	Get following list
x_get_tweets	Get user tweets
x_search_tweets	Search tweets
x_get_thread	Unroll thread
x_download_video	Extract video URL

Notes

• Always call browser.close() when done with Node.js scrapers
• Browser scripts stop on page navigation -- stay on the page while running
• All scripts support JSON and CSV export
• Increase limit parameter for deeper scraping (default varies by function)
• Rate limiting is built in -- avoid reducing delays below defaults