SKILL.md
Twitter Scraping
Auth Setup
All scrapers require an authtoken cookie from x.com (DevTools -> Application -> Cookies -> copy authtoken value).
Node.js Scraper API
npm install xactions -- all functions from src/scrapers/index.js.
import { createBrowser, createPage, loginWithCookie, scrapeProfile } from 'xactions';
const browser = await createBrowser();
const page = await createPage(browser);
await loginWithCookie(page, AUTH_TOKEN);
const profile = await scrapeProfile(page, 'nichxbt');
await browser.close();
| Function |
Purpose |
scrapeProfile(page, username) |
Profile data (bio, followers, following) |
scrapeFollowers(page, username, {limit}) |
Follower list |
scrapeFollowing(page, username, {limit}) |
Following list |
scrapeTweets(page, username, {limit}) |
User's tweets |
searchTweets(page, query, {limit}) |
Search results |
scrapeHashtag(page, tag, {limit}) |
Hashtag tweets |
scrapeThread(page, tweetUrl) |
Thread tweets |
scrapeMedia(page, username, {limit}) |
Media URLs |
exportToJSON(data, path) / exportToCSV(data, path) |
File export |
Browser Console Scripts
Standalone IIFEs -- paste into DevTools console on x.com. No dependencies.
| Script |
File |
Navigate to |
| Download video |
src/scrapers/videoDownloader.js |
Tweet with video |
| Export bookmarks |
src/scrapers/bookmarkExporter.js |
x.com/i/bookmarks |
| Unroll thread |
src/scrapers/threadUnroller.js |
Any thread |
| Find viral tweets |
src/scrapers/viralTweets.js |
User profile |
| Scrape followers |
scripts/scrapeFollowers.js |
x.com/USER/followers |
| Scrape following |
scripts/scrapeFollowing.js |
x.com/USER/following |
| Scrape profile |
scripts/scrapeProfile.js |
User profile |
| Scrape search |
scripts/scrapeSearch.js |
Search results |
| Scrape hashtag |
scripts/scrapeHashtag.js |
x.com/hashtag/TAG |
| Scrape likes |
scripts/scrapeLikes.js |
x.com/USER/likes |
| Scrape likers |
scripts/scrapeLikers.js |
Any tweet |
| Scrape replies |
scripts/scrapeReplies.js |
Any tweet |
| Scrape media |
scripts/scrapeMedia.js |
User media tab |
| Scrape DMs |
scripts/scrapeDMs.js |
x.com/messages |
| Scrape notifications |
scripts/scrapeNotifications.js |
x.com/notifications |
| Scrape list |
scripts/scrapeList.js |
Any X List |
| Scrape bookmarks |
scripts/scrapeBookmarks.js |
x.com/i/bookmarks |
| Export to CSV |
scripts/exportToCSV.js |
After any scrape |
| Link scraper |
scripts/linkScraper.js |
Any timeline |
| Quote retweets |
scripts/scrapeQuoteRetweets.js |
Any tweet |
Interactive browser toolbox (recommended for console scraping)
For paste-in-console scraping, prefer scripts/twitter/scraper-toolbox.js: an on-page control panel (start/pause/stop, live filters, user exclusions, keyword rules) that captures X's own GraphQL responses, so counts are exact integers and long posts are untruncated. Exports JSON/CSV/Markdown/TXT/HTML and copies to clipboard. Works on any timeline: profile, search, list, likes, bookmarks, home. Docs: scripts/twitter/README-scraper-toolbox.md.
MCP Scraping Tools
All scrapers available via the MCP server for AI agents:
| Tool |
Description |
xgetprofile |
Scrape user profile |
xgetfollowers |
Get follower list |
xgetfollowing |
Get following list |
xgettweets |
Get user tweets |
xsearchtweets |
Search tweets |
xgetthread |
Unroll thread |
xdownloadvideo |
Extract video URL |
Notes
- Always call
browser.close() when done with Node.js scrapers
- Browser scripts stop on page navigation -- stay on the page while running
- All scripts support JSON and CSV export
- Increase
limit parameter for deeper scraping (default varies by function)
- Rate limiting is built in -- avoid reducing delays below defaults