X.com Tweet Scraper

English | 中文

English

📖 Description

A reliable tool to scrape X.com (Twitter) user tweets using GraphQL API with Cookie authentication. This project documents all the pitfalls encountered during development and provides working solutions.

Key Features:

• ✅ GraphQL API scraping (primary method)
• ✅ Playwright browser scraping (fallback method)
• ✅ Auto-save checkpoint (resume from breakpoints)
• ✅ Compatible with old & new API response structures
• ✅ Handles rate limiting (429 errors)
• ✅ Skips paid content (TweetPreviewDisplay)

📊 Workflow Diagram

🚀 Quick Start

1. Get Your Cookies

1. Login to X.com in your browser
2. Open Developer Tools (F12) → Application → Cookies
3. Copy these cookies to a cookies.json file:
   • auth_token
   • ct0
   • twid
   • _mb_tk

Cookie file format (cookies.json):

[
  {
    "name": "auth_token",
    "value": "your_auth_token_value",
    "domain": ".x.com",
    "path": "/",
    "httpOnly": true,
    "secure": true,
    "sameSite": "Lax"
  },
  {
    "name": "ct0",
    "value": "your_ct0_value",
    "domain": ".x.com",
    "path": "/",
    "httpOnly": false,
    "secure": true,
    "sameSite": "Lax"
  }
]

2. Install Dependencies

Python (API method):

pip install -r requirements.txt

Node.js (Playwright method):

npm install
npx playwright install chromium

3. Run the Scraper

API method (recommended):

python src/scrape_api.py --username elonmusk

Playwright method (fallback):

node src/scrape_playwright.js

⚠️ Common Pitfalls

1. GraphQL Variable Names

Wrong: screen_name (snake_case)
Right: screenName (camelCase)

2. CSRF Token

Wrong: Using auth_token value for x-csrf-token header
Right: Use ct0 cookie value

3. API Response Structure

X.com has two response structures (old & new). Always check both:

user_result = data['data'].get('user_result_by_screen_name') or data['data'].get('user')

4. Cursor Extraction

Wrong: content.item.content.value
Right: content.value

For details, see references/.

📁 Project Structure

x-tweet-scraper/
├── src/
│   ├── scrape_api.py           # GraphQL API scraper (recommended)
│   ├── scrape_playwright.js   # Playwright browser scraper (fallback)
│   └── utils/
│       ├── cookie_manager.py  # Cookie reading/validation
│       └── response_parser.py # Response parsing (compatible with old & new structures)
├── references/
│   ├── api_structure.md       # API response structure documentation
│   ├── error_codes.md         # Error code reference
│   └── field_mapping.md       # Old & new field name mapping
├── examples/
│   ├── basic_usage.py         # Basic usage example
│   └── advanced_usage.py      # Advanced usage (filtering, search)
└── tests/
    ├── test_scraper.py        # Unit tests
    └── sample_response.json   # Test API response

📝 License

MIT License

---

中文

📖 简介

一个使用 GraphQL API 和 Cookie 认证可靠抓取 X.com (Twitter) 用户推文的工具。本项目记录了开发过程中遇到的所有坑点，并提供可行的解决方案。

核心功能：

• ✅ GraphQL API 抓取（主要方法）
• ✅ Playwright 浏览器抓取（备用方法）
• ✅ 自动保存检查点（支持断点续传）
• ✅ 兼容新旧 API 响应结构
• ✅ 处理限流（429 错误）
• ✅ 跳过付费内容（TweetPreviewDisplay）

📊 工作流程图

🚀 快速开始

1. 获取 Cookie

1. 在浏览器中登录 X.com
2. 打开开发者工具（F12）→ Application → Cookies
3. 复制以下 Cookie 到 cookies.json 文件：
   • auth_token
   • ct0
   • twid
   • _mb_tk

Cookie 文件格式（cookies.json）：

[
  {
    "name": "auth_token",
    "value": "你的auth_token值",
    "domain": ".x.com",
    "path": "/",
    "httpOnly": true,
    "secure": true,
    "sameSite": "Lax"
  },
  {
    "name": "ct0",
    "value": "你的ct0值",
    "domain": ".x.com",
    "path": "/",
    "httpOnly": false,
    "secure": true,
    "sameSite": "Lax"
  }
]

2. 安装依赖

Python（API 方法）：

pip install -r requirements.txt

Node.js（Playwright 方法）：

npm install
npx playwright install chromium

3. 运行抓取脚本

API 方法（推荐）：

python src/scrape_api.py --username elonmusk

Playwright 方法（备用）：

node src/scrape_playwright.js

⚠️ 常见坑点

1. GraphQL 变量名

错误： screen_name（下划线）
正确： screenName（驼峰）

2. CSRF Token

错误： 使用 auth_token 的值作为 x-csrf-token header
正确： 使用 ct0 Cookie 的值

3. API 响应结构

X.com 有两种响应结构（新旧并存）。必须同时检查：

user_result = data['data'].get('user_result_by_screen_name') or data['data'].get('user')

4. Cursor 提取

错误： content.item.content.value
正确： content.value

详细说明请查看 references/ 目录。

📁 项目结构

x-tweet-scraper/
├── src/
│   ├── scrape_api.py           # GraphQL API 抓取脚本（推荐）
│   ├── scrape_playwright.js   # Playwright 浏览器抓取脚本（备用）
│   └── utils/
│       ├── cookie_manager.py  # Cookie 读取/验证
│       └── response_parser.py # 响应解析（兼容新旧结构）
├── references/
│   ├── api_structure.md       # API 响应结构文档
│   ├── error_codes.md         # 错误代码对照表
│   └── field_mapping.md       # 新旧字段名映射
├── examples/
│   ├── basic_usage.py         # 基础使用示例
│   └── advanced_usage.py      # 高级用法（过滤、搜索）
└── tests/
    ├── test_scraper.py        # 单元测试
    └── sample_response.json   # 测试用 API 响应

📝 许可证

MIT 许可证

---

⭐ Star History

If this project helps you, please consider giving it a star!

如果这个项目对你有帮助，请考虑给它一个星标！