# Reddit 内容抓取方案调研报告

> **任务编号**: T-mn5uz39y
> **调研目标**: Reddit 内容抓取技术方案、高赞帖子筛选、戏剧性内容识别、短剧素材提取
> **目标子版块**: r/AmITheAsshole, r/relationship_advice, r/funny
> **调研日期**: 2026-03-25
> **可信度**: 高（多源交叉验证）

---

## 1. Reddit API 方案 vs 网页爬虫方案

### 1.1 方案对比总览

| 维度 | 官方 API (PRAW) | JSON 端点 | Old Reddit | 浏览器自动化 |
|------|-----------------|-----------|------------|--------------|
| **合规性** | ✅ 完全合规 | ⚠️ 灰色地带 | ⚠️ 可能违反 ToS | ❌ 高风险 |
| **认证要求** | OAuth 2.0 | 无需认证 | 无需认证 | 模拟登录 |
| **速率限制** | 100 QPM (免费) | 无明确限制 | 无明确限制 | 依赖实现 |
| **数据完整度** | 高 | 中 | 高 | 高 |
| **开发成本** | 中 | 低 | 中 | 高 |
| **反爬风险** | 无 | 低 | 中 | 高 |
| **单次请求限制** | 1000 posts | 100 posts | 500 comments | 无限制 |
| **典型耗时(1000条)** | ~20分钟 | ~5分钟 | ~3分钟 | ~9分钟 |

### 1.2 官方 API (PRAW) 详解

**适用场景**: 需要写操作、长期自动化、合规要求高的项目

#### 认证方式
```python
import praw

reddit = praw.Reddit(
    client_id="YOUR_CLIENT_ID",
    client_secret="YOUR_CLIENT_SECRET",
    user_agent="DramaScraper/1.0 by u/username",
    redirect_uri="http://localhost:8080"
)
```

#### 免费版限制
- **100 queries per minute (QPM)** - 按 10 分钟窗口平均计算
- **仅限非商业用途** (个人项目、学术研究、开源工具)
- 需要 OAuth 2.0 认证
- 无法访问 NSFW 内容（2023年7月5日后限制）
- 历史数据访问受限

#### 付费版定价
| 级别 | 价格 | 速率限制 |
|------|------|----------|
| Free | $0 | 100 QPM (非商业) |
| Commercial | $0.24 / 1000 API calls | 按量计费 |
| Premium | ~$12,000/年起 | 100-1000 QPM |
| Enterprise | 定制 | 1000+ QPM |

#### Rate Limit Headers
```
X-Ratelimit-Used: 当前窗口已使用请求数
X-Ratelimit-Remaining: 剩余请求数
X-Ratelimit-Reset: 限制重置倒计时（秒）
```

### 1.3 JSON 端点方案（推荐用于只读）

**适用场景**: 快速原型、小批量数据、预算有限

#### 端点模板
```
# 帖子列表
https://www.reddit.com/r/{subreddit}/{sort}.json?limit=100&t={time_period}

# 单帖详情
https://www.reddit.com/r/{subreddit}/comments/{post_id}.json?limit=100

# 用户历史
https://www.reddit.com/user/{username}/submitted.json
```

#### Sort 参数
- `hot` - 热门（Reddit 热榜算法）
- `new` - 最新
- `top` - 最高赞
- `rising` - 上升中
- `controversial` - 争议

#### Time 参数（仅 top/controversial）
- `hour` - 过去1小时
- `day` - 过去24小时
- `week` - 过去一周
- `month` - 过去一月
- `year` - 过去一年
- `all` - 全部时间

### 1.4 Old Reddit 方案（推荐用于评论抓取）

**适用场景**: 需要大量评论、绕过 API 限制

```
https://old.reddit.com/r/{subreddit}/comments/{post_id}/?limit=500
```

优点:
- 单次请求最多 500 条评论
- 无需 JavaScript 渲染
- HTML 结构简单易于解析

### 1.5 推荐方案

**混合策略**:
1. **主要使用 JSON 端点** - 快速获取帖子列表和基础数据
2. **Old Reddit 补充** - 当需要深度评论树时使用
3. **PRAW 作为备选** - 当需要写操作或遇到限制时切换

---

## 2. 高赞帖子筛选策略

### 2.1 Reddit Hot 算法解析

**核心公式** (Amir Salihefendic, 2010):
```
h = log₁₀(max(|u-d|, 1)) + sign(u-d) × (t - t₀)/45000
```

参数:
- `h` = Hot score
- `u` = upvotes 数量
- `d` = downvotes 数量
- `t` = Unix 时间戳
- `t₀` = 1134028003 (2005-12-08)
- `45000` = 12.5 小时的秒数

**关键洞察**:
- **对数权重**: 前 10 个 upvote = 接下来的 100 个 = 再接下来的 1000 个
- **时间衰减**: 每 12.5 小时，帖子自然下沉 1 个单位
- **首小时决定论**: 前 10 分钟的 10 个 upvote > 2 小时内的 100 个

### 2.2 筛选策略矩阵

| 目标 | Sort | Time | 筛选条件 |
|------|------|------|----------|
| **爆款内容** | `top` | `month/year/all` | score > 10000, upvote_ratio > 0.85 |
| **趋势内容** | `hot` | - | comment_count > 100, rising 速度 |
| **争议内容** | `controversial` | `week/month` | upvote_ratio 0.3-0.7, comments > 500 |
| **新热内容** | `rising` | - | upvote_velocity > 10/min |
| **长尾优质** | `top` | `all` | 手动按 score 降序 |

### 2.3 高互动指标阈值

```python
HIGH_ENGAGEMENT_THRESHOLDS = {
    "elite_tier": {
        "score": 10000,
        "comment_count": 2000,
        "upvote_ratio": 0.90
    },
    "high_tier": {
        "score": 5000,
        "comment_count": 1000,
        "upvote_ratio": 0.85
    },
    "good_tier": {
        "score": 1000,
        "comment_count": 500,
        "upvote_ratio": 0.80
    },
    "rising_tier": {
        "score": 100,
        "comment_count": 50,
        "growth_rate": 2.0  # 每小时翻倍
    }
}
```

---

## 3. 戏剧性内容识别指标

### 3.1 争议度指标

**Upvote Ratio 分类**:
| 类别 | 范围 | 特征 |
|------|------|------|
| **Consensus** | 0.80-1.00 | 广泛认同，争议低 |
| **Mild Debate** | 0.70-0.80 | 轻微分歧 |
| **Controversial** | 0.30-0.70 | 高度争议，两极分化 |
| **Heavily Downvoted** | 0.00-0.30 | 被踩严重，可能是钓鱼或极端观点 |

**争议度得分**:
```python
def controversy_score(upvotes, downvotes):
    """
    越接近 0.5 越具争议
    """
    total = upvotes + downvotes
    if total == 0:
        return 0
    ratio = upvotes / total
    return 1 - abs(ratio - 0.5) * 2  # 0-1, 1=完全争议
```

### 3.2 情绪关键词库

```python
DRAMA_KEYWORDS = {
    "conflict_triggers": [
        "aita", "asshole", "wrong", "fault", "blame",
        "cheated", "betrayed", "lied", "secret",
        "divorce", "breakup", "dumped", "ghosted",
        "kicked out", "cut off", "disowned",
        "inheritance", "will", "money", "debt",
        "wedding", "bridezilla", "groomzilla"
    ],
    "emotional_intensity": [
        "furious", "devastated", "heartbroken", "livid",
        "panic", "anxiety", "depression", "trauma",
        "suicidal", "therapy", "counseling",
        "cried", "screaming", "yelling", "fight"
    ],
    "relationship_types": [
        "boyfriend", "girlfriend", "husband", "wife",
        "ex", "mother-in-law", "father-in-law",
        "stepmom", "stepdad", "sibling", "sister", "brother",
        "best friend", "roommate", "coworker", "boss"
    ],
    "drama_markers": [
        "update", "plot twist", "turns out", "found out",
        "caught", "evidence", "proof", "messages",
        "dna test", "paternity", "surprise"
    ]
}
```

### 3.3 戏剧性评分算法

```python
def calculate_drama_score(post):
    """
    综合戏剧性评分 (0-100)
    """
    score = 0

    # 争议度权重 (25%)
    controversy = controversy_score(post.ups, post.downs)
    score += controversy * 25

    # 互动密度权重 (20%)
    engagement = min(post.num_comments / 1000, 1.0)
    score += engagement * 20

    # 关键词匹配 (30%)
    text = f"{post.title} {post.selftext}".lower()
    keyword_matches = sum(1 for kw in all_keywords if kw in text)
    score += min(keyword_matches / 5, 1.0) * 30

    # 更新帖加权 (15%)
    if any(marker in post.title.lower() for marker in ["update", "final update"]):
        score += 15

    # 长度适宜度 (10%) - 太短没内容，太长没人看
    word_count = len(post.selftext.split())
    if 200 <= word_count <= 2000:
        score += 10
    elif 100 <= word_count < 200:
        score += 5

    return min(score, 100)
```

### 3.4 AITA 特定指标

针对 r/AmITheAsshole 的特别指标:

| 指标 | 识别方式 | 权重 |
|------|----------|------|
| **判定结果** | 标题中的 NTA/YTA/ESH/NAH/INFO | 高 |
| **投票分布** | top_comment 的标签统计 | 中 |
| **情节复杂度** | 人物数量、时间跨度、冲突层数 | 中 |
| **道德困境** | 两难选择描述、价值冲突 | 高 |

---

## 4. 短剧素材提取维度

### 4.1 场景设定 (Scene Setting)

**提取字段**:
```json
{
  "scene_setting": {
    "primary_location": "主要发生地点",
    "time_context": "时间背景（节日、人生阶段）",
    "social_context": "社交背景（婚礼、葬礼、聚会）",
    "emotional_atmosphere": "情绪氛围（紧张、尴尬、欢乐）",
    "visual_elements": ["可视觉化的关键物件/场景"],
    "budget_friendly": "是否适合低成本拍摄"
  }
}
```

**常见 AITA 场景**:
- 家庭聚会/节日聚餐
- 婚礼/订婚仪式
- 住房相关（室友、房产继承）
- 职场冲突
- 育儿相关
- 医疗费用相关

### 4.2 人物关系 (Character Relationships)

**提取字段**:
```json
{
  "characters": [
    {
      "role": "角色身份（OP/当事人/旁观者）",
      "relation_to_op": "与主角的关系",
      "traits": ["性格特征"],
      "agenda": "人物目标/动机",
      "arc": "人物弧线（如果有）"
    }
  ],
  "relationship_web": {
    "central_conflict": "核心冲突关系",
    "alliances": ["联盟关系"],
    "tensions": ["紧张关系"]
  }
}
```

**典型人物配置**:
- **OP (Original Poster)**: 故事叙述者，通常是第一视角
- **Antagonist**: 对立角色（室友、前任、亲戚）
- **Enabler/Ally**: 支持方（伴侣、朋友）
- **Mediator**: 调解者（很少出现，但可增加戏剧张力）

### 4.3 冲突核心 (Core Conflict)

**提取字段**:
```json
{
  "conflict": {
    "type": "冲突类型",
    "inciting_incident": "导火索事件",
    "stakes": "利害关系/后果",
    "moral_dilemma": "道德困境描述",
    "escalation": "冲突升级过程",
    "climax": "高潮点"
  }
}
```

**冲突类型分类**:
| 类型 | 描述 | 示例 |
|------|------|------|
| **资源争夺** | 金钱、房产、物品归属 | 遗产分配、房租分摊 |
| **边界侵犯** | 个人空间/时间被侵占 | 未经同意进入房间 |
| **价值观冲突** | 道德观念分歧 | 育儿方式、生活方式 |
| **信任破裂** | 欺骗、背叛 | 出轨、隐瞒 |
| **角色期望** | 对某人应有行为的期待 | 伴娘义务、父母责任 |

### 4.4 精彩对话 (Dialogue Highlights)

**提取字段**:
```json
{
  "dialogue": {
    "quotable_lines": ["金句/爆点台词"],
    "confrontation_exchanges": ["对峙对话片段"],
    "emotional_peaks": ["情绪高潮对话"],
    "revelation_moments": ["真相揭示时刻"]
  }
}
```

**提取规则**:
- 直接引用原文的引号内容
- 识别 `"You are the asshole"` 等判定性语句
- 提取评论区的热门引用/模仿
- 记录讽刺、双关、反转语句

### 4.5 情感转折 (Emotional Turning Points)

**提取字段**:
```json
{
  "emotional_arc": {
    "opening_hook": "开篇钩子",
    "setup_emotion": "初始情绪状态",
    "complications": ["复杂化节点"],
    "turning_points": [
      {
        "trigger": "触发事件",
        "emotion_shift": "情绪变化",
        "narrative_impact": "叙事影响"
      }
    ],
    "resolution_type": "结局类型（和解/决裂/开放）",
    "reader_catharsis": "读者情感宣泄点"
  }
}
```

**转折类型**:
1. **Revelation (揭示)**: 发现隐藏信息
2. **Reversal (反转)**: 立场/形势逆转
3. **Escalation (升级)**: 冲突白热化
4. **Climax (高潮)**: 决定性时刻
5. **Resolution (解决)**: 最终结局

### 4.6 AITA 特有维度

```json
{
  "aita_specific": {
    "judgment": "最终判定 (NTA/YTA/ESH/NAH/INFO)",
    "top_comment_consensus": "热评共识",
    "vote_distribution": {
      "nta_percentage": "NTA 比例",
      "yta_percentage": "YTA 比例"
    },
    "moral_ambiguity": "道德模糊程度",
    "entertainment_value": "娱乐价值评分",
    "adaptation_potential": "改编潜力评分"
  }
}
```

---

## 5. 输出格式规范

### 5.1 标准化 JSON Schema

```json
{
  "metadata": {
    "source": "reddit",
    "subreddit": "AmITheAsshole",
    "scraped_at": "2026-03-25T10:30:00Z",
    "scraper_version": "1.0.0"
  },
  "post": {
    "id": "t3_abc123",
    "title": "原始标题",
    "author": "用户名",
    "created_utc": 1711300000,
    "url": "https://reddit.com/r/...",
    "permalink": "/r/AmITheAsshole/comments/...",
    "selftext": "正文内容",
    "score": 15420,
    "upvote_ratio": 0.87,
    "num_comments": 3400,
    "is_original_content": true,
    "link_flair_text": "判定标签",
    "edited": false
  },
  "analytics": {
    "drama_score": 78.5,
    "controversy_score": 0.35,
    "engagement_tier": "elite",
    "keyword_matches": ["冲突", "背叛"],
    "sentiment": "negative_intense"
  },
  "drama_elements": {
    "scene_setting": { },
    "characters": [ ],
    "conflict": { },
    "dialogue": { },
    "emotional_arc": { },
    "aita_specific": { }
  },
  "adaptation_notes": {
    "logline": "一句话故事线",
    "target_duration": "3-5分钟",
    "genre_tags": ["家庭伦理", "反转"],
    "production_difficulty": "低",
    "similar_works": ["参考作品"]
  }
}
```

### 5.2 数据源分类

**Elite 数据源** (直接用于创作):
- r/AmITheAsshole Top All Time (score > 10k)
- r/relationship_advice Top Month (score > 5k)
- 带 "UPDATE" 标签的热门帖

**Average 数据源** (训练/参考):
- r/AmITheAsshole Top Week (score 1k-10k)
- r/funny 高互动帖
- 有争议判定 (upvote_ratio 0.5-0.7) 的帖子

**Broad 数据源** (趋势分析):
- 实时 Rising 内容
- 每日新增帖子的元数据分析

---

## 6. 技术实现建议

### 6.1 Python 代码示例

```python
import requests
import json
import time
from datetime import datetime, timedelta
from typing import List, Dict, Optional

class RedditScraper:
    def __init__(self, user_agent: str = "DramaScraper/1.0"):
        self.headers = {
            "User-Agent": user_agent
        }
        self.base_url = "https://www.reddit.com"

    def fetch_posts(
        self,
        subreddit: str,
        sort: str = "hot",
        time_period: Optional[str] = None,
        limit: int = 100,
        after: Optional[str] = None
    ) -> List[Dict]:
        """
        获取帖子列表
        """
        url = f"{self.base_url}/r/{subreddit}/{sort}.json"
        params = {"limit": min(limit, 100)}

        if time_period and sort in ["top", "controversial"]:
            params["t"] = time_period
        if after:
            params["after"] = after

        response = requests.get(url, headers=self.headers, params=params)
        response.raise_for_status()

        data = response.json()
        posts = data["data"]["children"]

        return [self._extract_post_data(post["data"]) for post in posts]

    def _extract_post_data(self, post: Dict) -> Dict:
        """
        提取关键字段
        """
        return {
            "id": post.get("id"),
            "title": post.get("title"),
            "author": post.get("author"),
            "created_utc": post.get("created_utc"),
            "url": post.get("url"),
            "permalink": post.get("permalink"),
            "selftext": post.get("selftext", ""),
            "score": post.get("score"),
            "ups": post.get("ups"),
            "downs": post.get("downs"),
            "upvote_ratio": post.get("upvote_ratio"),
            "num_comments": post.get("num_comments"),
            "is_original_content": post.get("is_original_content"),
            "link_flair_text": post.get("link_flair_text"),
            "edited": post.get("edited"),
            "awards": post.get("total_awards_received", 0)
        }

    def calculate_drama_score(self, post: Dict) -> float:
        """
        计算戏剧性得分
        """
        score = 0.0

        # 争议度 (25%)
        upvote_ratio = post.get("upvote_ratio", 1.0)
        controversy = 1 - abs(upvote_ratio - 0.5) * 2
        score += controversy * 25

        # 互动密度 (20%)
        comments = post.get("num_comments", 0)
        score += min(comments / 2000, 1.0) * 20

        # 关键词匹配 (30%)
        title = post.get("title", "").lower()
        text = post.get("selftext", "").lower()
        full_text = f"{title} {text}"

        drama_keywords = [
            "aita", "asshole", "cheated", "betrayed", "divorce",
            "wedding", "inheritance", "secret", "lied", "update",
            "plot twist", "turns out", "found out", "caught"
        ]
        matches = sum(1 for kw in drama_keywords if kw in full_text)
        score += min(matches / 5, 1.0) * 30

        # 更新帖加权 (15%)
        if "update" in title:
            score += 15

        # 长度适宜度 (10%)
        word_count = len(text.split())
        if 200 <= word_count <= 2000:
            score += 10
        elif 100 <= word_count < 200:
            score += 5

        return min(score, 100)

    def fetch_with_pagination(
        self,
        subreddit: str,
        target_count: int = 1000,
        **kwargs
    ) -> List[Dict]:
        """
        分页获取大量帖子
        """
        all_posts = []
        after = None

        while len(all_posts) < target_count:
            posts = self.fetch_posts(
                subreddit=subreddit,
                limit=100,
                after=after,
                **kwargs
            )

            if not posts:
                break

            all_posts.extend(posts)
            after = f"t3_{posts[-1]['id']}"

            # 礼貌延迟
            time.sleep(2)

        return all_posts[:target_count]
```

### 6.2 反爬策略

```python
# 请求间隔
MIN_DELAY = 2.0  # 秒
MAX_DELAY = 5.0

# User-Agent 轮换
USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36",
]

# 失败重试
MAX_RETRIES = 3
RETRY_DELAY = 30
```

---

## 7. 数据源 URL 模板

### 7.1 目标子版块

```
r/AmITheAsshole      - 道德困境、人际冲突
r/relationship_advice - 恋爱关系、情感问题
r/funny              - 轻松幽默、生活趣事
```

### 7.2 API 端点模板

```python
URL_TEMPLATES = {
    # JSON 端点
    "json_posts": "https://www.reddit.com/r/{subreddit}/{sort}.json",
    "json_comments": "https://www.reddit.com/r/{subreddit}/comments/{post_id}.json",

    # Old Reddit
    "old_reddit_post": "https://old.reddit.com{permalink}",
    "old_reddit_comments": "https://old.reddit.com/r/{subreddit}/comments/{post_id}/?limit=500",

    # 网页版
    "web_post": "https://www.reddit.com{permalink}",
    "web_subreddit": "https://www.reddit.com/r/{subreddit}/{sort}",

    # 特定筛选
    "top_all_time": "https://www.reddit.com/r/{subreddit}/top.json?t=all&limit=100",
    "top_year": "https://www.reddit.com/r/{subreddit}/top.json?t=year&limit=100",
    "top_month": "https://www.reddit.com/r/{subreddit}/top.json?t=month&limit=100",
    "controversial_week": "https://www.reddit.com/r/{subreddit}/controversial.json?t=week&limit=100",
    "rising": "https://www.reddit.com/r/{subreddit}/rising.json?limit=100"
}
```

### 7.3 字段映射表

| JSON 字段 | 内部名称 | 数据类型 | 描述 |
|-----------|----------|----------|------|
| `id` | `post_id` | string | 帖子唯一ID |
| `title` | `title` | string | 标题 |
| `author` | `author` | string | 作者用户名 |
| `created_utc` | `created_at` | int | UTC时间戳 |
| `selftext` | `content` | string | 正文内容 |
| `score` | `score` | int | 净得分(up-down) |
| `ups` | `upvotes` | int | 点赞数 |
| `downs` | `downvotes` | int | 点踩数(通常不显示) |
| `upvote_ratio` | `upvote_ratio` | float | 点赞比例 |
| `num_comments` | `comment_count` | int | 评论数 |
| `permalink` | `permalink` | string | 相对链接 |
| `url` | `url` | string | 外部链接 |
| `is_original_content` | `is_oc` | bool | 是否原创 |
| `link_flair_text` | `flair` | string | 标签/分类 |
| `edited` | `is_edited` | bool/int | 是否编辑过 |
| `total_awards_received` | `awards` | int | 获奖数 |
| `num_crossposts` | `crosspost_count` | int | 转发数 |

---

## 8. 数据质量与验证

### 8.1 数据清洗规则

```python
CLEANING_RULES = {
    # 移除内容
    "remove_deleted": True,      # 删除 [deleted] 作者的内容
    "remove_removed": True,      # 删除 [removed] 内容
    "remove_short": 50,          # 移除少于50字的帖子
    "remove_low_effort": True,   # 移除低质量内容

    # 文本处理
    "normalize_whitespace": True,
    "remove_tracking_params": True,  # 移除URL跟踪参数
    "decode_html_entities": True,
}
```

### 8.2 多源验证策略

| 验证项 | 方法 | 通过标准 |
|--------|------|----------|
| 数据完整性 | 字段非空检查 | 核心字段100%填充 |
| 时间一致性 | 时间戳范围验证 | 不超过当前时间 |
| 数值合理性 | 范围检查 | score >= 0, ratio 0-1 |
| 重复检测 | ID去重 | 无重复记录 |
| 内容新鲜度 | 时间戳检查 | 根据需求设定阈值 |

---

## 9. 参考资源

### 9.1 数据来源

- [Reddit Data API Wiki](https://support.reddithelp.com/hc/en-us/articles/16160319875092-Reddit-Data-API-Wiki) - 官方 API 文档
- [PRAW Documentation](https://praw.readthedocs.io/) - Python Reddit API Wrapper
- [Reddit Algorithm Explained](https://upvote.net/blog/reddit-algorithm) - 算法解析
- [Reddit Scraping Guide 2025](https://roundproxies.com/blog/reddit/) - 抓取技术指南

### 9.2 学术参考

- Salihefendic, A. (2010). "How Reddit's Algorithm Works"
- ArXiv 2411.11581 - OASIS: Open Agent Social Interaction Simulations
- ArXiv 2503.03500 - TDA-Based Controversy Detection in Reddit Data

---

## 10. 总结与建议

### 推荐技术栈

1. **首选方案**: JSON 端点 + 定时轮询
   - 开发成本低
   - 无需认证
   - 足够满足大部分需求

2. **备选方案**: PRAW API
   - 需要写操作时启用
   - 更高的合规性保障

3. **深度分析**: Old Reddit 爬取
   - 评论树完整抓取
   - 配合代理池使用

### 核心筛选指标优先级

1. **upvote_ratio** (0.3-0.7) - 识别争议内容
2. **num_comments** (>1000) - 高互动 = 高戏剧性潜力
3. **keyword_matches** - 情绪/冲突关键词
4. **update_flag** - 更新帖通常更有故事性
5. **score** (>5000) - 大众认可度

### 下一步行动

1. 实现基础爬虫 (JSON 端点)
2. 建立数据库存储结构
3. 开发 drama_score 评分算法
4. 构建 AITA 特定的字段提取器
5. 测试小规模数据抓取并验证质量

---

*报告生成时间: 2026-03-25*
*置信度: 高*
*版本: 1.0*