# Reddit Scraper Benchmark 评估基准

> 版本: 1.0.0
> 适用: Reddit 内容抓取与短剧素材提取技能评估

---

## 1. 评估维度概览

| 维度 | 权重 | 说明 |
|------|------|------|
| **抓取准确性** | 25% | 数据抓取的完整性和准确性 |
| **内容筛选质量** | 30% | 戏剧性内容识别的准确度 |
| **提取完整性** | 25% | 短剧素材提取的全面性 |
| **JSON格式规范性** | 20% | 输出格式符合规范程度 |

---

## 2. 评分标准 (0-100分制)

### 2.1 抓取准确性 (25分)

| 分值 | 标准 | 说明 |
|------|------|------|
| 23-25 | 优秀 | 所有必需字段完整抓取，无数据丢失，字段映射100%准确 |
| 18-22 | 良好 | 必需字段完整，1-2个可选字段缺失，映射准确率>95% |
| 13-17 | 合格 | 关键字段完整，3-5个字段存在小问题，映射准确率>85% |
| 8-12 | 较差 | 关键字段缺失或错误，映射准确率<85% |
| 0-7 | 不合格 | 严重数据丢失，无法使用 |

**检查点**
- source.platform, source.subreddit, source.post_id 必须准确
- engagement.upvotes, engagement.comments 数值正确
- content.title, content.body 内容完整无截断

### 2.2 内容筛选质量 (30分)

| 分值 | 标准 | 说明 |
|------|------|------|
| 27-30 | 优秀 | 戏剧性评分与人工评估一致率>90%，标签命中率>85% |
| 21-26 | 良好 | 一致率>80%，标签命中率>75%，少量误判 |
| 15-20 | 合格 | 一致率>70%，基本筛选逻辑正确 |
| 9-14 | 较差 | 一致率<70%，存在明显误判 |
| 0-8 | 不合格 | 筛选逻辑混乱，大量误判 |

**检查点**
- dramatic_score 计算符合公式
- drama_tags 标签合理（至少命中1个）
- 高戏剧性内容（>7.0分）被正确识别

### 2.3 提取完整性 (25分)

| 分值 | 标准 | 说明 |
|------|------|------|
| 23-25 | 优秀 | drama_elements 所有子字段完整，conflict结构完整，emotional_arc清晰 |
| 18-22 | 良好 | 主要字段完整，1-2个次要字段缺失，描述基本准确 |
| 13-17 | 合格 | 关键字段存在，但描述较简略或部分不准确 |
| 8-12 | 较差 | 多个关键字段缺失，描述质量差 |
| 0-7 | 不合格 | 提取逻辑混乱，信息严重缺失 |

**检查点**
- setting: location, mood 正确识别
- characters: 至少识别 protagonist
- conflict: trigger, escalation, climax 完整
- dialogue_highlights: 至少提取1条金句
- emotional_arc: 情绪变化描述清晰

### 2.4 JSON格式规范性 (20分)

| 分值 | 标准 | 说明 |
|------|------|------|
| 18-20 | 优秀 | 完全符合 schema，字段类型正确，枚举值合规，无多余字段 |
| 14-17 | 良好 | 基本符合 schema，1-2个小问题（如字段顺序）|
| 10-13 | 合格 | 主要结构正确，存在3-5个格式问题 |
| 6-9 | 较差 | 结构混乱，多个字段类型错误 |
| 0-5 | 不合格 | 无法通过 JSON 验证 |

**检查点**
- 字段类型符合定义（string/number/array）
- 枚举值在允许范围内
- 日期格式 ISO8601
- 无未定义字段

---

## 3. 基准分数线

| 等级 | 分数 | 说明 |
|------|------|------|
| **Elite** | ≥90 | 顶尖水平，可直接用于生产环境 |
| **Industry Avg** | 75 | 行业平均水平，需要一定人工复核 |
| **Acceptable** | 60 | 最低可接受水平，需大量人工干预 |
| **Failed** | <60 | 不合格，无法使用 |

**推荐使用标准**
- 生产部署: ≥85分
- 测试通过: ≥75分
- 需要优化: 60-75分
- 重新开发: <60分

---

## 4. 测试用例

### 4.1 测试用例 #1 - 家庭冲突（高戏剧性）

**输入**
```json
{
  "title": "AITA for telling my mother-in-law she can't see our baby until she apologizes?",
  "body": "My MIL has always been overbearing. Last week she showed up unannounced while I was recovering from C-section. She started criticizing how I was holding the baby, saying I was 'doing it wrong'. My husband was at work. I told her to leave. She called me a 'terrible mother' and left in a huff. Now she's telling family I banned her from seeing her grandchild. My husband thinks I should apologize to keep the peace.",
  "upvotes": 15234,
  "comments": 2341,
  "upvote_ratio": 0.87
}
```

**预期输出标准**
- dramatic_score: ≥8.0（高争议性，强情绪）
- drama_tags: ["family_conflict", "high_emotion"]
- conflict.type: "family_expectation_vs_boundaries"
- hook_strength: ≥8

### 4.2 测试用例 #2 - 职场冲突（中戏剧性）

**输入**
```json
{
  "title": "My boss took credit for my project in front of the CEO",
  "body": "I've been working on this marketing campaign for 3 months. Yesterday in the quarterly meeting, when the CEO asked about the results, my boss presented it as his own work. I was sitting right there. He didn't mention my name once. After the meeting, he pulled me aside and said 'that's just how things work around here.' I feel devastated but I need this job.",
  "upvotes": 8932,
  "comments": 876,
  "upvote_ratio": 0.92
}
```

**预期输出标准**
- dramatic_score: 7.0-8.5
- drama_tags: ["workplace_drama"]
- conflict.type: "power_abuse_credit_stealing"
- emotional_arc: 包含 "devastated", "conflicted"

### 4.3 测试用例 #3 - 情感纠纷（高戏剧性）

**输入**
```json
{
  "title": "I found out my fiancé cheated on his bachelor party",
  "body": "We're supposed to get married in two weeks. His best man got drunk and told me everything last night. My fiancé says it was 'just a mistake' and 'didn't mean anything.' He begged me not to cancel the wedding. Our families have spent thousands on deposits. I don't know if I can ever trust him again, but the shame of calling off the wedding... I don't know what to do.",
  "upvotes": 28456,
  "comments": 4523,
  "upvote_ratio": 0.79
}
```

**预期输出标准**
- dramatic_score: ≥8.5
- drama_tags: ["relationship_crisis", "moral_dilemma", "high_emotion"]
- viral_elements: 包含 "relatable", "controversial"
- genre_fit: ["relationship drama"]

### 4.4 测试用例 #4 - 道德困境（中戏剧性）

**输入**
```json
{
  "title": "WIBTA if I reported my coworker who's stealing from the company?",
  "body": "I noticed Sarah has been taking office supplies home - expensive stuff like monitors and printers. She's a single mom and I know she's struggling. But last week I saw her taking cash from the petty cash box. If I report her, she'll probably get fired. If I don't, and audit finds out I knew, I could lose my job too. We've been friends for 3 years.",
  "upvotes": 6543,
  "comments": 1234,
  "upvote_ratio": 0.81
}
```

**预期输出标准**
- dramatic_score: 6.5-7.5
- drama_tags: ["moral_dilemma", "workplace_drama"]
- conflict.type: "loyalty_vs_integrity"
- characters: 至少识别 protagonist 和 supporting (Sarah)

### 4.5 测试用例 #5 - 低戏剧性内容（负样本）

**输入**
```json
{
  "title": "My favorite coffee shop closed down today",
  "body": "Just wanted to share that the little cafe where I've been getting my morning coffee for 5 years closed today. The owner is retiring. Going to miss this place and the staff who knew my order by heart.",
  "upvotes": 234,
  "comments": 45,
  "upvote_ratio": 0.96
}
```

**预期输出标准**
- dramatic_score: <5.0
- drama_tags: [] (空或极少)
- hook_strength: <5
- script_potential.estimated_duration: 不宜超过 "1-2 min"

---

## 5. 评估流程

### 5.1 自动化测试
```bash
# 运行基准测试
npm run benchmark:reddit-scraper

# 输出格式
{
  "overall_score": 78,
  "dimensions": {
    "crawl_accuracy": 22,
    "filter_quality": 24,
    "extraction_completeness": 20,
    "json_format": 18
  },
  "test_cases": [
    { "id": 1, "score": 85, "passed": true },
    { "id": 2, "score": 76, "passed": true },
    ...
  ]
}
```

### 5.2 人工复核
- 对 dramatic_score 偏差 >1.5 的样本进行人工标注
- 抽检 10% 的 drama_tags 标注准确性
- 检查 conflict 结构完整性

---

## 6. 版本历史

| 版本 | 日期 | 变更 |
|------|------|------|
| 1.0.0 | 2026-03-25 | 初始版本，基于 Hunter 调研数据构建 |

---

*基准文档结束*