# Reddit Scraper V2 - 评估基准文档

> **生成日期**: 2026-03-25
> **数据来源**: 280 条合成 Reddit 帖子
> **基准版本**: v1.0

---

## 1. 评估维度定义

### 评分维度总览

| 维度 | 权重 | 评分范围 | 核心考察点 |
|------|------|----------|------------|
| **抓取准确性** | 25% | 0-100 | 字段完整度、数据正确性 |
| **内容筛选质量** | 30% | 0-100 | 戏剧性识别准确率 |
| **提取完整性** | 25% | 0-100 | drama_elements 字段覆盖度 |
| **JSON格式规范性** | 20% | 0-100 | schema 合规度 |

### 总分计算公式

```
总分 = 抓取准确性 × 0.25 + 内容筛选质量 × 0.30 + 提取完整性 × 0.25 + JSON格式规范性 × 0.20
```

---

## 2. 各维度评分标准

### 2.1 抓取准确性 (25%)

**定义**: 抓取的数据字段是否完整、准确

| 等级 | 分数 | 标准 |
|------|------|------|
| **优秀** | 90-100 | 所有必需字段 100% 完整，无缺失/错误 |
| **良好** | 75-89 | 必需字段 ≥95% 完整，偶有小错误 |
| **及格** | 60-74 | 必需字段 ≥85% 完整，部分可修复错误 |
| **不及格** | <60 | 必需字段 <85% 完整，大量错误 |

**评分细则**:

```javascript
// 字段权重表
const FIELD_WEIGHTS = {
  'id': 10,           // 唯一标识，必须
  'subreddit': 10,    // 数据源标识
  'title': 15,        // 核心内容
  'author': 5,        // 作者信息
  'selftext': 15,     // 正文内容
  'score': 10,        // 互动数据
  'num_comments': 10, // 互动数据
  'upvote_ratio': 10, // 互动数据
  'created_utc': 5,   // 时间戳
  'permalink': 5,     // 链接
  'drama_elements': 5 // 扩展字段
};

// 计算方法
accuracy_score = Σ(存在字段 × 权重) / Σ(总权重) × 100
```

**扣分项**:
- 字段类型错误: -5分/处
- 数据格式错误: -3分/处
- 字段缺失: -10分/处（必需字段）

---

### 2.2 内容筛选质量 (30%)

**定义**: 识别戏剧性内容的准确程度

| 等级 | 分数 | 标准 |
|------|------|------|
| **优秀** | 90-100 | 戏剧性识别准确率 ≥95%，误判率 <3% |
| **良好** | 75-89 | 戏剧性识别准确率 85-95%，误判率 3-8% |
| **及格** | 60-74 | 戏剧性识别准确率 70-85%，误判率 8-15% |
| **不及格** | <60 | 戏剧性识别准确率 <70%，误判率 >15% |

**评分指标**:

```
准确性 = (正确识别的戏剧性帖子 + 正确识别的非戏剧性帖子) / 总帖子数
精确率 = 正确识别的戏剧性帖子 / 所有识别为戏剧性的帖子
召回率 = 正确识别的戏剧性帖子 / 所有真实戏剧性帖子
F1 Score = 2 × (精确率 × 召回率) / (精确率 + 召回率)
```

**戏剧性判定标准**:

| 指标 | 阈值 | 说明 |
|------|------|------|
| dramatic_score | ≥6.0 | 戏剧性候选判定线 |
| dramatic_level | high/medium | 高/中戏剧性 |
| is_drama_candidate | true | 布尔标记 |

---

### 2.3 提取完整性 (25%)

**定义**: drama_elements 结构化信息的完整程度

| 等级 | 分数 | 标准 |
|------|------|------|
| **优秀** | 90-100 | drama_elements 100% 覆盖，所有子字段完整 |
| **良好** | 75-89 | drama_elements ≥95% 覆盖，关键子字段完整 |
| **及格** | 60-74 | drama_elements ≥85% 覆盖，部分子字段缺失 |
| **不及格** | <60 | drama_elements <85% 覆盖，关键信息缺失 |

**drama_elements 字段要求**:

```json
{
  "protagonist": "required",        // 主角身份，必须
  "antagonist": "required",         // 对抗者身份，必须
  "conflict_type": "required",      // 冲突类型，必须
  "key_quotes": "recommended",      // 关键台词，推荐（≥1条）
  "dramatic_moments": "recommended" // 戏剧时刻，推荐（≥1个）
}
```

**评分权重**:

| 子字段 | 权重 | 评分标准 |
|--------|------|----------|
| protagonist | 20% | 存在且合理 |
| antagonist | 20% | 存在且合理 |
| conflict_type | 30% | 存在且在有效枚举内 |
| key_quotes | 20% | ≥1条为满分，0条为0分 |
| dramatic_moments | 10% | ≥1个为满分，0个为0分 |

---

### 2.4 JSON格式规范性 (20%)

**定义**: 输出 JSON 是否符合定义的 schema

| 等级 | 分数 | 标准 |
|------|------|------|
| **优秀** | 90-100 | 100% 符合 schema，无额外字段 |
| **良好** | 75-89 | ≥95% 符合 schema，偶有不影响使用的偏差 |
| **及格** | 60-74 | ≥85% 符合 schema，部分字段类型/格式问题 |
| **不及格** | <60 | <85% 符合 schema，存在结构性问题 |

**Schema 合规检查项**:

| 检查项 | 权重 | 说明 |
|--------|------|------|
| 必需字段存在 | 30% | 所有 required 字段必须存在 |
| 字段类型正确 | 25% | 字段值类型符合定义 |
| 枚举值合规 | 25% | 枚举字段值在定义范围内 |
| 格式合规 | 15% | 日期、URL 等格式正确 |
| 无冗余字段 | 5% | 无未定义的额外字段 |

---

## 3. 行业基准线

### 3.1 分数线定义

| 等级 | 分数区间 | 说明 |
|------|----------|------|
| **精英级** | ≥90 | 生产就绪，可自动化部署 |
| **优秀** | 80-89 | 质量良好，少量人工审核 |
| **行业平均** | 70-79 | 可接受水平，需要人工检查 |
| **及格线** | 60-69 | 勉强可用，大量人工干预 |
| **不及格** | <60 | 不可接受，需重新开发 |

### 3.2 基准线数值

```yaml
industry_avg_score: 75    # 行业平均分
elite_avg_score: 90       # 精英级平均分
passing_score: 60         # 及格线
recommended_threshold: 80 # 推荐使用阈值
```

### 3.3 版本演进目标

| 版本 | 目标分数 | 关键改进点 |
|------|----------|------------|
| v1.0 (当前) | 75 | 建立基准，完成 280 条数据验证 |
| v1.1 | 80 | 优化戏剧性识别算法，提高准确率 |
| v1.2 | 85 | 完善 drama_elements 提取，减少缺失 |
| v2.0 | 90 | 全面自动化，无需人工审核 |

---

## 4. 测试用例集

### 4.1 测试用例概览

共 **10 个测试用例**，覆盖：
- 高戏剧性 (≥7.0): 4 个
- 中戏剧性 (5.0-7.0): 3 个
- 低戏剧性 (<5.0): 3 个

### 4.2 测试用例详情

#### 用例 #1 - 高戏剧性家庭冲突
```json
{
  "id": "t3_test01",
  "subreddit": "AmITheAsshole",
  "title": "AITA for refusing to give my inheritance to my sister?",
  "score": 18500,
  "num_comments": 3200,
  "upvote_ratio": 0.72,
  "drama_elements": {
    "protagonist": "sister",
    "antagonist": "sister",
    "conflict_type": "inheritance_dispute",
    "key_quotes": [{"text": "You don't deserve it!", "speaker": "sister"}],
    "dramatic_moments": [{"type": "confrontation", "intensity": 9}]
  },
  "expected_dramatic_score": 8.5,
  "expected_dramatic_level": "high",
  "expected_is_drama_candidate": true
}
```

#### 用例 #2 - 高戏剧性感情危机
```json
{
  "id": "t3_test02",
  "subreddit": "relationship_advice",
  "title": "My boyfriend cheated and wants me back, what should I do?",
  "score": 12000,
  "num_comments": 2800,
  "upvote_ratio": 0.78,
  "drama_elements": {
    "protagonist": "girlfriend",
    "antagonist": "boyfriend",
    "conflict_type": "relationship_crisis",
    "key_quotes": [{"text": "I can change, I promise!", "speaker": "boyfriend"}],
    "dramatic_moments": [{"type": "reveal", "intensity": 10}]
  },
  "expected_dramatic_score": 8.8,
  "expected_dramatic_level": "high",
  "expected_is_drama_candidate": true
}
```

#### 用例 #3 - 高戏剧性职场斗争
```json
{
  "id": "t3_test03",
  "subreddit": "AmITheAsshole",
  "title": "AITA for reporting my boss to HR?",
  "score": 15000,
  "num_comments": 2500,
  "upvote_ratio": 0.75,
  "drama_elements": {
    "protagonist": "employee",
    "antagonist": "boss",
    "conflict_type": "workplace_drama",
    "key_quotes": [{"text": "You'll regret this!", "speaker": "boss"}],
    "dramatic_moments": [{"type": "climax", "intensity": 8}]
  },
  "expected_dramatic_score": 8.2,
  "expected_dramatic_level": "high",
  "expected_is_drama_candidate": true
}
```

#### 用例 #4 - 高戏剧性婚礼变故
```json
{
  "id": "t3_test04",
  "subreddit": "relationship_advice",
  "title": "My fiancé called off the wedding yesterday",
  "score": 9800,
  "num_comments": 1900,
  "upvote_ratio": 0.80,
  "drama_elements": {
    "protagonist": "fiancée",
    "antagonist": "fiancé",
    "conflict_type": "wedding_drama",
    "key_quotes": [{"text": "I can't do this anymore.", "speaker": "fiancé"}],
    "dramatic_moments": [{"type": "twist", "intensity": 9}]
  },
  "expected_dramatic_score": 7.5,
  "expected_dramatic_level": "high",
  "expected_is_drama_candidate": true
}
```

#### 用例 #5 - 中戏剧性沟通障碍
```json
{
  "id": "t3_test05",
  "subreddit": "relationship_advice",
  "title": "My partner and I keep misunderstanding each other",
  "score": 3500,
  "num_comments": 800,
  "upvote_ratio": 0.85,
  "drama_elements": {
    "protagonist": "partner",
    "antagonist": "partner",
    "conflict_type": "communication_breakdown",
    "key_quotes": [{"text": "That's not what I meant!", "speaker": "protagonist"}],
    "dramatic_moments": [{"type": "confrontation", "intensity": 5}]
  },
  "expected_dramatic_score": 5.8,
  "expected_dramatic_level": "medium",
  "expected_is_drama_candidate": true
}
```

#### 用例 #6 - 中戏剧性信任问题
```json
{
  "id": "t3_test06",
  "subreddit": "AmITheAsshole",
  "title": "AITA for checking my girlfriend's phone?",
  "score": 5200,
  "num_comments": 1200,
  "upvote_ratio": 0.68,
  "drama_elements": {
    "protagonist": "boyfriend",
    "antagonist": "girlfriend",
    "conflict_type": "trust_issues",
    "key_quotes": [{"text": "You don't trust me at all!", "speaker": "girlfriend"}],
    "dramatic_moments": [{"type": "reveal", "intensity": 6}]
  },
  "expected_dramatic_score": 6.2,
  "expected_dramatic_level": "medium",
  "expected_is_drama_candidate": true
}
```

#### 用例 #7 - 中戏剧性社交尴尬
```json
{
  "id": "t3_test07",
  "subreddit": "funny",
  "title": "I accidentally called my teacher 'mom' today",
  "score": 8500,
  "num_comments": 400,
  "upvote_ratio": 0.92,
  "drama_elements": {
    "protagonist": "student",
    "antagonist": "teacher",
    "conflict_type": "social_awkwardness",
    "key_quotes": [],
    "dramatic_moments": [{"type": "confrontation", "intensity": 4}]
  },
  "expected_dramatic_score": 4.2,
  "expected_dramatic_level": "low",
  "expected_is_drama_candidate": false
}
```

#### 用例 #8 - 低戏剧性荒诞情境
```json
{
  "id": "t3_test08",
  "subreddit": "funny",
  "title": "My cat tried to cook dinner",
  "score": 12000,
  "num_comments": 200,
  "upvote_ratio": 0.95,
  "drama_elements": {
    "protagonist": "cat_owner",
    "antagonist": "cat",
    "conflict_type": "absurd_situation",
    "key_quotes": [],
    "dramatic_moments": []
  },
  "expected_dramatic_score": 3.5,
  "expected_dramatic_level": "low",
  "expected_is_drama_candidate": false
}
```

#### 用例 #9 - 低戏剧性笑话
```json
{
  "id": "t3_test09",
  "subreddit": "jokes",
  "title": "Why did the chicken cross the road?",
  "score": 8000,
  "num_comments": 150,
  "upvote_ratio": 0.88,
  "drama_elements": {
    "protagonist": "narrator",
    "antagonist": "chicken",
    "conflict_type": "wordplay",
    "key_quotes": [],
    "dramatic_moments": []
  },
  "expected_dramatic_score": 2.1,
  "expected_dramatic_level": "low",
  "expected_is_drama_candidate": false
}
```

#### 用例 #10 - 低戏剧性观察喜剧
```json
{
  "id": "t3_test10",
  "subreddit": "StandUpComedy",
  "title": "On dating apps: \"It's like a job interview but worse\"",
  "score": 3500,
  "num_comments": 120,
  "upvote_ratio": 0.90,
  "drama_elements": {
    "protagonist": "comedian",
    "antagonist": "dating_app",
    "conflict_type": "observational_humor",
    "key_quotes": [{"text": "It's like a job interview but worse", "speaker": "comedian"}],
    "dramatic_moments": []
  },
  "expected_dramatic_score": 3.8,
  "expected_dramatic_level": "low",
  "expected_is_drama_candidate": false
}
```

### 4.3 测试用例验证矩阵

| 用例 | subreddit | dramatic_level | conflict_type | 关键验证点 |
|------|-----------|----------------|---------------|------------|
| #1 | AITA | high | inheritance_dispute | 家庭冲突，高争议 |
| #2 | relationship_advice | high | relationship_crisis | 感情背叛，强情感 |
| #3 | AITA | high | workplace_drama | 职场对抗，权力冲突 |
| #4 | relationship_advice | high | wedding_drama | 重大人生事件 |
| #5 | relationship_advice | medium | communication_breakdown | 渐进式冲突 |
| #6 | AITA | medium | trust_issues | 道德两难 |
| #7 | funny | low | social_awkwardness | 喜剧元素 |
| #8 | funny | low | absurd_situation | 荒诞幽默 |
| #9 | jokes | low | wordplay | 纯文字内容 |
| #10 | StandUpComedy | low | observational_humor | 观察喜剧 |

---

## 5. 自动化测试脚本

```javascript
// benchmark_validator.js
const fs = require('fs');

const TEST_CASES = [
  { id: 't3_test01', expected_level: 'high', expected_score_min: 7.0 },
  { id: 't3_test02', expected_level: 'high', expected_score_min: 7.0 },
  { id: 't3_test03', expected_level: 'high', expected_score_min: 7.0 },
  { id: 't3_test04', expected_level: 'high', expected_score_min: 7.0 },
  { id: 't3_test05', expected_level: 'medium', expected_score_min: 5.0, expected_score_max: 7.0 },
  { id: 't3_test06', expected_level: 'medium', expected_score_min: 5.0, expected_score_max: 7.0 },
  { id: 't3_test07', expected_level: 'low', expected_score_max: 5.0 },
  { id: 't3_test08', expected_level: 'low', expected_score_max: 5.0 },
  { id: 't3_test09', expected_level: 'low', expected_score_max: 5.0 },
  { id: 't3_test10', expected_level: 'low', expected_score_max: 5.0 }
];

function validateBenchmark(posts) {
  let passed = 0;
  const results = [];

  TEST_CASES.forEach(test => {
    const post = posts.find(p => p.id === test.id);
    if (!post) {
      results.push({ id: test.id, status: 'NOT_FOUND' });
      return;
    }

    const levelMatch = post.dramatic_level === test.expected_level;
    const scoreInRange = (!test.expected_score_min || post.dramatic_score >= test.expected_score_min) &&
                         (!test.expected_score_max || post.dramatic_score <= test.expected_score_max);

    const success = levelMatch && scoreInRange;
    if (success) passed++;

    results.push({
      id: test.id,
      status: success ? 'PASS' : 'FAIL',
      expected: test.expected_level,
      actual: post.dramatic_level,
      score: post.dramatic_score
    });
  });

  return {
    total: TEST_CASES.length,
    passed,
    failed: TEST_CASES.length - passed,
    accuracy: (passed / TEST_CASES.length * 100).toFixed(1) + '%',
    results
  };
}

// 运行验证
const posts = JSON.parse(fs.readFileSync('synthetic_reddit_data.json', 'utf8'));
const report = validateBenchmark(posts);
console.log('Benchmark Validation Report:');
console.log(JSON.stringify(report, null, 2));
```

---

## 6. 质量检查清单

### 预发布检查

```markdown
- [ ] 所有 10 个测试用例通过
- [ ] 总体准确率 ≥75%
- [ ] 字段完整度 100%
- [ ] JSON Schema 100% 合规
- [ ] 戏剧性评分可复现（相同输入产生相同输出）
- [ ] 文档与代码实现一致
```

### 持续监控指标

| 指标 | 目标值 | 告警阈值 |
|------|--------|----------|
| 抓取成功率 | >98% | <95% |
| 字段完整度 | 100% | <95% |
| 戏剧性识别准确率 | >85% | <75% |
| 平均处理时间 | <2s/条 | >5s/条 |

---

> **文档版本**: v1.0
> **最后更新**: 2026-03-25
> **下次审查**: 2026-04-25