benchmark.md
15.2 KB
Reddit Scraper V2 - 评估基准文档
生成日期: 2026-03-25 数据来源: 280 条合成 Reddit 帖子 基准版本: v1.0
1. 评估维度定义
评分维度总览
| 维度 | 权重 | 评分范围 | 核心考察点 |
|---|---|---|---|
| 抓取准确性 | 25% | 0-100 | 字段完整度、数据正确性 |
| 内容筛选质量 | 30% | 0-100 | 戏剧性识别准确率 |
| 提取完整性 | 25% | 0-100 | drama_elements 字段覆盖度 |
| JSON格式规范性 | 20% | 0-100 | schema 合规度 |
总分计算公式
总分 = 抓取准确性 × 0.25 + 内容筛选质量 × 0.30 + 提取完整性 × 0.25 + JSON格式规范性 × 0.20
2. 各维度评分标准
2.1 抓取准确性 (25%)
定义: 抓取的数据字段是否完整、准确
| 等级 | 分数 | 标准 |
|---|---|---|
| 优秀 | 90-100 | 所有必需字段 100% 完整，无缺失/错误 |
| 良好 | 75-89 | 必需字段 ≥95% 完整，偶有小错误 |
| 及格 | 60-74 | 必需字段 ≥85% 完整，部分可修复错误 |
| 不及格 | 必需字段 |
评分细则:
// 字段权重表
const FIELD_WEIGHTS = {
'id': 10, // 唯一标识，必须
'subreddit': 10, // 数据源标识
'title': 15, // 核心内容
'author': 5, // 作者信息
'selftext': 15, // 正文内容
'score': 10, // 互动数据
'num_comments': 10, // 互动数据
'upvote_ratio': 10, // 互动数据
'created_utc': 5, // 时间戳
'permalink': 5, // 链接
'drama_elements': 5 // 扩展字段
};
// 计算方法
accuracy_score = Σ(存在字段 × 权重) / Σ(总权重) × 100
扣分项:
- 字段类型错误: -5分/处
- 数据格式错误: -3分/处
- 字段缺失: -10分/处（必需字段）
2.2 内容筛选质量 (30%)
定义: 识别戏剧性内容的准确程度
| 等级 | 分数 | 标准 |
|---|---|---|
| 优秀 | 90-100 | 戏剧性识别准确率 ≥95%，误判率 |
| 良好 | 75-89 | 戏剧性识别准确率 85-95%，误判率 3-8% |
| 及格 | 60-74 | 戏剧性识别准确率 70-85%，误判率 8-15% |
| 不及格 | 戏剧性识别准确率 15% |
评分指标:
准确性 = (正确识别的戏剧性帖子 + 正确识别的非戏剧性帖子) / 总帖子数
精确率 = 正确识别的戏剧性帖子 / 所有识别为戏剧性的帖子
召回率 = 正确识别的戏剧性帖子 / 所有真实戏剧性帖子
F1 Score = 2 × (精确率 × 召回率) / (精确率 + 召回率)
戏剧性判定标准:
| 指标 | 阈值 | 说明 |
|---|---|---|
| dramatic_score | ≥6.0 | 戏剧性候选判定线 |
| dramatic_level | high/medium | 高/中戏剧性 |
| is_drama_candidate | true | 布尔标记 |
2.3 提取完整性 (25%)
定义: drama_elements 结构化信息的完整程度
| 等级 | 分数 | 标准 |
|---|---|---|
| 优秀 | 90-100 | drama_elements 100% 覆盖，所有子字段完整 |
| 良好 | 75-89 | drama_elements ≥95% 覆盖，关键子字段完整 |
| 及格 | 60-74 | drama_elements ≥85% 覆盖，部分子字段缺失 |
| 不及格 | drama_elements |
drama_elements 字段要求:
{
"protagonist": "required", // 主角身份，必须
"antagonist": "required", // 对抗者身份，必须
"conflict_type": "required", // 冲突类型，必须
"key_quotes": "recommended", // 关键台词，推荐（≥1条）
"dramatic_moments": "recommended" // 戏剧时刻，推荐（≥1个）
}
评分权重:
| 子字段 | 权重 | 评分标准 |
|---|---|---|
| protagonist | 20% | 存在且合理 |
| antagonist | 20% | 存在且合理 |
| conflict_type | 30% | 存在且在有效枚举内 |
| key_quotes | 20% | ≥1条为满分，0条为0分 |
| dramatic_moments | 10% | ≥1个为满分，0个为0分 |
2.4 JSON格式规范性 (20%)
定义: 输出 JSON 是否符合定义的 schema
| 等级 | 分数 | 标准 |
|---|---|---|
| 优秀 | 90-100 | 100% 符合 schema，无额外字段 |
| 良好 | 75-89 | ≥95% 符合 schema，偶有不影响使用的偏差 |
| 及格 | 60-74 | ≥85% 符合 schema，部分字段类型/格式问题 |
| 不及格 |
Schema 合规检查项:
| 检查项 | 权重 | 说明 |
|---|---|---|
| 必需字段存在 | 30% | 所有 required 字段必须存在 |
| 字段类型正确 | 25% | 字段值类型符合定义 |
| 枚举值合规 | 25% | 枚举字段值在定义范围内 |
| 格式合规 | 15% | 日期、URL 等格式正确 |
| 无冗余字段 | 5% | 无未定义的额外字段 |
3. 行业基准线
3.1 分数线定义
| 等级 | 分数区间 | 说明 |
|---|---|---|
| 精英级 | ≥90 | 生产就绪，可自动化部署 |
| 优秀 | 80-89 | 质量良好，少量人工审核 |
| 行业平均 | 70-79 | 可接受水平，需要人工检查 |
| 及格线 | 60-69 | 勉强可用，大量人工干预 |
| 不及格 | 不可接受，需重新开发 |
3.2 基准线数值
industry_avg_score: 75 # 行业平均分
elite_avg_score: 90 # 精英级平均分
passing_score: 60 # 及格线
recommended_threshold: 80 # 推荐使用阈值
3.3 版本演进目标
| 版本 | 目标分数 | 关键改进点 |
|---|---|---|
| v1.0 (当前) | 75 | 建立基准，完成 280 条数据验证 |
| v1.1 | 80 | 优化戏剧性识别算法，提高准确率 |
| v1.2 | 85 | 完善 drama_elements 提取，减少缺失 |
| v2.0 | 90 | 全面自动化，无需人工审核 |
4. 测试用例集
4.1 测试用例概览
共 10 个测试用例，覆盖：
- 高戏剧性 (≥7.0): 4 个
- 中戏剧性 (5.0-7.0): 3 个
- 低戏剧性 (
4.2 测试用例详情
用例 #1 - 高戏剧性家庭冲突
{
"id": "t3_test01",
"subreddit": "AmITheAsshole",
"title": "AITA for refusing to give my inheritance to my sister?",
"score": 18500,
"num_comments": 3200,
"upvote_ratio": 0.72,
"drama_elements": {
"protagonist": "sister",
"antagonist": "sister",
"conflict_type": "inheritance_dispute",
"key_quotes": [{"text": "You don't deserve it!", "speaker": "sister"}],
"dramatic_moments": [{"type": "confrontation", "intensity": 9}]
},
"expected_dramatic_score": 8.5,
"expected_dramatic_level": "high",
"expected_is_drama_candidate": true
}
用例 #2 - 高戏剧性感情危机
{
"id": "t3_test02",
"subreddit": "relationship_advice",
"title": "My boyfriend cheated and wants me back, what should I do?",
"score": 12000,
"num_comments": 2800,
"upvote_ratio": 0.78,
"drama_elements": {
"protagonist": "girlfriend",
"antagonist": "boyfriend",
"conflict_type": "relationship_crisis",
"key_quotes": [{"text": "I can change, I promise!", "speaker": "boyfriend"}],
"dramatic_moments": [{"type": "reveal", "intensity": 10}]
},
"expected_dramatic_score": 8.8,
"expected_dramatic_level": "high",
"expected_is_drama_candidate": true
}
用例 #3 - 高戏剧性职场斗争
{
"id": "t3_test03",
"subreddit": "AmITheAsshole",
"title": "AITA for reporting my boss to HR?",
"score": 15000,
"num_comments": 2500,
"upvote_ratio": 0.75,
"drama_elements": {
"protagonist": "employee",
"antagonist": "boss",
"conflict_type": "workplace_drama",
"key_quotes": [{"text": "You'll regret this!", "speaker": "boss"}],
"dramatic_moments": [{"type": "climax", "intensity": 8}]
},
"expected_dramatic_score": 8.2,
"expected_dramatic_level": "high",
"expected_is_drama_candidate": true
}
用例 #4 - 高戏剧性婚礼变故
{
"id": "t3_test04",
"subreddit": "relationship_advice",
"title": "My fiancé called off the wedding yesterday",
"score": 9800,
"num_comments": 1900,
"upvote_ratio": 0.80,
"drama_elements": {
"protagonist": "fiancée",
"antagonist": "fiancé",
"conflict_type": "wedding_drama",
"key_quotes": [{"text": "I can't do this anymore.", "speaker": "fiancé"}],
"dramatic_moments": [{"type": "twist", "intensity": 9}]
},
"expected_dramatic_score": 7.5,
"expected_dramatic_level": "high",
"expected_is_drama_candidate": true
}
用例 #5 - 中戏剧性沟通障碍
{
"id": "t3_test05",
"subreddit": "relationship_advice",
"title": "My partner and I keep misunderstanding each other",
"score": 3500,
"num_comments": 800,
"upvote_ratio": 0.85,
"drama_elements": {
"protagonist": "partner",
"antagonist": "partner",
"conflict_type": "communication_breakdown",
"key_quotes": [{"text": "That's not what I meant!", "speaker": "protagonist"}],
"dramatic_moments": [{"type": "confrontation", "intensity": 5}]
},
"expected_dramatic_score": 5.8,
"expected_dramatic_level": "medium",
"expected_is_drama_candidate": true
}
用例 #6 - 中戏剧性信任问题
{
"id": "t3_test06",
"subreddit": "AmITheAsshole",
"title": "AITA for checking my girlfriend's phone?",
"score": 5200,
"num_comments": 1200,
"upvote_ratio": 0.68,
"drama_elements": {
"protagonist": "boyfriend",
"antagonist": "girlfriend",
"conflict_type": "trust_issues",
"key_quotes": [{"text": "You don't trust me at all!", "speaker": "girlfriend"}],
"dramatic_moments": [{"type": "reveal", "intensity": 6}]
},
"expected_dramatic_score": 6.2,
"expected_dramatic_level": "medium",
"expected_is_drama_candidate": true
}
用例 #7 - 中戏剧性社交尴尬
{
"id": "t3_test07",
"subreddit": "funny",
"title": "I accidentally called my teacher 'mom' today",
"score": 8500,
"num_comments": 400,
"upvote_ratio": 0.92,
"drama_elements": {
"protagonist": "student",
"antagonist": "teacher",
"conflict_type": "social_awkwardness",
"key_quotes": [],
"dramatic_moments": [{"type": "confrontation", "intensity": 4}]
},
"expected_dramatic_score": 4.2,
"expected_dramatic_level": "low",
"expected_is_drama_candidate": false
}
用例 #8 - 低戏剧性荒诞情境
{
"id": "t3_test08",
"subreddit": "funny",
"title": "My cat tried to cook dinner",
"score": 12000,
"num_comments": 200,
"upvote_ratio": 0.95,
"drama_elements": {
"protagonist": "cat_owner",
"antagonist": "cat",
"conflict_type": "absurd_situation",
"key_quotes": [],
"dramatic_moments": []
},
"expected_dramatic_score": 3.5,
"expected_dramatic_level": "low",
"expected_is_drama_candidate": false
}
用例 #9 - 低戏剧性笑话
{
"id": "t3_test09",
"subreddit": "jokes",
"title": "Why did the chicken cross the road?",
"score": 8000,
"num_comments": 150,
"upvote_ratio": 0.88,
"drama_elements": {
"protagonist": "narrator",
"antagonist": "chicken",
"conflict_type": "wordplay",
"key_quotes": [],
"dramatic_moments": []
},
"expected_dramatic_score": 2.1,
"expected_dramatic_level": "low",
"expected_is_drama_candidate": false
}
用例 #10 - 低戏剧性观察喜剧
{
"id": "t3_test10",
"subreddit": "StandUpComedy",
"title": "On dating apps: \"It's like a job interview but worse\"",
"score": 3500,
"num_comments": 120,
"upvote_ratio": 0.90,
"drama_elements": {
"protagonist": "comedian",
"antagonist": "dating_app",
"conflict_type": "observational_humor",
"key_quotes": [{"text": "It's like a job interview but worse", "speaker": "comedian"}],
"dramatic_moments": []
},
"expected_dramatic_score": 3.8,
"expected_dramatic_level": "low",
"expected_is_drama_candidate": false
}
4.3 测试用例验证矩阵
| 用例 | subreddit | dramatic_level | conflict_type | 关键验证点 |
|---|---|---|---|---|
| #1 | AITA | high | inheritance_dispute | 家庭冲突，高争议 |
| #2 | relationship_advice | high | relationship_crisis | 感情背叛，强情感 |
| #3 | AITA | high | workplace_drama | 职场对抗，权力冲突 |
| #4 | relationship_advice | high | wedding_drama | 重大人生事件 |
| #5 | relationship_advice | medium | communication_breakdown | 渐进式冲突 |
| #6 | AITA | medium | trust_issues | 道德两难 |
| #7 | funny | low | social_awkwardness | 喜剧元素 |
| #8 | funny | low | absurd_situation | 荒诞幽默 |
| #9 | jokes | low | wordplay | 纯文字内容 |
| #10 | StandUpComedy | low | observational_humor | 观察喜剧 |
5. 自动化测试脚本
// benchmark_validator.js
const fs = require('fs');
const TEST_CASES = [
{ id: 't3_test01', expected_level: 'high', expected_score_min: 7.0 },
{ id: 't3_test02', expected_level: 'high', expected_score_min: 7.0 },
{ id: 't3_test03', expected_level: 'high', expected_score_min: 7.0 },
{ id: 't3_test04', expected_level: 'high', expected_score_min: 7.0 },
{ id: 't3_test05', expected_level: 'medium', expected_score_min: 5.0, expected_score_max: 7.0 },
{ id: 't3_test06', expected_level: 'medium', expected_score_min: 5.0, expected_score_max: 7.0 },
{ id: 't3_test07', expected_level: 'low', expected_score_max: 5.0 },
{ id: 't3_test08', expected_level: 'low', expected_score_max: 5.0 },
{ id: 't3_test09', expected_level: 'low', expected_score_max: 5.0 },
{ id: 't3_test10', expected_level: 'low', expected_score_max: 5.0 }
];
function validateBenchmark(posts) {
let passed = 0;
const results = [];
TEST_CASES.forEach(test => {
const post = posts.find(p => p.id === test.id);
if (!post) {
results.push({ id: test.id, status: 'NOT_FOUND' });
return;
}
const levelMatch = post.dramatic_level === test.expected_level;
const scoreInRange = (!test.expected_score_min || post.dramatic_score >= test.expected_score_min) &&
(!test.expected_score_max || post.dramatic_score <= test.expected_score_max);
const success = levelMatch && scoreInRange;
if (success) passed++;
results.push({
id: test.id,
status: success ? 'PASS' : 'FAIL',
expected: test.expected_level,
actual: post.dramatic_level,
score: post.dramatic_score
});
});
return {
total: TEST_CASES.length,
passed,
failed: TEST_CASES.length - passed,
accuracy: (passed / TEST_CASES.length * 100).toFixed(1) + '%',
results
};
}
// 运行验证
const posts = JSON.parse(fs.readFileSync('synthetic_reddit_data.json', 'utf8'));
const report = validateBenchmark(posts);
console.log('Benchmark Validation Report:');
console.log(JSON.stringify(report, null, 2));
6. 质量检查清单
预发布检查
- [ ] 所有 10 个测试用例通过
- [ ] 总体准确率 ≥75%
- [ ] 字段完整度 100%
- [ ] JSON Schema 100% 合规
- [ ] 戏剧性评分可复现（相同输入产生相同输出）
- [ ] 文档与代码实现一致
持续监控指标
| 指标 | 目标值 | 告警阈值 |
|---|---|---|
| 抓取成功率 | >98% | |
| 字段完整度 | 100% | |
| 戏剧性识别准确率 | >85% | |
| 平均处理时间 | >5s/条 |
文档版本: v1.0 最后更新: 2026-03-25 下次审查: 2026-04-25