benchmark.md 2.64 KB

热点研究员 Benchmark

测试数据集

Test Set 1: 基础抓取 (10条)

[
  {"title": "Tech companies face new regulations", "source": "HN", "category": "tech"},
  {"title": "World Cup qualifier results", "source": "BBC", "category": "sports"},
  {"title": "Oil prices surge amid tensions", "source": "Al Jazeera", "category": "business"},
  {"title": "Climate summit reaches agreement", "source": "BBC", "category": "world"},
  {"title": "Celebrity news trending", "source": "Various", "category": "entertainment"},
  {"title": "Military developments in conflict zone", "source": "Al Jazeera", "category": "military"},
  {"title": "Stock market update", "source": "BBC", "category": "business"},
  {"title": "Entertainment award ceremony", "source": "Various", "category": "entertainment"},
  {"title": "Technology breakthrough announced", "source": "HN", "category": "tech"},
  {"title": "International trade negotiations", "source": "Al Jazeera", "category": "world"}
]

Test Set 2: 深度研究 (5条)

[
  {
    "topic": "Major international conflict",
    "required_fields": ["background", "key_figures", "story_arc"]
  },
  {
    "topic": "Celebrity scandal",
    "required_fields": ["background", "key_figures", "dialogue_samples"]
  },
  {
    "topic": "Sports championship",
    "required_fields": ["background", "key_figures", "story_arc"]
  },
  {
    "topic": "Business merger",
    "required_fields": ["background", "key_figures", "adaptation_notes"]
  },
  {
    "topic": "Tech innovation",
    "required_fields": ["background", "key_figures", "story_arc"]
  }
]

评分维度

1. 抓取完整性 (40分)

指标 得分
标题完整 10
来源标注 5
分类准确 10
时间戳 5
热度评分 10

2. 数据质量 (30分)

指标 得分
无重复 10
无截断 10
编码正确 5
格式规范 5

3. 研究深度 (20分)

指标 得分
背景调查 8
人物分析 6
剧情建议 6

4. 效率 (10分)

指标 得分
处理速度 5
资源利用 5

合格标准

  • 通过: ≥ 85分
  • 优秀: ≥ 95分

测试命令

# 抓取测试
python -c "
from skills.hot_topic_researcher import HotTopicResearcher
researcher = HotTopicResearcher()
results = researcher.fetch_all()
score = evaluate(results)
print(f'Score: {score}')
"

# 研究测试
python -c "
from skills.hot_topic_researcher import DeepResearcher
researcher = DeepResearcher()
report = researcher.investigate('topic_name')
score = evaluate_report(report)
print(f'Score: {score}')
"