benchmark_v1.md
3.35 KB
trend-scout-cn Benchmark v1
Objective
Evaluate the quality of Chinese social-media trending-data extraction outputs produced by the trend-scout-cn skill across 6 scoring dimensions.
Scoring Rubric (Weights sum to 100%; Max score = 100)
| # | Dimension | Weight | Description | 5 (Excellent) | 3 (Acceptable) | 1 (Poor) |
|---|---|---|---|---|---|---|
| 1 | Data Accuracy | 20% | Are rank, hot_value, topic text, author, likes, and timestamps faithful to the source? | All numeric fields parsed correctly; topic text matches source exactly; no hallucinated authors or likes. | Minor truncation or 1-2 off-by-one rank errors; 90%+ fields correct. | Major hallucinations, swapped ranks, or fabricated hot_values/authors. |
| 2 | Anti-Crawl Strategy | 15% | Does the output reflect robust request handling (UA rotation, referer, cookie, fallback, rate-limit)? | All 6+ sources include correct headers/fallbacks and respect robots.txt/delay >=2s. |
Basic headers present but 1-2 sources miss fallback or delay notes. | No headers, no fallback, ignores rate limits or bypasses compliance. |
| 3 | Schema Normalization | 20% | Are keys, types, and enums (platform, trend_direction, duration) consistent across sources? | Uniform schema across all platforms; nulls handled gracefully; enums strictly normalized (up/down/stable). | Minor inconsistencies (e.g., mixed snake/camel case) or missing null defaults. | Wildly inconsistent schemas, invalid enums, or malformed JSON structure. |
| 4 | Platform Coverage | 20% | Does the extraction cover the configured platforms proportionally without omissions? | All scheduled sources present and proportionally represented (weibo, bilibili, douyin, xiaohongshu, jjwxc, plus zhihu/baidu/toutiao/douban where seeded). | 1 minor source missing or heavily under-sampled. | Multiple configured sources completely missing. |
| 5 | Insight Extraction | 15% | Are related_topics, category, and trend_direction inferred or extracted meaningfully? | related_topics deduplicated and aligned; category matched from source taxonomy; direction inferred from change. | Some empty related_topics or generic categories, but mostly correct. | No related_topics, no categories, random trend_direction values. |
| 6 | Compliance | 10% | Does the output respect data retention, exclude private data, and stay within ToS? | Retention flag respect_platform_tos explicitly true. | Minor ToS note missing; retention implied but not stated. | Private data exposed, retention ignored, or explicit ToS violation. |
Calculation
Each dimension is scored 1-5, then weighted:
Total = (D1 * 4 + D2 * 3 + D3 * 4 + D4 * 4 + D5 * 3 + D6 * 2)
Equivalently:
Total = D1*0.20 + D2*0.15 + D3*0.20 + D4*0.20 + D5*0.15 + D6*0.10
Max = 5.0 per dimension -> 100 total.
Target Score
>= 95 / 100
Notes for Evaluators
- Use the
seed_data.jsonground-truth for Data Accuracy checks. - For Anti-Crawl Strategy, cross-check headers and fallback entries against
crawl_config.json. - Schema Normalization should match the JSON structure defined in
crawl_config.json>sources[].parser.fields. - Any output containing fabricated
author,rank, orhot_valueshould be penalized heavily under Data Accuracy. - Compliance failures are zero-tolerance items; a single PII leak caps this dimension at 1 point.