benchmark_v1.md 3.35 KB

trend-scout-cn Benchmark v1

Objective

Evaluate the quality of Chinese social-media trending-data extraction outputs produced by the trend-scout-cn skill across 6 scoring dimensions.

Scoring Rubric (Weights sum to 100%; Max score = 100)

# Dimension Weight Description 5 (Excellent) 3 (Acceptable) 1 (Poor)
1 Data Accuracy 20% Are rank, hot_value, topic text, author, likes, and timestamps faithful to the source? All numeric fields parsed correctly; topic text matches source exactly; no hallucinated authors or likes. Minor truncation or 1-2 off-by-one rank errors; 90%+ fields correct. Major hallucinations, swapped ranks, or fabricated hot_values/authors.
2 Anti-Crawl Strategy 15% Does the output reflect robust request handling (UA rotation, referer, cookie, fallback, rate-limit)? All 6+ sources include correct headers/fallbacks and respect robots.txt/delay >=2s. Basic headers present but 1-2 sources miss fallback or delay notes. No headers, no fallback, ignores rate limits or bypasses compliance.
3 Schema Normalization 20% Are keys, types, and enums (platform, trend_direction, duration) consistent across sources? Uniform schema across all platforms; nulls handled gracefully; enums strictly normalized (up/down/stable). Minor inconsistencies (e.g., mixed snake/camel case) or missing null defaults. Wildly inconsistent schemas, invalid enums, or malformed JSON structure.
4 Platform Coverage 20% Does the extraction cover the configured platforms proportionally without omissions? All scheduled sources present and proportionally represented (weibo, bilibili, douyin, xiaohongshu, jjwxc, plus zhihu/baidu/toutiao/douban where seeded). 1 minor source missing or heavily under-sampled. Multiple configured sources completely missing.
5 Insight Extraction 15% Are related_topics, category, and trend_direction inferred or extracted meaningfully? related_topics deduplicated and aligned; category matched from source taxonomy; direction inferred from change. Some empty related_topics or generic categories, but mostly correct. No related_topics, no categories, random trend_direction values.
6 Compliance 10% Does the output respect data retention, exclude private data, and stay within ToS? Retention flag respect_platform_tos explicitly true. Minor ToS note missing; retention implied but not stated. Private data exposed, retention ignored, or explicit ToS violation.

Calculation

Each dimension is scored 1-5, then weighted:

Total = (D1 * 4 + D2 * 3 + D3 * 4 + D4 * 4 + D5 * 3 + D6 * 2)

Equivalently:

Total = D1*0.20 + D2*0.15 + D3*0.20 + D4*0.20 + D5*0.15 + D6*0.10

Max = 5.0 per dimension -> 100 total.

Target Score

>= 95 / 100

Notes for Evaluators

  1. Use the seed_data.json ground-truth for Data Accuracy checks.
  2. For Anti-Crawl Strategy, cross-check headers and fallback entries against crawl_config.json.
  3. Schema Normalization should match the JSON structure defined in crawl_config.json > sources[].parser.fields.
  4. Any output containing fabricated author, rank, or hot_value should be penalized heavily under Data Accuracy.
  5. Compliance failures are zero-tolerance items; a single PII leak caps this dimension at 1 point.