# trend-scout-cn Benchmark v1 ## Objective Evaluate the quality of Chinese social-media trending-data extraction outputs produced by the `trend-scout-cn` skill across 6 scoring dimensions. ## Scoring Rubric (Weights sum to 100%; Max score = 100) | # | Dimension | Weight | Description | 5 (Excellent) | 3 (Acceptable) | 1 (Poor) | |---|-----------|--------|-------------|---------------|----------------|----------| | 1 | Data Accuracy | 20% | Are rank, hot_value, topic text, author, likes, and timestamps faithful to the source? | All numeric fields parsed correctly; topic text matches source exactly; no hallucinated authors or likes. | Minor truncation or 1-2 off-by-one rank errors; 90%+ fields correct. | Major hallucinations, swapped ranks, or fabricated hot_values/authors. | | 2 | Anti-Crawl Strategy | 15% | Does the output reflect robust request handling (UA rotation, referer, cookie, fallback, rate-limit)? | All 6+ sources include correct headers/fallbacks and respect `robots.txt`/delay >=2s. | Basic headers present but 1-2 sources miss fallback or delay notes. | No headers, no fallback, ignores rate limits or bypasses compliance. | | 3 | Schema Normalization | 20% | Are keys, types, and enums (platform, trend_direction, duration) consistent across sources? | Uniform schema across all platforms; nulls handled gracefully; enums strictly normalized (up/down/stable). | Minor inconsistencies (e.g., mixed snake/camel case) or missing null defaults. | Wildly inconsistent schemas, invalid enums, or malformed JSON structure. | | 4 | Platform Coverage | 20% | Does the extraction cover the configured platforms proportionally without omissions? | All scheduled sources present and proportionally represented (weibo, bilibili, douyin, xiaohongshu, jjwxc, plus zhihu/baidu/toutiao/douban where seeded). | 1 minor source missing or heavily under-sampled. | Multiple configured sources completely missing. | | 5 | Insight Extraction | 15% | Are related_topics, category, and trend_direction inferred or extracted meaningfully? | related_topics deduplicated and aligned; category matched from source taxonomy; direction inferred from change. | Some empty related_topics or generic categories, but mostly correct. | No related_topics, no categories, random trend_direction values. | | 6 | Compliance | 10% | Does the output respect data retention, exclude private data, and stay within ToS? | Retention flag <=90 days; no PII leaked; `respect_platform_tos` explicitly true. | Minor ToS note missing; retention implied but not stated. | Private data exposed, retention ignored, or explicit ToS violation. | ## Calculation Each dimension is scored 1-5, then weighted: ``` Total = (D1 * 4 + D2 * 3 + D3 * 4 + D4 * 4 + D5 * 3 + D6 * 2) ``` Equivalently: ``` Total = D1*0.20 + D2*0.15 + D3*0.20 + D4*0.20 + D5*0.15 + D6*0.10 ``` Max = 5.0 per dimension -> 100 total. ## Target Score **>= 95 / 100** ## Notes for Evaluators 1. Use the `seed_data.json` ground-truth for Data Accuracy checks. 2. For Anti-Crawl Strategy, cross-check headers and fallback entries against `crawl_config.json`. 3. Schema Normalization should match the JSON structure defined in `crawl_config.json` > `sources[].parser.fields`. 4. Any output containing fabricated `author`, `rank`, or `hot_value` should be penalized heavily under Data Accuracy. 5. Compliance failures are zero-tolerance items; a single PII leak caps this dimension at 1 point.