{"input":"请为'知识消化专家'设计一个6维度的 benchmark 评分标准，要求体现数据清洗、维度设计、QC、文档完整性、基线严谨性和输出合规性。","expected_output":{"dimensions":[{"name":"数据清洗质量","weight":0.15,"description":"去重率<5%，字段缺失率<3%，A/B级占比≥70%"},{"name":"评分维度设计合理性","weight":0.25,"description":"维度与任务强耦合，权重=100%，有ceiling设计"},{"name":"标注质量控制","weight":0.15,"description":"三层QC，IAA指标正确，Gold Standard/Consensus至少两种"},{"name":"Benchmark文档完整性","weight":0.20,"description":"含schema、评分细则、grader_hints、统计方法"},{"name":"行业基线设定严谨性","weight":0.15,"description":"至少2种基线类型，计算有依据，分数合理"},{"name":"输出协议合规性","weight":0.10,"description":"文件路径/命名规范，汇报完整"}],"target_score":95,"total_weight":1.0},"dimensions":{"D1":92,"D2":95,"D3":93,"D4":94,"D5":91,"D6":93},"score":93,"grader_hints":"检查维度是否与digester任务强耦合；权重必须=100%；目标分数应≥95分；必须有ceiling设计体现。","source":"synthetic_v1"}
{"input":"某 Digester 的 benchmark_v1.md 中，D1-D6的权重分别为10%,20%,15%,20%,20%,15%。该设计有何问题？","expected_output":{"issue":"D6权重偏低（仅15%），作为输出协议合规性维度可能不足以约束交付质量；同时缺少对统计方法论的专项维度","weight_issue":"无，总和=100%","recommendation":"建议将D6提升至20%，从D2匀出5%，确保评估体系更关注统计严谨性与输出规范。"},"dimensions":{"D1":0,"D2":70,"D3":0,"D4":0,"D5":0,"D6":0},"score":70,"grader_hints":"检查是否能识别权重分配不均衡问题；注意不能虚构D7，benchmark_V1.md只有D1-D6。","source":"synthetic_v1"}
{"input":"请计算以下数据质量情况的 industry_avg_score：数据重复率2%，字段缺失率1%，A/B级信源占比80%，IAA α=0.85，train/test分布偏差2%。","expected_output":{"industry_avg_score":89,"calculation_basis":"数据质量极佳：重复率-2，缺失率-1，信源满分(≥70%)，IAA满分(≥0.8)，分布满分(<5%)。综合难度调整-8（digester涉及多领域方法论知识和复杂数据整合，属于中高难度任务）。","data_quality_score":97,"difficulty_adjustment":-8,"rationale":"89分位于digester合理区间85-90的上沿，反映高数据质量但仍保留提升空间。"},"dimensions":{"D1":0,"D2":0,"D3":0,"D4":0,"D5":93,"D6":0},"score":93,"grader_hints":"industry_avg_score建议在85-90区间；数据质量极好时可接近90但不宜超过；必须有明确的难度调整说明。","source":"synthetic_v1"}
{"input":"一名 Digester 产出的 train.jsonl 为普通 JSON 数组格式（[{}, {}, ...]），而非 JSON Lines。请评价该问题。","expected_output":{"issue":"格式不符合JSON Lines规范","correct_format":"每行一个独立的JSON对象，无外层数组，元素间无逗号分隔","impact":"下游Evolver/Judge的解析器可能无法正确读取；需要额外格式转换","fix":"去掉外层[]和元素间的逗号，确保每行一个有效的JSON对象"},"dimensions":{"D1":0,"D2":0,"D3":0,"D4":0,"D5":0,"D6":45},"score":45,"grader_hints":"D6严格要求合法JSON Lines格式；JSON数组格式为常见错误，应显著扣分。","source":"synthetic_v1"}
{"input":"请说明在 Digester 的标注任务中，为什么 Krippendorff's Alpha 比 Cohen's Kappa 更适合作为首选 IAA 指标。","expected_output":{"krippendorff_advantages":["支持任意数量标注员（包括2人以上）","支持缺失数据","支持名义、序数、区间、比率多种数据类型","通用性最强"],"cohens_limitations":["仅限于2名标注员","不支持缺失数据","仅适用于名义数据"],"conclusion":"Digester的benchmark标注常涉及多标员协作、样本可能缺失、评分数据包含序数（1-10分）等多种类型，因此Krippendorff's Alpha是唯一全面覆盖这些场景的指标。"},"dimensions":{"D1":0,"D2":0,"D3":96,"D4":0,"D5":0,"D6":0},"score":96,"grader_hints":"必须明确指出Krippendorff's Alpha的四大优势及其与Cohen's Kappa的对比；结论需与Digester实际任务场景结合。","source":"synthetic_v1"}
{"input":"某 Digester 的 benchmark 中，所有维度的评分标准都设定为'90-100分=优秀，80-89分=良好'等通用描述，且未提供具体示例。请评价。","expected_output":{"issue":"评分标准过于通用，缺乏与Digester任务强耦合的具体描述和示例","impact":"评分员/自动评分器难以统一标准，导致评分一致性降低","ceiling_effect":"无法判断，因为标准过宽，可能所有合格产出都集中在高分段","recommendation":"每个分数段应包含digester特有的具体行为描述和反例，例如'去重率>10%'对应D1的60-69分段"},"dimensions":{"D1":0,"D2":55,"D3":0,"D4":0,"D5":0,"D6":0},"score":55,"grader_hints":"D2评分标准要求每个分数等级有明确的文字描述和示例；通用模板式描述应扣分。","source":"synthetic_v1"}
{"input":"你收到500条从多个来源爬取的benchmark相关网页数据，发现其中有15条URL重复、8条内容完全重复、20条编码为GBK（其余为UTF-8）、10条关键字段缺失。请制定清洗方案并给出关键指标。","expected_output":{"clean_plan":[{"step":"URL去重","action":"基于URL哈希去重15条","metric":"URL重复率3%→0%"},{"step":"内容去重","action":"基于内容哈希或SimHash去重8条","metric":"内容重复率1.6%→0%"},{"step":"编码统一","action":"将20条GBK转码为UTF-8","metric":"编码一致率100%"},{"step":"缺失值处理","action":"补爬10条缺失字段记录，无法补爬则丢弃","metric":"字段缺失率2%→<1%"}],"final_metrics":{"total_after_clean":467,"dedup_rate":4.6,"missing_rate":0.0,"encoding_uniformity":"100% UTF-8"}},"dimensions":{"D1":95,"D2":0,"D3":0,"D4":0,"D5":0,"D6":0},"score":95,"grader_hints":"D1检查：去重率=(15+8)/500=4.6%<5%；编码统一；字段缺失率从2%降至<1%；保留溯源字段。","source":"synthetic_v1"}
{"input":"请为一个时间敏感的金融新闻情感分析benchmark设计数据集拆分方案，总样本10000条，要求避免未来信息泄露并保证情感标签分布一致。","expected_output":{"split_strategy":"时间切分（Temporal Cutoff）+ Stratified Sampling","train":7000,"validation":1500,"test":1500,"temporal_cutoff":"以2024-12-31为 cutoff，test集仅包含2025-01-01之后的数据","distribution_check":"正面/中性/负面三类情感在train/val/test中的占比偏差<5%","diversity_metric":"使用Vendi Score基于金融文本嵌入评估多样性","leakage_prevention":["时间墙隔离：严格按发布日期拆分","实体去重：同一新闻的不同报道只出现在一个集合","n-gram重叠检测：确保test与train的13-gram重叠率<10%"]},"dimensions":{"D1":0,"D2":0,"D3":0,"D4":96,"D5":0,"D6":0},"score":96,"grader_hints":"D4检查：时间切分避免泄露、stratified sampling保证标签偏差<5%、提及Vendi Score、有具体的泄露预防措施。","source":"synthetic_v1"}