skill_satire_scraper.md
17.7 KB
Skill: Satire News Scraper for Short Drama
技能ID: skill_satire_news_scraper 版本: v1.0.0 分类: L1-Content / Satire Analysis 置信度: 0.89 创建时间: 2026-03-25 创建者: Evolver Agent
1. 技能描述
从讽刺新闻网站抓取内容，识别荒诞元素，评估短剧改编潜力，输出结构化数据供短剧创作使用。
1.1 核心能力
- 多源抓取: 支持 The Onion、ClickHole、Babylon Bee 等8个主流讽刺新闻源
- 讽刺识别: 自动识别7种讽刺类型，计算荒诞指数(absurdity_score)
- 短剧适配: 评估内容的短剧改编潜力，提取场景/人物/对话元素
- 智能输出: 生成标准化的JSON输出，包含完整的短剧改编建议
1.2 支持的数据源
| 网站 | 类型 | 抓取方式 | 优先级 |
|---|---|---|---|
| The Onion | 死板式新闻 | 网页抓取 | P0 |
| ClickHole | 病毒内容 | RSS | P0 |
| Babylon Bee | 保守讽刺 | RSS | P1 |
| Reductress | 女性媒体讽刺 | RSS | P1 |
| The Hard Times | 朋克文化 | RSS | P1 |
| McSweeney's | 文学幽默 | RSS | P2 |
| The Beaverton | 加拿大讽刺 | RSS | P2 |
| Waterford Whispers | 爱尔兰讽刺 | RSS | P2 |
2. 输入参数
{
"sources": {
"type": "array",
"description": "目标讽刺新闻源列表",
"default": ["the_onion", "clickhole", "babylon_bee"],
"items": {
"type": "string",
"enum": ["the_onion", "clickhole", "babylon_bee", "reductress", "the_hard_times", "mcsweeneys", "the_beaverton", "waterford_whispers"]
}
},
"fetch_mode": {
"type": "string",
"description": "抓取模式",
"default": "rss",
"enum": ["rss", "web", "auto"]
},
"limit_per_source": {
"type": "integer",
"description": "每源抓取数量",
"default": 20,
"minimum": 1,
"maximum": 100
},
"min_absurdity_score": {
"type": "number",
"description": "最小荒诞分数阈值",
"default": 6.0,
"minimum": 0,
"maximum": 10
},
"adaptation_filter": {
"type": "string",
"description": "短剧适配等级筛选",
"default": "B",
"enum": ["A", "B", "C", "D", "all"]
},
"include_political": {
"type": "boolean",
"description": "是否包含政治讽刺内容(需本地化)",
"default": false
},
"extract_drama_elements": {
"type": "boolean",
"description": "是否提取短剧元素(场景/人物/对话)",
"default": true
}
}
3. 输出Schema
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"required": ["metadata", "articles"],
"properties": {
"metadata": {
"type": "object",
"required": ["fetch_time", "total_articles", "sources"],
"properties": {
"fetch_time": { "type": "string", "format": "date-time" },
"total_articles": { "type": "integer" },
"sources": { "type": "array", "items": { "type": "string" } },
"filter_stats": {
"type": "object",
"properties": {
"fetched": { "type": "integer" },
"filtered_by_absurdity": { "type": "integer" },
"filtered_by_adaptation": { "type": "integer" },
"included": { "type": "integer" }
}
}
}
},
"articles": {
"type": "array",
"items": {
"$ref": "#/definitions/SatireArticle"
}
}
},
"definitions": {
"SatireArticle": {
"type": "object",
"required": ["id", "title", "content", "source", "satire_analysis"],
"properties": {
"id": { "type": "string", "description": "文章唯一ID" },
"title": { "type": "string", "description": "文章标题" },
"content": { "type": "string", "description": "文章正文内容" },
"source": { "type": "string", "description": "来源网站" },
"category": { "type": "string", "description": "文章分类" },
"url": { "type": "string", "description": "原文链接" },
"published_at": { "type": "string", "format": "date-time" },
"satire_analysis": {
"type": "object",
"required": ["absurdity_score", "satire_type", "markers"],
"properties": {
"absurdity_score": {
"type": "number",
"minimum": 0,
"maximum": 10,
"description": "荒诞指数 0-10"
},
"satire_type": {
"type": "string",
"enum": ["deadpan_news", "viral_content", "conservative_satire", "feminist_media_satire", "subculture_satire", "literary_humor", "national_satire", "mild_satire", "none"]
},
"markers": {
"type": "array",
"items": { "type": "string" },
"description": "命中的讽刺标记词"
},
"premise": { "type": "string", "description": "荒诞前提摘要" },
"twist": { "type": "string", "description": "反转/笑点说明" }
}
},
"drama_adaptation": {
"type": "object",
"required": ["adaptability_rating", "recommended_duration"],
"properties": {
"adaptability_rating": {
"type": "string",
"enum": ["A", "B", "C", "D"],
"description": "A=直接改编, B=需调整, C=大幅改写, D=不适合"
},
"recommended_duration": {
"type": "string",
"enum": ["15s", "60s", "3-5min"],
"description": "建议短剧时长"
},
"adaptation_notes": { "type": "string", "description": "改编建议" },
"localization_required": { "type": "boolean" }
}
},
"drama_elements": {
"type": "object",
"properties": {
"setting": {
"type": "object",
"properties": {
"location": { "type": "string" },
"mood": { "type": "string" }
}
},
"characters": {
"type": "array",
"items": {
"type": "object",
"properties": {
"role": { "type": "string", "enum": ["protagonist", "antagonist", "supporting"] },
"type": { "type": "string" },
"trait": { "type": "string" }
}
}
},
"dialogue": {
"type": "array",
"items": { "type": "string" },
"description": "可直接使用的对话"
},
"visual_gags": {
"type": "array",
"items": { "type": "string" },
"description": "视觉笑点建议"
}
}
}
}
}
}
}
4. 执行逻辑
4.1 主流程
┌─────────────────┐
│ 1. 参数校验 │
└────────┬────────┘
▼
┌─────────────────┐
│ 2. 数据源抓取 │ ← RSS/网页抓取
└────────┬────────┘
▼
┌─────────────────┐
│ 3. 数据清洗 │ ← 去重、格式化
└────────┬────────┘
▼
┌─────────────────┐
│ 4. 讽刺分析 │ ← 计算absurdity_score
└────────┬────────┘
▼
┌─────────────────┐
│ 5. 短剧适配评估 │ ← 评级A/B/C/D
└────────┬────────┘
▼
┌─────────────────┐
│ 6. 元素提取 │ ← 场景/人物/对话
└────────┬────────┘
▼
┌─────────────────┐
│ 7. 输出结果 │ ← 生成JSON
└─────────────────┘
4.2 数据源抓取策略
RSS 模式(推荐)
RSS_FEEDS = {
"clickhole": "https://clickhole.com/feed/",
"babylon_bee": "https://babylonbee.com/feed/",
"reductress": "https://reductress.com/feed/",
"the_hard_times": "https://thehardtimes.net/feed/",
"mcsweeneys": "http://feeds.feedburner.com/mcsweeneys",
"the_beaverton": "https://www.thebeaverton.com/feed/",
"waterford_whispers": "https://waterfordwhispersnews.com/feed/"
}
网页抓取模式(The Onion)
WEB_SELECTORS = {
"container": "article.post",
"title": "h1.headline",
"content": ".entry-content",
"category": ".category-tag"
}
4.3 Absurdity Score 计算算法
def calculate_absurdity_score(article):
"""
计算荒诞指数 (0-10)
基于四大维度加权计算
"""
# 1. 标题讽刺标记 (0-3分)
headline_markers = {
"local_man": ["Local Man", "Area Man", "Area Woman"],
"study_finds": ["Study Finds", "Report:", "Experts Say"],
"absurd_escalation": ["somehow", "unexpectedly", "tragically"],
"literal_interpretation": ["literally", "actually", "definitely"],
"emotional_manipulation": ["Heartbreaking", "Devastating", "Inspiring"]
}
headline_score = 0
title_lower = article.title.lower()
for marker_type, markers in headline_markers.items():
if any(m.lower() in title_lower for m in markers):
headline_score += 0.6
headline_score = min(3, headline_score)
# 2. 内容荒诞密度 (0-2.5分)
absurdity_keywords = [
"aliens", "time travel", "talking", "supernatural",
"billion", "million", "37", "entire family",
"turns out", "little did they know", "shocked to discover"
]
content_lower = article.content.lower()
keyword_count = sum(1 for kw in absurdity_keywords if kw in content_lower)
density_score = min(2.5, keyword_count * 0.3)
# 3. 叙事反转强度 (0-2.5分)
twist_indicators = [
("somehow", 1.0),
("turns out", 0.8),
("unexpectedly", 0.8),
("but actually", 0.7),
("discovered that", 0.6)
]
twist_score = sum(score for indicator, score in twist_indicators
if indicator in content_lower)
twist_score = min(2.5, twist_score)
# 4. 短剧适配度 (0-2分)
adaptability_indicators = {
"scene_clear": ["office", "home", "restaurant", "party"],
"dialogue_rich": ['"', "said", "told"],
"character_distinct": ["local man", "dad", "boss", "teacher"]
}
adapt_score = 0
for indicator_type, keywords in adaptability_indicators.items():
if any(kw in content_lower for kw in keywords):
adapt_score += 0.7
adapt_score = min(2, adapt_score)
total = headline_score + density_score + twist_score + adapt_score
return round(total, 1)
4.4 讽刺类型识别规则
| 类型 | 识别规则 | 代表源 |
|---|---|---|
deadpan_news |
AP新闻格式 + 荒谬内容 | The Onion |
viral_content |
列表/测验/点击诱饵格式 | ClickHole |
conservative_satire |
传统价值观视角反转 | Babylon Bee |
feminist_media_satire |
女性媒体风格 + relatability | Reductress |
subculture_satire |
亚文化黑话 + 内部梗 | The Hard Times |
literary_humor |
高品位文学风格 | McSweeney's |
national_satire |
国家/地域特色讽刺 | Beaverton/Waterford |
识别函数
def detect_satire_type(article):
source_mapping = {
"the_onion": "deadpan_news",
"clickhole": "viral_content",
"babylon_bee": "conservative_satire",
"reductress": "feminist_media_satire",
"the_hard_times": "subculture_satire",
"mcsweeneys": "literary_humor",
"the_beaverton": "national_satire",
"waterford_whispers": "national_satire"
}
# 基于来源初步判断
base_type = source_mapping.get(article.source, "unknown")
# 基于内容微调
title_lower = article.title.lower()
if "quiz" in title_lower or "which" in title_lower:
return "viral_content"
elif any(x in title_lower for x in ["study finds", "report:", "local man"]):
return "deadpan_news"
elif "open letter" in title_lower:
return "literary_humor"
return base_type
4.5 短剧适配评级算法
def assess_drama_adaptation(article, absurdity_score):
"""
评估短剧改编潜力，返回 A/B/C/D 评级
"""
scores = {
"scene_visualization": 0, # 场景可视化 (0-2.5)
"dialogue_richness": 0, # 对话丰富度 (0-2.5)
"conflict_clarity": 0, # 冲突明确性 (0-2.0)
"twist_impact": 0, # 反转冲击力 (0-2.0)
"performance_feasibility": 0 # 表演可行性 (0-1.0)
}
content = article.content.lower()
title = article.title.lower()
# 场景可视化
locations = ["office", "home", "restaurant", "party", "wedding", "hospital"]
if any(loc in content for loc in locations):
scores["scene_visualization"] = 2.0
elif "local" in title:
scores["scene_visualization"] = 1.5
else:
scores["scene_visualization"] = 1.0
# 对话丰富度
quote_count = content.count('"')
if quote_count >= 4:
scores["dialogue_richness"] = 2.5
elif quote_count >= 2:
scores["dialogue_richness"] = 1.8
elif "said" in content or "told" in content:
scores["dialogue_richness"] = 1.2
else:
scores["dialogue_richness"] = 0.8
# 冲突明确性
conflict_keywords = ["refuses", "demanding", "accused", "argument", "fight"]
if any(kw in content for kw in conflict_keywords):
scores["conflict_clarity"] = 2.0
elif absurdity_score > 7:
scores["conflict_clarity"] = 1.5
else:
scores["conflict_clarity"] = 1.0
# 反转冲击力
if absurdity_score >= 8:
scores["twist_impact"] = 2.0
elif absurdity_score >= 6.5:
scores["twist_impact"] = 1.5
else:
scores["twist_impact"] = 0.8
# 表演可行性
if any(x in content for x in ["dragon", "alien", "explosion"]):
scores["performance_feasibility"] = 0.3 # 需要特效
else:
scores["performance_feasibility"] = 1.0
total = sum(scores.values())
# 评级标准
if total >= 8 and absurdity_score >= 7:
return "A" # 直接改编
elif total >= 6.5:
return "B" # 需少量调整
elif total >= 4.5:
return "C" # 需大幅改写
else:
return "D" # 不适合
5. 错误处理
5.1 错误码定义
| 错误码 | 描述 | 处理建议 |
|---|---|---|
RSS_FETCH_ERROR |
RSS feed获取失败 | 切换至网页抓取模式 |
WEB_SCRAPE_ERROR |
网页抓取失败 | 检查选择器，降低请求频率 |
RATE_LIMITED |
请求频率超限 | 降低频率至20req/min，使用代理 |
PARSE_ERROR |
内容解析失败 | 检查编码(强制UTF-8) |
NO_RESULTS |
无匹配结果 | 放宽筛选条件 |
TIMEOUT_ERROR |
请求超时 | 增加超时时间，分批请求 |
5.2 重试策略
retry_policy = {
"max_attempts": 3,
"backoff_factor": 2,
"initial_delay": 1,
"retryable_errors": ["TIMEOUT_ERROR", "RATE_LIMITED", "RSS_FETCH_ERROR"]
}
6. 示例代码
6.1 Python 调用示例
import requests
import json
# 调用 Skill API
response = requests.post(
"https://api.lemonai.com/v1/skills/satire-scraper/execute",
headers={"Authorization": "Bearer YOUR_API_KEY"},
json={
"sources": ["the_onion", "clickhole", "babylon_bee"],
"fetch_mode": "auto",
"limit_per_source": 10,
"min_absurdity_score": 7.0,
"adaptation_filter": "B",
"extract_drama_elements": True
}
)
result = response.json()
# 处理高适配度内容
for article in result["articles"]:
if article["drama_adaptation"]["adaptability_rating"] == "A":
print(f"[{article['satire_analysis']['absurdity_score']}] {article['title']}")
print(f"讽刺类型: {article['satire_analysis']['satire_type']}")
print(f"荒诞前提: {article['satire_analysis']['premise']}")
print(f"建议时长: {article['drama_adaptation']['recommended_duration']}")
print("---")
6.2 直接抓取示例
import feedparser
# RSS 抓取示例
def fetch_from_rss(source_url):
feed = feedparser.parse(source_url)
articles = []
for entry in feed.entries[:20]:
article = {
"title": entry.title,
"content": entry.get("summary", entry.get("description", "")),
"url": entry.link,
"published_at": entry.get("published", "")
}
# 应用 absurdity_score 算法
article["absurdity_score"] = calculate_absurdity_score(article)
if article["absurdity_score"] >= 6.0:
articles.append(article)
return articles
# 抓取 ClickHole
clickhole_articles = fetch_from_rss("https://clickhole.com/feed/")
7. 性能指标
| 指标 | 基准值 | 说明 |
|---|---|---|
| 单次请求耗时 | 抓取3个源，每源20条 | |
| 日调用上限 | 500次 | 受数据源限制 |
| 成功率 | > 95% | 包括重试后成功 |
| absurdity_score MAE | 与人工评分平均绝对误差 | |
| 类型分类准确率 | > 75% | 讽刺类型识别准确率 |
8. 版本历史
| 版本 | 日期 | 变更内容 |
|---|---|---|
| v1.0.0 | 2026-03-25 | 初始版本，支持8个讽刺新闻源 |
9. 参考资料
技能定义版本: v1.0.0 最后更新: 2026-03-25 维护者: Evolver Agent