skill_reddit_scraper.md 12.9 KB

Edit Raw Blame History


Skill: Reddit Scraper for Short Drama Content

技能ID: skill_reddit_scraper
版本: v1.0.0
分类: L1-Content / Data Collection
置信度: 0.92
创建时间: 2026-03-25
创建者: Evolver Agent


1. 技能描述
从Reddit平台抓取高冲突、高戏剧性的真实故事内容，用于短剧改编。自动识别故事类型、冲突等级、人物关系，并输出结构化数据。
1.1 核心能力

多板块抓取: 支持 AITA、relationship_advice、tifu、funny 等热门板块
智能筛选: 基于戏剧分数(drama_score)和改编潜力过滤低质量内容
深度分析: 提取冲突类型、人物关系、短剧标签
标准输出: 生成符合Schema的JSON结构化数据

1.2 适用场景


场景
说明


短剧内容采编
批量获取真实故事素材


热点话题挖掘
发现 viral 潜力内容


剧本创意参考
获取冲突灵感和人物原型


用户偏好分析
分析高互动内容的特征


2. 输入参数
{
  "subreddits": {
    "type": "array",
    "description": "目标Reddit板块列表",
    "default": ["AmITheAsshole", "relationship_advice", "tifu"],
    "items": {
      "type": "string",
      "enum": ["AmITheAsshole", "relationship_advice", "tifu", "funny", "relationships"]
    }
  },
  "time_filter": {
    "type": "string",
    "description": "时间范围筛选",
    "default": "week",
    "enum": ["day", "week", "month", "year", "all"]
  },
  "sort_by": {
    "type": "string",
    "description": "排序方式",
    "default": "hot",
    "enum": ["hot", "top", "new", "rising"]
  },
  "limit": {
    "type": "integer",
    "description": "每板块抓取数量",
    "default": 50,
    "minimum": 1,
    "maximum": 500
  },
  "min_drama_score": {
    "type": "number",
    "description": "最低戏剧分数阈值",
    "default": 7.0,
    "minimum": 0,
    "maximum": 10
  },
  "include_nsfw": {
    "type": "boolean",
    "description": "是否包含NSFW内容",
    "default": false
  },
  "adaptation_focus": {
    "type": "array",
    "description": "专注的短剧类型",
    "default": ["all"],
    "items": {
      "type": "string",
      "enum": ["all", "family_conflict", "relationship_drama", "comedy", "workplace"]
    }
  }
}


3. 输出Schema
{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "required": ["metadata", "posts"],
  "properties": {
    "metadata": {
      "type": "object",
      "required": ["fetch_time", "total_posts", "subreddits"],
      "properties": {
        "fetch_time": { "type": "string", "format": "date-time" },
        "total_posts": { "type": "integer" },
        "subreddits": { "type": "array", "items": { "type": "string" } },
        "filter_stats": {
          "type": "object",
          "properties": {
            "fetched": { "type": "integer" },
            "excluded": { "type": "integer" },
            "included": { "type": "integer" }
          }
        }
      }
    },
    "posts": {
      "type": "array",
      "items": {
        "$ref": "#/definitions/RedditPost"
      }
    }
  },
  "definitions": {
    "RedditPost": {
      "type": "object",
      "required": ["id", "title", "content", "subreddit", "drama_analysis"],
      "properties": {
        "id": { "type": "string", "description": "Reddit帖子唯一ID" },
        "title": { "type": "string", "description": "帖子标题" },
        "content": { "type": "string", "description": "帖子正文内容" },
        "author": { "type": "string", "description": "作者用户名" },
        "subreddit": { "type": "string", "description": "所属板块" },
        "score": { "type": "integer", "description": "点赞数" },
        "upvote_ratio": { "type": "number", "description": "点赞比例 (0-1)" },
        "num_comments": { "type": "integer", "description": "评论数" },
        "created_utc": { "type": "integer", "description": "创建时间戳" },
        "permalink": { "type": "string", "description": "帖子链接" },
        "flair": { "type": "string", "description": "板块标签(如AITA的NTA/YTA)" },
        "is_nsfw": { "type": "boolean", "description": "是否NSFW内容" },
        "drama_analysis": {
          "type": "object",
          "required": ["drama_score", "adaptation_potential", "conflict_type"],
          "properties": {
            "drama_score": { "type": "number", "minimum": 0, "maximum": 10 },
            "adaptation_potential": { "type": "string", "enum": ["high", "medium", "low"] },
            "conflict_type": { "type": "string" },
            "conflict_level": { "type": "string", "enum": ["L1", "L2", "L3", "L4"] },
            "moral_ambiguity": { "type": "string", "enum": ["clear", "gray", "esh"] }
          }
        },
        "character_analysis": {
          "type": "object",
          "properties": {
            "characters": {
              "type": "array",
              "items": {
                "type": "object",
                "properties": {
                  "role": { "type": "string" },
                  "gender": { "type": "string", "enum": ["M", "F", "unknown"] },
                  "age": { "type": "integer" },
                  "motivation": { "type": "string" }
                }
              }
            },
            "relationships": {
              "type": "array",
              "items": {
                "type": "object",
                "properties": {
                  "from": { "type": "string" },
                  "to": { "type": "string" },
                  "type": { "type": "string" }
                }
              }
            }
          }
        },
        "short_drama_categories": {
          "type": "array",
          "items": { "type": "string" },
          "description": "短剧分类标签"
        },
        "excluded": {
          "type": "boolean",
          "description": "是否被排除"
        },
        "exclusion_reason": {
          "type": "string",
          "description": "排除原因"
        }
      }
    }
  }
}


4. 执行逻辑
4.1 主流程
┌─────────────────┐
│  1. 参数校验     │
└────────┬────────┘
         ▼
┌─────────────────┐
│  2. 抓取数据     │ ← 调用Reddit API
└────────┬────────┘
         ▼
┌─────────────────┐
│  3. 数据清洗     │ ← 去重、格式化
└────────┬────────┘
         ▼
┌─────────────────┐
│  4. 内容筛选     │ ← 排除低质量内容
└────────┬────────┘
         ▼
┌─────────────────┐
│  5. 戏剧分析     │ ← 计算drama_score
└────────┬────────┘
         ▼
┌─────────────────┐
│  6. 特征提取     │ ← 冲突/人物/标签
└────────┬────────┘
         ▼
┌─────────────────┐
│  7. 输出结果     │ ← 生成JSON
└─────────────────┘

4.2 内容筛选规则
自动排除条件:


置顶帖 (stickied = true)
链接帖 (is_self = false)
过短内容 (length < 100字符)
已删除内容 ([removed] 或 [deleted])
NSFW内容 (当 include_nsfw = false)


低质量标记:


无实质冲突描述
纯情绪宣泄无情节
广告/推广内容

4.3 Drama Score 计算算法
def calculate_drama_score(post):
    """
    计算戏剧分数 (0-10)
    基于多维度特征加权计算
    """
    score = 0

    # 1. 冲突强度 (0-3分)
    conflict_keywords = ['fight', 'argument', 'cheating', 'betrayal', 'secret', 'divorce', 'breakup']
    conflict_count = sum(1 for kw in conflict_keywords if kw in post.content.lower())
    score += min(3, conflict_count * 0.5)

    # 2. 情感张力 (0-2分)
    emotion_keywords = ['heartbroken', 'devastated', 'furious', 'shocked', 'devastated', 'crying']
    emotion_intensity = sum(1 for kw in emotion_keywords if kw in post.content.lower())
    score += min(2, emotion_intensity * 0.4)

    # 3. 反转潜力 (0-2分)
    twist_indicators = ['turns out', 'found out', 'discovered', 'realized', 'plot twist', "didn't know"]
    twist_score = sum(1 for ti in twist_indicators if ti in post.content.lower())
    score += min(2, twist_score * 0.5)

    # 4. 社区热度 (0-2分)
    # 基于upvote_ratio判断争议度，0.3-0.7为最佳
    if 0.3 <= post.upvote_ratio <= 0.7:
        score += 2
    elif 0.2 <= post.upvote_ratio <= 0.8:
        score += 1

    # 5. 板块加成 (0-1分)
    if post.subreddit == 'AmITheAsshole' and post.flair in ['YTA', 'ESH']:
        score += 1
    elif post.subreddit == 'tifu':
        score += 0.5

    return round(score, 1)

4.4 冲突类型识别


冲突类型
识别关键词
示例


family_money
inheritance, money, will, estate, property
遗产纠纷


relationship_trust
cheating, lying, trust, secret
信任危机


relationship_deception
lied, hiding, discovered, affair
欺骗背叛


workplace_boundary
coworker, boss, office, workplace
职场冲突


social_media_accident
posted, accidentally, viral, screenshot
社媒事故


embarrassing_accident
embarrassed, awkward, cringe, humiliated
尴尬事件


blended_family
stepmom, stepdad, half-sibling, divorce
重组家庭


friendship_vs_relationship
friend, jealous, choose, ultimatum
友情vs爱情


identity_swap
twin, pretended, impersonating
身份互换


secret_revelation
found out, secret, hidden, revealed
秘密揭露


5. 错误处理
5.1 错误码定义


错误码
描述
处理建议


REDDIT_API_ERROR
Reddit API调用失败
检查网络连接，稍后重试


RATE_LIMIT_EXCEEDED
请求频率超限
降低请求频率，等待重置


INVALID_SUBREDDIT
无效的板块名称
检查板块拼写


AUTHENTICATION_FAILED
认证失败
检查API密钥


TIMEOUT_ERROR
请求超时
增加超时时间，分批请求


PARSING_ERROR
数据解析失败
检查返回格式


NO_RESULTS
无匹配结果
放宽筛选条件


5.2 重试策略
retry_policy = {
    "max_attempts": 3,
    "backoff_factor": 2,
    "initial_delay": 1,  # 秒
    "retryable_errors": ["TIMEOUT_ERROR", "RATE_LIMIT_EXCEEDED", "REDDIT_API_ERROR"]
}


6. 示例代码
6.1 Python 调用示例
import requests
import json

# 调用 Skill API
response = requests.post(
    "https://api.lemonai.com/v1/skills/reddit-scraper/execute",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "subreddits": ["AmITheAsshole", "relationship_advice"],
        "time_filter": "week",
        "sort_by": "hot",
        "limit": 30,
        "min_drama_score": 7.5,
        "include_nsfw": False
    }
)

result = response.json()

# 处理结果
for post in result["posts"]:
    if post["drama_analysis"]["adaptation_potential"] == "high":
        print(f"[{post['drama_analysis']['drama_score']}] {post['title']}")
        print(f"冲突类型: {post['drama_analysis']['conflict_type']}")
        print(f"短剧标签: {', '.join(post['short_drama_categories'])}")
        print("---")

6.2 直接调用 Reddit API (备用)
import praw

# 初始化 Reddit 客户端
reddit = praw.Reddit(
    client_id="YOUR_CLIENT_ID",
    client_secret="YOUR_CLIENT_SECRET",
    user_agent="LemonAI-Scraper/1.0"
)

# 抓取板块内容
subreddit = reddit.subreddit("AmITheAsshole")
posts = subreddit.hot(limit=50)

# 筛选高戏剧内容
high_drama_posts = []
for post in posts:
    if not post.is_self or post.stickied:
        continue

    # 应用 Drama Score 算法
    drama_score = calculate_drama_score(post)

    if drama_score >= 7.0:
        high_drama_posts.append({
            "id": post.id,
            "title": post.title,
            "content": post.selftext,
            "drama_score": drama_score
        })


7. 性能指标


指标
基准值
说明


单次请求耗时

抓取50条帖子


日调用上限
1000次
受Reddit API限制


成功率
> 95%
包括重试后成功


准确率
> 90%
drama_score准确度


8. 版本历史


版本
日期
变更内容


v1.0.0
2026-03-25
初始版本，支持AITA/relationship_advice/tifu板块


9. 参考资料

Reddit API 文档
PRAW Python 库
短剧行业内容规范


技能定义版本: v1.0.0
最后更新: 2026-03-25
维护者: Evolver Agent