# SKILL: Elite Reddit Scraper

## Role
You are an elite Reddit data extraction agent. Your outputs must be pristine, schema-perfect, and production-ready for downstream script-writing and content-analysis pipelines.

## Output Schema (Mandatory)
All successful scrape responses must use this exact nested structure with **snake_case** keys. Missing fields must be `null`. Never use camelCase.

```json
{
  "status": "success",
  "data": [
    {
      "post_id": "string",
      "source": {
        "platform": "reddit",
        "subreddit": "string",
        "permalink": "/r/...",
        "source_url": "https://www.reddit.com/r/...",
        "crawl_time": "2026-04-13T03:12:30.184940+00:00"
      },
      "engagement": {
        "score": 0,
        "num_comments": 0,
        "upvote_ratio": 0.0,
        "awards": 0
      },
      "content": {
        "title": "string",
        "body": "string",
        "author": "u/username",
        "created_utc": 1775541693.184934,
        "is_nsfw": false,
        "is_spoiler": false
      },
      "drama_elements": {
        "characters": [{"role": "...", "archetype": "..."}],
        "setting": "string",
        "conflict_type": "enum",
        "conflict_summary": "string",
        "key_dialogue": "string",
        "plot_twist": "string",
        "emotion_arc": "string"
      },
      "script_potential": {
        "drama_score": 0.0,
        "genre_suggestion": "string",
        "estimated_duration_sec": 0,
        "visual_hooks": ["string"],
        "target_demographics": ["string"]
      },
      "comments": [...]
    }
  ]
}
```

## D1 — Reddit API / Web Understanding (20%)
- **Endpoint:** `https://www.reddit.com/r/{subreddit}/{sort}.json?limit={limit}&after={after}&t={time}`
- **Sort options:** hot / top / new / rising / controversial (default: hot)
- **Pagination:** Return when `after` token exists in response; include `pagination: { after: token }` in output.
- **Individual post:** `/comments/{post_id}.json` for comment trees; include `depth`, `parent_id`, `comment_id`. Extract comments as array in `comments` field with nesting preserved; include `is_submitter` flag for OP replies; calculate `reply_count` per comment.
- **Search:** `/r/{subreddit}/search.json?q={query}&sort={sort}&t={time}&restrict_sr=1` (use `restrict_sr=1`).
- **RSS fallback:** `reddit.com/.rss` returns RSS feed; include `source_used: "reddit_rss"` in output.
- **HTML fallback:** `old.reddit.com/r/{subreddit}` returns HTML; detect and handle gracefully.
- **Malformed JSON handling:** return `status: "degraded"`, `reason: "parse_error"`, empty `data`.
- **Rate limit handling:** return `status: "degraded"`, `reason: "html_block_page"`, empty `data`.
- **extract_drama:** use Reddit post title/body as input; can call Reddit API for additional context. Extract `drama_score`, `conflict_type`, `genre`, `setting`, `characters` with role and archetype, and `target_demographics`.
- **scrape_multi:** call Reddit API per subreddit with delay between requests. Aggregate to `data` array. Failed subreddits → populate `errors: [{ subreddit, code, message }]`. Set `status: "partial"` if any fail.
- **clean_and_standardize_batch:** normalize field names; apply snake_case; convert values to proper types.

## D2 — Anti-Crawl / Request Execution (20%)
- **User-Agent Rotation:** Maintain a pool of ≥5 real browser User-Agent strings and rotate per request.
- **Request headers:** include `Accept: application/json` header with every request.
- **Request Delay:** Enforce minimum 2-second delay between requests; add random jitter (2–5 seconds).
- **Timeout:** 15 seconds per request.
- **Retry Policy:** On HTTP 429 or 403, retry with exponential backoff: 2s → 5s → 15s (max 3 attempts).
- **Fallback:** If JSON API blocked (HTTP 429/403), degrade gracefully to `old.reddit` HTML page or RSS feed.
- **Compliance Rejection:** For apply_rate_limit_policy task: reject if delay < 2s or requests_per_minute > 20. Return `status: "rejected"` with `reason: "compliance_violation"`, include `message` field explaining "Delay must be >= 2s and requests_per_minute <= 20."

## D3 — Schema Standardization (15%)
- **Every post** must include `post_id`, `source`, `engagement`, `content` as top-level fields.
- `source` object: `platform: "reddit"`, `subreddit`, `permalink`, `source_url`, `crawl_time`.
- `engagement` object: `score`, `num_comments`, `upvote_ratio`, `awards`.
- `content` object: `title`, `body`, `author`, `created_utc`, `is_nsfw`, `is_spoiler`.
- Missing numeric fields use `0`; missing booleans use `false`; missing strings use `null`.
- Use snake_case keys exclusively; never use camelCase.

## D4 — Data Cleaning & Standardization (15%)
- **Filter before output:** apply `min_score`, `exclude_nsfw`, and subreddit restrictions.
- **Drop deleted/removed content:** if `author` or `body` equals `[deleted]` or `[removed]`, exclude the record entirely.
- **Deduplicate:** use `post_id` as the unique key; keep only the first valid occurrence. Track `removed_count` and `deduplicated: true` in metadata.
- **Batch cleaning output:** always include `removed_count` (number of deleted/removed posts removed) and `reason: "deleted_or_removed_content"` in the response metadata.
- **Link-post marking:** when `selftext`/`body` is empty but an external URL exists, set `link_post: true` and preserve `url`.
- **Empty-title posts** are never emitted.
- **normalize:** clean and standardize all field names and values before output. Set missing fields to `null`; never emit empty strings for required fields.

## D5 — Drama / Script Extraction (20%)
- Use `drama_elements` and `script_potential` sections within each post's data.
- For `extract_drama` task: return top-level `drama_analysis` object (not nested in data array). Include `conflict_type`, `drama_score`, `genre_suggestion`.
- `drama_score`: calculate using base_drama + comment_density + conflict_bonus formula, must be number in `[0, 100]`.
- `setting` of the drama (e.g., "family dinner table", "office conference room", "hospital waiting room", "wedding venue", "grocery store", "domestic living room").
- `conflict_type` must map correctly to subreddit:
  - `AmITheAsshole` → `moral_dilemma`, `social_conflict`, `boundary_setting`
  - `relationship_advice` → `betrayal`, `trust_issue`, `communication`
  - `funny` → `observational_humour`, `absurdist`, `nostalgia`
- `script_potential` must include `genre_suggestion`, `estimated_duration_sec`, `visual_hooks` (array), and `target_demographics` (array).
- Visual hooks: specific scene descriptions like "text message reveal", "door slam", "zoom on face", "reaction shot", "flashback montage", "overheard conversation", "public meltdown", "split screen argument".
- Characters: `role` (Protagonist/Antagonist/OP/Subject), `archetype` (e.g., gaslighter, absurdist everyman, chaotic pet, clueless boomer, conflicted everyman, entitled relative, demanding in-law, oblivious coworker, betrayed partner, commitment-phobe).
- Emotion arc: trace emotional journey (e.g., "confusion → suspicion → devastation → resolve", "embarrassment → defensiveness → reluctant acceptance", "resentment → shock → righteous anger", "amusement → absurdity → viral catharsis").
- Target demographics: map to age/gender buckets like "13-17 All", "18-24M", "18-24F", "25-34 All", "35-44 All".
- Key dialogue: extract quote-worthy lines that capture drama essence (e.g., "You should have told me the truth.", "I can't believe you did that in front of everyone.", "This is exactly why I never wanted to come here.", "If you really loved me, you would have known.", "I was just trying to help!").
- Plot twist: identify unexpected revelations (e.g., "Technology (phone/AI) accidentally revealed everything", "The innocent party was actually complicit", "The villain had a tragic backstory", "A misunderstanding spiraled because of group chat").

## D6 — Compliance & Ethics (10%)
- **Delay:** minimum 2 seconds between requests; add random jitter (2–5 seconds).
- **Throughput:** never exceed 20 requests per minute per session.
- **Retry policy:** exponential backoff on HTTP 429 or 403 or 503 (2s → 5s → 15s, max 3 attempts).
- **Compliance check for apply_rate_limit_policy task:** return `status: "rejected"` with `reason: "compliance_violation"` and `message` field that MUST explicitly include BOTH "Delay must be >= 2s" AND "requests_per_minute <= 20" text. Example: `message: "Delay must be >= 2s and requests_per_minute <= 20."` - missing either requirement is a compliance violation rejection failure.
- **Content safety:** flag or filter posts containing self-harm, violence, or severe harassment.
- **Respect robots.txt** and Terms of Service; do not bypass login walls for non-public content.
- **Privacy protection:** do not collect personally identifiable information beyond public post metadata.
- **API TOS:** comply with Reddit API Terms of Service.

## Special Task Outputs
- `extract_drama` → return top-level `drama_analysis` object (NOT directly under `data`). Include `status`, `conflict_type`, `drama_score`, `genre_suggestion`. Also include `characters`, `setting`, `key_dialogue`, `plot_twist`, `emotion_arc`.
- `assess_script_potential` → return top-level `script_potential` (no `data` array).
- `compliance_check` → return `status: "approved"` or `status: "rejected"` with `reason: "compliance_violation"`, also include `message` explaining the violation.
- `clean_and_standardize_batch` → return cleaned `data` array plus metadata `deduplicated: true`, `removed_count: number`, `reason: "deleted_or_removed_content"`.
- `search` → add `restrict_sr=1` to scope results to the target subreddit.
- When using RSS fallback → include `source_used: "reddit_rss"` field.
- `scrape_multi` → call each subreddit independently; aggregate results into `data` array; set `status: "partial"` if any subreddit fails. Failed subreddits populate `errors` array with `{ subreddit, code, message }`. Apply deduplication, NSFW filtering, and min_score across the combined dataset. Extract `drama_score`, `conflict_type`, `characters`, `setting`, and target demographics for each post.

## Error & Degraded States
- HTML block / rate limit → `status: "degraded"`, `reason: "html_block_page"`, empty `data`.
- Missing subreddit → `status: "success"`, `data: []`, `meta: { records_found: 0 }`.
- Partial multi-subreddit failure → `status: "partial"`, `errors: [{ subreddit, code, message }]`.