# Global Trend Discovery: Deep Research Report ## For `trend-scout-global` Agent Template **Version:** v1 **Date:** 2026-04-13 **Scope:** Global (English / International) social-media and web-platform trend discovery APIs, parsers, schemas, and operational constraints. --- ## 1. Major Global Platforms for Trend Discovery ### 1.1 Google Trends **What it is:** Google’s official trend-exploration product showing search interest over time, by region, and by topic. **Discovery Surfaces** - `trends.google.com/trending` — real-time and daily trending searches by country. - `trends.google.com/explore` — keyword comparison, interest over time, interest by region, related queries/topics. - Year-in-Search (`top_charts`) — annual ranked lists. **Key Characteristics** - All numbers are **relative (0–100)** per query window, **not absolute** search volume. - Time-granularity auto-adjusts: short windows = hourly; ≤9 months = daily; >9 months = weekly/monthly. - Great for validating whether a social-media spike has corresponding search demand. ### 1.2 Twitter / X **What it is:** Real-time conversational trend surface for news, memes, politics, entertainment, and finance. **Discovery Surfaces** - `/explore/tabs/trending` (web) — trending timeline for your region. - `/explore/tabs/news` — news-focused trends. - `/explore/tabs/sports` — sports-focused trends. - Embedded timeline widgets and GraphQL endpoints serve these feeds. **Key Characteristics** - Fastest trend cycle of any major platform (minutes to hours). - Trends are localized by country / metro. - Heavy bot detection and aggressive endpoint rotation since 2024. ### 1.3 YouTube **What it is:** The world’s largest video platform with a public Trending feed and category-specific leaderboards. **Discovery Surfaces** - `youtube.com/feed/trending` — aggregated trending videos by country. - Category tabs: Music, Gaming, Movies, News. - YouTube Explore / Shorts trending — short-form viral content. **Key Characteristics** - Trending feed is curated, not purely algorithmic (includes editorial signals). - Strong overlap with global music releases, gaming launches, and internet culture moments. - `yt-dlp` and Invidious are the dominant unofficial access patterns. ### 1.4 Reddit **What it is:** The “front page of the internet” with highly engaged niche communities. **Discovery Surfaces** - `/r/all` — top-voted content across all public subreddits. - `/r/popular` — filtered popular feed by geography. - `/r/{subreddit}/hot`, `/r/{subreddit}/rising` — subreddit-level trends. - `/r/OutOfTheLoop`, `/r/memes`, `/r/technology`, `/r/wallstreetbets` — category proxies. **Key Characteristics** - Conversations here surface grassroots trends before mainstream media. - Rising feed is especially useful for detecting early-stage viral stories. - Strong moderation and API authentication requirements since 2023. ### 1.5 TikTok **What it is:** The dominant short-form video platform for Generation Z and Alpha global culture. **Discovery Surfaces** - TikTok Creative Center — official portal for trending hashtags, songs, creators, videos (ads and organic). - Discover page (in-app) — personalized trending hashtags and sounds. - TikTok for You Page (FYP) signals — indirectly observable via viral creator aggregators. **Key Characteristics** - Trends often start on TikTok and migrate to Instagram Reels / YouTube Shorts 24–48 hours later. - Creative Center is the only reliable *web* surface; organic FYP data requires scraping. - Strong anti-bot (device fingerprinting, TLS fingerprinting, CAPTCHA). ### 1.6 BuzzSumo-Style Content Discovery (Alternatives) **What they are:** Cross-platform content intelligence tools that show most-shared / top-performing articles and videos. **Notable Free/API-Accessible Alternatives** - **NewsAPI** — 100 req/day free; JSON news headlines by keyword, source, date. - **GNews API** — 100 req/day free; curated global news. - **Exploding Topics** — freemium trend spotting; $39/mo Pro (API usually paid). - **Fe.ly / Mention / Marketing Miner** — freemium brand-monitoring or SEO tools with limited or paid-only APIs. --- ## 2. APIs and Unofficial Endpoints ### 2.1 Google Trends (Unofficial: pytrends / py-Trends equivalents) There is **no official public API** for Google Trends. The dominant access layer is the open-source Python library **pytrends** (`GeneralMills/pytrends`, archived April 2025 but still the canonical 2024 reference). **How it works** - `TrendReq.build_payload(kw_list, timeframe, geo, gprop, cat)` constructs a request mimicking the browser. - Under the hood, Google returns a **widget JSON** payload with tokens. - Pytrends then resolves each widget (`TIMESERIES`, `GEO_MAP`, `RELATED_QUERIES`, etc.) into pandas DataFrames. **Endpoint Pattern (pseudo)** - Exploration: `POST https://trends.google.com/trends/api/explore` (returns widgets). - Widget fetch: `GET https://trends.google.com/trends/api/widgetdata/...` (tokenized). ### 2.2 Twitter / X (GraphQL Unofficial Endpoints) **Official API:** ~$100+/mo Basic tier (2024) with strict limits. Free tier largely removed. **Unofficial / Scraping Stack** - **vladkens/twscrape** — Python library that queries X’s internal GraphQL endpoints with authenticated account rotation. - Trends method: `api.trends(category)` e.g. `api.trends("news")` or `api.trends("VGltZWxpbmU6DAC2CwABAAAACHRyZW5kaW5nAAA")` - **How it works:** - Pools multiple authenticated accounts. - Automatically resolves latest GraphQL `doc_id`s. - Handles guest-token and TLS-fingerprinting updates. **Key GraphQL themes (2024)** - X rotates `doc_id`s and cookie-validation rules frequently (observed rotations in Jan, Apr, Jul, Oct 2024). - Rate limits are **hidden**; expect throttling after high-volume requests and IP bans without proxy rotation. ### 2.3 YouTube (Unofficial: InnerTube / Invidious / yt-dlp) **Official API:** YouTube Data API v3 `videos().list(chart="mostPopular")` — quota-limited (~100 units/day baseline free tier is effectively unusable at scale). **Best Unofficial Patterns** #### A) Invidious Instances (REST API) - Invidious is a privacy-friendly YouTube front-end that exposes a clean JSON API. - **Endpoint:** `GET /api/v1/trending` - **Example instances (rotate for reliability):** - `https://vid.puffyan.us/api/v1/trending` - `https://y.com.sb/api/v1/trending` - **Parameters:** - `region` (ISO country code, e.g. `US`, `GB`, `JP`) - `type` — `default`, `music`, `gaming`, `movies` - **Response:** Array of video objects with `title`, `videoId`, `author`, `authorId`, `published`, `viewCount`, `lengthSeconds`, `videoThumbnails`, `description`, etc. #### B) yt-dlp (Feed Extraction) - `yt-dlp` can directly extract `https://www.youtube.com/feed/trending`. - Under the hood it hits YouTube’s **InnerTube** `POST /youtubei/v1/browse` with `browseId: "FEtrending"`. - Good for embedding in Python data pipelines (`yt_dlp.YoutubeDL` class). ### 2.4 Reddit (Official but OAuth-Required) **Official API:** Free read access with OAuth, but unauthenticated traffic is blocked since June 2023. **Public JSON Feeds (with User-Agent)** - `https://www.reddit.com/r/all/hot.json?limit=100` - `https://www.reddit.com/r/all/rising.json?limit=100` - `https://www.reddit.com/r/popular.json?limit=100` - `https://www.reddit.com/r/{subreddit}/top.json?t=day&limit=100` **OAuth Rate Limits (2024)** - **100 queries per minute** per OAuth client ID, averaged over a 10-minute window. - Headers returned: - `X-Ratelimit-Used` - `X-Ratelimit-Remaining` - `X-Ratelimit-Reset` **Third-Party APIs (paid)** - `apidirect.io` — $0.003/request; free tier 50 req/month. - Various Reddit scraper marketplaces on RapidAPI. ### 2.5 TikTok (Unofficial Scrapers) **Official API:** TikTok for Developers API is invite-only / restricted. **Unofficial Patterns** - **Apify Actors** (best commercial fit): - `doliz/tiktok-creative-center-scraper` — trending hashtags, songs, creators, videos, products. - `codebyte/tiktok-creative-center-top-ads` — top ads and creators. - **Open-source options:** - `Russell-Newton/TikTokPy` — general TikTok profile/video scraper via Playwright. - `Shorya777/tiktok-data-scraper-rag-recommender` — open-source Creative Center scraper. - **Anti-bot layers:** sigi-state hydration, device fingerprinting, TLS fingerprinting, CAPTCHA. --- ## 3. How to Parse and Normalize Trending Data ### Normalization Pipeline Every source returns different shapes. A robust `trend-scout-global` agent should coerce all inputs into a common schema. **Recommended Normalization Steps** 1. **Timestamp normalization** — all `crawl_time` or `published` fields to ISO 8601 UTC. 2. **Geo normalization** — map region codes to ISO-3166-1 alpha-2 (`US`, `GB`, `JP`, `GLOBAL`). 3. **Category inference** — use keyword matching or LLM classification into: - `entertainment` - `technology` - `politics` - `sports` - `gaming` - `finance` - `memes / internet_culture` 4. **Engagement scoring** — where available, store raw counts; where not available (Google Trends), store the platform’s relative score with a confidence flag. 5. **Deduplication** — canonicalize titles/queries by lowercasing, stripping punctuation, and merging near-duplicates via Jaccard or embedding similarity. ### Parsing Techniques by Platform - **Google Trends:** Parse the `widgetdata` JSON; strip the leading `")]}',\n"` artifact before `json.loads()`. Use `interest_over_time` DataFrame → `isPartial` bool; `related_queries` dict of `top`/`rising` DataFrames. - **Twitter / X (twscrape):** Returns structured `Trend` models — extract `name`, `query`, `url`, `tweet_volume` (may be null), and relative `rank`. - **YouTube (Invidious):** Parse the `/api/v1/trending` JSON array directly. Each item has `title`, `videoId`, `author`, `published`, `viewCount`, `lengthSeconds`, `description`. - **Reddit:** Parse `data.children[].data` from the `.json` feed. Key fields: `title`, `subreddit`, `author`, `score`, `upvote_ratio`, `num_comments`, `url`, `created_utc`, `permalink`. - **TikTok (Apify / scraper):** Parse JSON lines from Apify dataset or local HTML/JS state (`#SIGI_STATE`). Key fields: `hashtag`, `video_count`, `creator`, `music_title`, `view_count`, `like_count`. --- ## 4. Categories of Trends | Category | Platforms of Origin | Typical Signals | |----------|---------------------|-----------------| | **Entertainment** | YouTube, TikTok, Twitter/X | Music releases, movie trailers, celebrity moments, viral dances | | **Technology** | Reddit (r/technology, r/gadgets), Twitter/X, YouTube | Product launches, AI announcements, teardowns, security news | | **Politics** | Twitter/X, Reddit (r/politics, r/worldnews), Google Trends | Election polls, debates, policy announcements, protests | | **Memes / Internet Culture** | TikTok, Twitter/X, Reddit (r/memes, r/dankmemes) | Image macros, audio trends, ironic formats, copypastas | | **Sports** | Twitter/X, YouTube, Reddit (r/sports) | Match results, athlete transfers, highlights | | **Gaming** | YouTube Gaming, Reddit (r/gaming, r/pcgaming), Twitch | Releases, speedruns, esports finals, patch notes | | **Finance** | Reddit (r/wallstreetbets), Twitter/X, Google Trends | Stock/crypto movements, earnings, macro events | --- ## 5. Rate Limits, Anti-Bot Measures, and API Key Requirements ### 5.1 Summary Table | Platform | API Key Required | Rate Limit | Anti-Bot Layers | Mitigation Strategy | |----------|------------------|------------|-----------------|---------------------| | **Google Trends** | No (unofficial) | ~1,400–2,000 req/day/IP | IP throttling, 429s | 60s backoff on 429; proxy rotation; 5 keywords max per request | | **Twitter / X** | No (unofficial) / Yes (official) | Hidden / strict on official API | TLS fingerprinting, cookie rotation, IP reputation, CAPTCHA | Account pools + proxy rotation + active library (twscrape) | | **YouTube** | No (Invidious) / Yes (official API v3) | 100 units/day free quota (official) | Bot detection on direct HTML | Use Invidious instances; rotate instances; fallback to yt-dlp | | **Reddit** | Yes (OAuth app) | 100 QPM (official); 10-min avg window | OAuth enforcement, User-Agent blocks | OAuth client + custom User-Agent + respect headers | | **TikTok** | No (unofficial) | Unclear / heavily throttled | sigi-state hydration, fingerprinting, CAPTCHA | Dedicated scraper (Apify Actor) or Playwright-based tool | | **NewsAPI** | Yes | 100 req/day (free) | API-key auth standard | Upgrade or cache aggressively | | **GNews API** | Yes | 100 req/day (free) | API-key auth standard | Upgrade or cache aggressively | ### 5.2 Operational Recommendations 1. **Assume every source will break.** Abstract each crawler behind a common interface so endpoint changes only affect one module. 2. **Run crawlers asynchronously** with bounded concurrency (e.g., `asyncio.Semaphore(3)`). 3. **Cache aggressively.** Trend data has a half-life of minutes to hours; cache feed-level results for 5–15 minutes and aggregate-level results for 1–4 hours. 4. **Rotate proxies / instances.** For Google Trends, YouTube Invidious, and Twitter/X, proxy rotation is the difference between a stable pipeline and a banned IP. 5. **Monitor health.** Track response codes, latency, and parse-failure rates per source. Disable a source automatically if it fails >30% of requests over 10 minutes. --- ## 6. Data Schemas for Trend Entries ### 6.1 Common Normalized Schema (Recommended for Agent Output) Every trend entry should normalize down to the following JSON structure: ```json { "id": "sha256-of-source+title+timestamp", "platform": "google_trends | twitter | youtube | reddit | tiktok | newsapi", "source_url": "https://...", "crawl_time": "2026-04-13T10:00:00Z", "published_time": "2026-04-13T09:30:00Z", "region": "US", "category": "technology", "title": "Apple Intelligence", "query": "Apple Intelligence", "url": "https://trends.google.com/trends/explore?q=Apple+Intelligence", "author": "", "engagement": { "score": 100, "score_type": "relative_0_100", "raw_views": null, "raw_comments": null, "raw_shares": null }, "metadata": { "rank": 1, "is_realtime": false, "tags": ["AI", "Apple"] } } ``` ### 6.2 Platform-Specific Schema Details #### Google Trends (via pytrends) - `interest_over_time` DataFrame: - Index: `DatetimeIndex` (date) - Columns: one `int64` per keyword (0–100) - Flag: `isPartial` (bool) - `related_queries`: - Dict keys: `top`, `rising` - Values: DataFrames with `query`, `value` (relative relevance) - `trending_searches`: - DataFrame with `title`, `formattedTraffic`, `relatedQueries` #### Twitter / X (via twscrape) - `Trend` object fields: - `name` (display text) - `query` (search query) - `url` (trend URL) - `tweet_volume` (int or null) - `rank` (implied by list order) #### YouTube (via Invidious) - Video object fields: - `title`, `videoId`, `author`, `authorId` - `published` (Unix timestamp or ISO) - `viewCount` (int) - `lengthSeconds` (int) - `videoThumbnails` (array of `{url, quality, width, height}`) - `description` (string) #### Reddit (via `.json` feed) - `children[].data` fields: - `title`, `subreddit_name_prefixed` (`r/...`) - `author`, `score` (net upvotes), `upvote_ratio` - `num_comments`, `url`, `permalink` - `created_utc` (Unix timestamp) - `thumbnail`, `media`, `is_video` #### TikTok (via Apify / scraper) - Creative Center trend item fields: - `hashtag` or `music_title` - `video_count` or `posts_count` - `view_count` or ` popularity_score` - `creator` (for creator trends) - `region` - `trend_type` (`hashtag`, `sound`, `video`, `creator`) --- ## 7. Data Sources and References 1. **Google Trends / pytrends** - PyPI: `https://pypi.org/project/pytrends/` - GitHub (archived 2025): `https://github.com/GeneralMills/pytrends` - HU Berlin rate-limit paper (2024): `https://edoc.hu-berlin.de/...` (working paper estimating ~2,000 queries/day) - CRAN R wrapper (2024): `https://cran.r-project.org/package=PytrendsLongitudinalR` 2. **Twitter / X Scraping** - `vladkens/twscrape`: `https://github.com/vladkens/twscrape` - Scrapfly guide: `https://scrapfly.io/blog/posts/how-to-scrape-twitter` - DeepWiki architecture: `https://deepwiki.com/vladkens/twscrape` 3. **YouTube Unofficial APIs** - Invidious API docs: `https://docs.invidious.io/` - `yt-dlp` GitHub: `https://github.com/yt-dlp/yt-dlp` - ScrapeCreators blog: `https://scrapecreators.com/blog/unofficial-youtube-api` 4. **Reddit API** - Reddit Data API Wiki: `https://support.reddithelp.com/hc/en-us/articles/16160319875092-Reddit-Data-API-Wiki` - Reddit JSON feed guide: `https://rapidevelopers.com/api-automations/how-to-automate-reddit-trend-monitoring-using-the-api` 5. **TikTok Scraping** - Apify TikTok Creative Center Scraper: `https://apify.com/doliz/tiktok-creative-center-scraper` - `Russell-Newton/TikTokPy`: `https://github.com/Russell-Newton/TikTokPy` - Scrapfly TikTok guide: `https://scrapfly.io/blog/posts/how-to-scrape-tiktok-python-json` 6. **BuzzSumo Alternatives / Content APIs** - NewsAPI: `https://newsapi.org/` - GNews API: `https://gnews.io/` - Exploding Topics alternatives article: `https://explodingtopics.com/blog/buzzsumo-alternatives` --- ## 8. Quick-Reference Cheat Sheet | Goal | Best Free Tool/API | Fallback | |------|--------------------|----------| | Validate search demand | pytrends (Google Trends) | SERP scraping | | Real-time conversation trends | twscrape (Twitter/X) | Apify X Trends Scraper | | Video viral leaderboards | Invidous `/api/v1/trending` | yt-dlp | | Grassroots early signals | Reddit OAuth JSON feeds | apidirect.io | | Short-form culture trends | Apify TikTok Creative Center | TikTokPy (Playwright) | | Global news headlines | NewsAPI | GNews API | | Cross-platform share counts | Exploding Topics (free tier) | Marketing Miner (free tier) | --- *End of knowhow_v1.md*