Global Trend Discovery: Deep Research Report
For trend-scout-global Agent Template
Version: v1
Date: 2026-04-13
Scope: Global (English / International) social-media and web-platform trend discovery APIs, parsers, schemas, and operational constraints.
1. Major Global Platforms for Trend Discovery
1.1 Google Trends
What it is: Google’s official trend-exploration product showing search interest over time, by region, and by topic.
Discovery Surfaces
trends.google.com/trending— real-time and daily trending searches by country.trends.google.com/explore— keyword comparison, interest over time, interest by region, related queries/topics.- Year-in-Search (
top_charts) — annual ranked lists.
Key Characteristics
- All numbers are relative (0–100) per query window, not absolute search volume.
- Time-granularity auto-adjusts: short windows = hourly; ≤9 months = daily; >9 months = weekly/monthly.
- Great for validating whether a social-media spike has corresponding search demand.
1.2 Twitter / X
What it is: Real-time conversational trend surface for news, memes, politics, entertainment, and finance.
Discovery Surfaces
/explore/tabs/trending(web) — trending timeline for your region./explore/tabs/news— news-focused trends./explore/tabs/sports— sports-focused trends.- Embedded timeline widgets and GraphQL endpoints serve these feeds.
Key Characteristics
- Fastest trend cycle of any major platform (minutes to hours).
- Trends are localized by country / metro.
- Heavy bot detection and aggressive endpoint rotation since 2024.
1.3 YouTube
What it is: The world’s largest video platform with a public Trending feed and category-specific leaderboards.
Discovery Surfaces
youtube.com/feed/trending— aggregated trending videos by country.- Category tabs: Music, Gaming, Movies, News.
- YouTube Explore / Shorts trending — short-form viral content.
Key Characteristics
- Trending feed is curated, not purely algorithmic (includes editorial signals).
- Strong overlap with global music releases, gaming launches, and internet culture moments.
yt-dlpand Invidious are the dominant unofficial access patterns.
1.4 Reddit
What it is: The “front page of the internet” with highly engaged niche communities.
Discovery Surfaces
/r/all— top-voted content across all public subreddits./r/popular— filtered popular feed by geography./r/{subreddit}/hot,/r/{subreddit}/rising— subreddit-level trends./r/OutOfTheLoop,/r/memes,/r/technology,/r/wallstreetbets— category proxies.
Key Characteristics
- Conversations here surface grassroots trends before mainstream media.
- Rising feed is especially useful for detecting early-stage viral stories.
- Strong moderation and API authentication requirements since 2023.
1.5 TikTok
What it is: The dominant short-form video platform for Generation Z and Alpha global culture.
Discovery Surfaces
- TikTok Creative Center — official portal for trending hashtags, songs, creators, videos (ads and organic).
- Discover page (in-app) — personalized trending hashtags and sounds.
- TikTok for You Page (FYP) signals — indirectly observable via viral creator aggregators.
Key Characteristics
- Trends often start on TikTok and migrate to Instagram Reels / YouTube Shorts 24–48 hours later.
- Creative Center is the only reliable web surface; organic FYP data requires scraping.
- Strong anti-bot (device fingerprinting, TLS fingerprinting, CAPTCHA).
1.6 BuzzSumo-Style Content Discovery (Alternatives)
What they are: Cross-platform content intelligence tools that show most-shared / top-performing articles and videos.
Notable Free/API-Accessible Alternatives
- NewsAPI — 100 req/day free; JSON news headlines by keyword, source, date.
- GNews API — 100 req/day free; curated global news.
- Exploding Topics — freemium trend spotting; $39/mo Pro (API usually paid).
- Fe.ly / Mention / Marketing Miner — freemium brand-monitoring or SEO tools with limited or paid-only APIs.
2. APIs and Unofficial Endpoints
2.1 Google Trends (Unofficial: pytrends / py-Trends equivalents)
There is no official public API for Google Trends. The dominant access layer is the open-source Python library pytrends (GeneralMills/pytrends, archived April 2025 but still the canonical 2024 reference).
How it works
TrendReq.build_payload(kw_list, timeframe, geo, gprop, cat)constructs a request mimicking the browser.- Under the hood, Google returns a widget JSON payload with tokens.
- Pytrends then resolves each widget (
TIMESERIES,GEO_MAP,RELATED_QUERIES, etc.) into pandas DataFrames.
Endpoint Pattern (pseudo)
- Exploration:
POST https://trends.google.com/trends/api/explore(returns widgets). - Widget fetch:
GET https://trends.google.com/trends/api/widgetdata/...(tokenized).
2.2 Twitter / X (GraphQL Unofficial Endpoints)
Official API: ~$100+/mo Basic tier (2024) with strict limits. Free tier largely removed.
Unofficial / Scraping Stack
- vladkens/twscrape — Python library that queries X’s internal GraphQL endpoints with authenticated account rotation.
- Trends method:
api.trends(category)
e.g.api.trends("news")orapi.trends("VGltZWxpbmU6DAC2CwABAAAACHRyZW5kaW5nAAA") - How it works:
- Pools multiple authenticated accounts.
- Automatically resolves latest GraphQL
doc_ids. - Handles guest-token and TLS-fingerprinting updates.
Key GraphQL themes (2024)
- X rotates
doc_ids and cookie-validation rules frequently (observed rotations in Jan, Apr, Jul, Oct 2024). - Rate limits are hidden; expect throttling after high-volume requests and IP bans without proxy rotation.
2.3 YouTube (Unofficial: InnerTube / Invidious / yt-dlp)
Official API: YouTube Data API v3 videos().list(chart="mostPopular") — quota-limited (~100 units/day baseline free tier is effectively unusable at scale).
Best Unofficial Patterns
A) Invidious Instances (REST API)
- Invidious is a privacy-friendly YouTube front-end that exposes a clean JSON API.
- Endpoint:
GET /api/v1/trending - Example instances (rotate for reliability):
https://vid.puffyan.us/api/v1/trendinghttps://y.com.sb/api/v1/trending
- Parameters:
region(ISO country code, e.g.US,GB,JP)type—default,music,gaming,movies
- Response: Array of video objects with
title,videoId,author,authorId,published,viewCount,lengthSeconds,videoThumbnails,description, etc.
B) yt-dlp (Feed Extraction)
yt-dlpcan directly extracthttps://www.youtube.com/feed/trending.- Under the hood it hits YouTube’s InnerTube
POST /youtubei/v1/browsewithbrowseId: "FEtrending". - Good for embedding in Python data pipelines (
yt_dlp.YoutubeDLclass).
2.4 Reddit (Official but OAuth-Required)
Official API: Free read access with OAuth, but unauthenticated traffic is blocked since June 2023.
Public JSON Feeds (with User-Agent)
https://www.reddit.com/r/all/hot.json?limit=100https://www.reddit.com/r/all/rising.json?limit=100https://www.reddit.com/r/popular.json?limit=100https://www.reddit.com/r/{subreddit}/top.json?t=day&limit=100
OAuth Rate Limits (2024)
- 100 queries per minute per OAuth client ID, averaged over a 10-minute window.
- Headers returned:
X-Ratelimit-UsedX-Ratelimit-RemainingX-Ratelimit-Reset
Third-Party APIs (paid)
apidirect.io— $0.003/request; free tier 50 req/month.- Various Reddit scraper marketplaces on RapidAPI.
2.5 TikTok (Unofficial Scrapers)
Official API: TikTok for Developers API is invite-only / restricted.
Unofficial Patterns
- Apify Actors (best commercial fit):
doliz/tiktok-creative-center-scraper— trending hashtags, songs, creators, videos, products.codebyte/tiktok-creative-center-top-ads— top ads and creators.
- Open-source options:
Russell-Newton/TikTokPy— general TikTok profile/video scraper via Playwright.Shorya777/tiktok-data-scraper-rag-recommender— open-source Creative Center scraper.
- Anti-bot layers: sigi-state hydration, device fingerprinting, TLS fingerprinting, CAPTCHA.
3. How to Parse and Normalize Trending Data
Normalization Pipeline
Every source returns different shapes. A robust trend-scout-global agent should coerce all inputs into a common schema.
Recommended Normalization Steps
- Timestamp normalization — all
crawl_timeorpublishedfields to ISO 8601 UTC. - Geo normalization — map region codes to ISO-3166-1 alpha-2 (
US,GB,JP,GLOBAL). - Category inference — use keyword matching or LLM classification into:
entertainmenttechnologypoliticssportsgamingfinancememes / internet_culture
- Engagement scoring — where available, store raw counts; where not available (Google Trends), store the platform’s relative score with a confidence flag.
- Deduplication — canonicalize titles/queries by lowercasing, stripping punctuation, and merging near-duplicates via Jaccard or embedding similarity.
Parsing Techniques by Platform
- Google Trends: Parse the
widgetdataJSON; strip the leading")]}',\n"artifact beforejson.loads(). Useinterest_over_timeDataFrame →isPartialbool;related_queriesdict oftop/risingDataFrames. - Twitter / X (twscrape): Returns structured
Trendmodels — extractname,query,url,tweet_volume(may be null), and relativerank. - YouTube (Invidious): Parse the
/api/v1/trendingJSON array directly. Each item hastitle,videoId,author,published,viewCount,lengthSeconds,description. - Reddit: Parse
data.children[].datafrom the.jsonfeed. Key fields:title,subreddit,author,score,upvote_ratio,num_comments,url,created_utc,permalink. - TikTok (Apify / scraper): Parse JSON lines from Apify dataset or local HTML/JS state (
#SIGI_STATE). Key fields:hashtag,video_count,creator,music_title,view_count,like_count.
4. Categories of Trends
| Category | Platforms of Origin | Typical Signals |
|---|---|---|
| Entertainment | YouTube, TikTok, Twitter/X | Music releases, movie trailers, celebrity moments, viral dances |
| Technology | Reddit (r/technology, r/gadgets), Twitter/X, YouTube | Product launches, AI announcements, teardowns, security news |
| Politics | Twitter/X, Reddit (r/politics, r/worldnews), Google Trends | Election polls, debates, policy announcements, protests |
| Memes / Internet Culture | TikTok, Twitter/X, Reddit (r/memes, r/dankmemes) | Image macros, audio trends, ironic formats, copypastas |
| Sports | Twitter/X, YouTube, Reddit (r/sports) | Match results, athlete transfers, highlights |
| Gaming | YouTube Gaming, Reddit (r/gaming, r/pcgaming), Twitch | Releases, speedruns, esports finals, patch notes |
| Finance | Reddit (r/wallstreetbets), Twitter/X, Google Trends | Stock/crypto movements, earnings, macro events |
5. Rate Limits, Anti-Bot Measures, and API Key Requirements
5.1 Summary Table
| Platform | API Key Required | Rate Limit | Anti-Bot Layers | Mitigation Strategy |
|---|---|---|---|---|
| Google Trends | No (unofficial) | ~1,400–2,000 req/day/IP | IP throttling, 429s | 60s backoff on 429; proxy rotation; 5 keywords max per request |
| Twitter / X | No (unofficial) / Yes (official) | Hidden / strict on official API | TLS fingerprinting, cookie rotation, IP reputation, CAPTCHA | Account pools + proxy rotation + active library (twscrape) |
| YouTube | No (Invidious) / Yes (official API v3) | 100 units/day free quota (official) | Bot detection on direct HTML | Use Invidious instances; rotate instances; fallback to yt-dlp |
| Yes (OAuth app) | 100 QPM (official); 10-min avg window | OAuth enforcement, User-Agent blocks | OAuth client + custom User-Agent + respect headers | |
| TikTok | No (unofficial) | Unclear / heavily throttled | sigi-state hydration, fingerprinting, CAPTCHA | Dedicated scraper (Apify Actor) or Playwright-based tool |
| NewsAPI | Yes | 100 req/day (free) | API-key auth standard | Upgrade or cache aggressively |
| GNews API | Yes | 100 req/day (free) | API-key auth standard | Upgrade or cache aggressively |
5.2 Operational Recommendations
- Assume every source will break. Abstract each crawler behind a common interface so endpoint changes only affect one module.
- Run crawlers asynchronously with bounded concurrency (e.g.,
asyncio.Semaphore(3)). - Cache aggressively. Trend data has a half-life of minutes to hours; cache feed-level results for 5–15 minutes and aggregate-level results for 1–4 hours.
- Rotate proxies / instances. For Google Trends, YouTube Invidious, and Twitter/X, proxy rotation is the difference between a stable pipeline and a banned IP.
- Monitor health. Track response codes, latency, and parse-failure rates per source. Disable a source automatically if it fails >30% of requests over 10 minutes.
6. Data Schemas for Trend Entries
6.1 Common Normalized Schema (Recommended for Agent Output)
Every trend entry should normalize down to the following JSON structure:
{
"id": "sha256-of-source+title+timestamp",
"platform": "google_trends | twitter | youtube | reddit | tiktok | newsapi",
"source_url": "https://...",
"crawl_time": "2026-04-13T10:00:00Z",
"published_time": "2026-04-13T09:30:00Z",
"region": "US",
"category": "technology",
"title": "Apple Intelligence",
"query": "Apple Intelligence",
"url": "https://trends.google.com/trends/explore?q=Apple+Intelligence",
"author": "",
"engagement": {
"score": 100,
"score_type": "relative_0_100",
"raw_views": null,
"raw_comments": null,
"raw_shares": null
},
"metadata": {
"rank": 1,
"is_realtime": false,
"tags": ["AI", "Apple"]
}
}
6.2 Platform-Specific Schema Details
Google Trends (via pytrends)
interest_over_timeDataFrame:- Index:
DatetimeIndex(date) - Columns: one
int64per keyword (0–100) - Flag:
isPartial(bool)
- Index:
related_queries:- Dict keys:
top,rising - Values: DataFrames with
query,value(relative relevance)
- Dict keys:
trending_searches:- DataFrame with
title,formattedTraffic,relatedQueries
- DataFrame with
Twitter / X (via twscrape)
Trendobject fields:name(display text)query(search query)url(trend URL)tweet_volume(int or null)rank(implied by list order)
YouTube (via Invidious)
- Video object fields:
title,videoId,author,authorIdpublished(Unix timestamp or ISO)viewCount(int)lengthSeconds(int)videoThumbnails(array of{url, quality, width, height})description(string)
Reddit (via .json feed)
children[].datafields:title,subreddit_name_prefixed(r/...)author,score(net upvotes),upvote_rationum_comments,url,permalinkcreated_utc(Unix timestamp)thumbnail,media,is_video
TikTok (via Apify / scraper)
- Creative Center trend item fields:
hashtagormusic_titlevideo_countorposts_countview_countorpopularity_scorecreator(for creator trends)regiontrend_type(hashtag,sound,video,creator)
7. Data Sources and References
Google Trends / pytrends
- PyPI:
https://pypi.org/project/pytrends/ - GitHub (archived 2025):
https://github.com/GeneralMills/pytrends - HU Berlin rate-limit paper (2024):
https://edoc.hu-berlin.de/...(working paper estimating ~2,000 queries/day) - CRAN R wrapper (2024):
https://cran.r-project.org/package=PytrendsLongitudinalR
- PyPI:
Twitter / X Scraping
vladkens/twscrape:https://github.com/vladkens/twscrape- Scrapfly guide:
https://scrapfly.io/blog/posts/how-to-scrape-twitter - DeepWiki architecture:
https://deepwiki.com/vladkens/twscrape
YouTube Unofficial APIs
- Invidious API docs:
https://docs.invidious.io/ yt-dlpGitHub:https://github.com/yt-dlp/yt-dlp- ScrapeCreators blog:
https://scrapecreators.com/blog/unofficial-youtube-api
- Invidious API docs:
Reddit API
- Reddit Data API Wiki:
https://support.reddithelp.com/hc/en-us/articles/16160319875092-Reddit-Data-API-Wiki - Reddit JSON feed guide:
https://rapidevelopers.com/api-automations/how-to-automate-reddit-trend-monitoring-using-the-api
- Reddit Data API Wiki:
TikTok Scraping
- Apify TikTok Creative Center Scraper:
https://apify.com/doliz/tiktok-creative-center-scraper Russell-Newton/TikTokPy:https://github.com/Russell-Newton/TikTokPy- Scrapfly TikTok guide:
https://scrapfly.io/blog/posts/how-to-scrape-tiktok-python-json
- Apify TikTok Creative Center Scraper:
BuzzSumo Alternatives / Content APIs
- NewsAPI:
https://newsapi.org/ - GNews API:
https://gnews.io/ - Exploding Topics alternatives article:
https://explodingtopics.com/blog/buzzsumo-alternatives
- NewsAPI:
8. Quick-Reference Cheat Sheet
| Goal | Best Free Tool/API | Fallback |
|---|---|---|
| Validate search demand | pytrends (Google Trends) | SERP scraping |
| Real-time conversation trends | twscrape (Twitter/X) | Apify X Trends Scraper |
| Video viral leaderboards | Invidous /api/v1/trending |
yt-dlp |
| Grassroots early signals | Reddit OAuth JSON feeds | apidirect.io |
| Short-form culture trends | Apify TikTok Creative Center | TikTokPy (Playwright) |
| Global news headlines | NewsAPI | GNews API |
| Cross-platform share counts | Exploding Topics (free tier) | Marketing Miner (free tier) |
End of knowhow_v1.md