knowhow_v1.md 17.7 KB

Global Trend Discovery: Deep Research Report

For trend-scout-global Agent Template

Version: v1
Date: 2026-04-13
Scope: Global (English / International) social-media and web-platform trend discovery APIs, parsers, schemas, and operational constraints.


1. Major Global Platforms for Trend Discovery

What it is: Google’s official trend-exploration product showing search interest over time, by region, and by topic.

Discovery Surfaces

  • trends.google.com/trending — real-time and daily trending searches by country.
  • trends.google.com/explore — keyword comparison, interest over time, interest by region, related queries/topics.
  • Year-in-Search (top_charts) — annual ranked lists.

Key Characteristics

  • All numbers are relative (0–100) per query window, not absolute search volume.
  • Time-granularity auto-adjusts: short windows = hourly; ≤9 months = daily; >9 months = weekly/monthly.
  • Great for validating whether a social-media spike has corresponding search demand.

1.2 Twitter / X

What it is: Real-time conversational trend surface for news, memes, politics, entertainment, and finance.

Discovery Surfaces

  • /explore/tabs/trending (web) — trending timeline for your region.
  • /explore/tabs/news — news-focused trends.
  • /explore/tabs/sports — sports-focused trends.
  • Embedded timeline widgets and GraphQL endpoints serve these feeds.

Key Characteristics

  • Fastest trend cycle of any major platform (minutes to hours).
  • Trends are localized by country / metro.
  • Heavy bot detection and aggressive endpoint rotation since 2024.

1.3 YouTube

What it is: The world’s largest video platform with a public Trending feed and category-specific leaderboards.

Discovery Surfaces

  • youtube.com/feed/trending — aggregated trending videos by country.
  • Category tabs: Music, Gaming, Movies, News.
  • YouTube Explore / Shorts trending — short-form viral content.

Key Characteristics

  • Trending feed is curated, not purely algorithmic (includes editorial signals).
  • Strong overlap with global music releases, gaming launches, and internet culture moments.
  • yt-dlp and Invidious are the dominant unofficial access patterns.

1.4 Reddit

What it is: The “front page of the internet” with highly engaged niche communities.

Discovery Surfaces

  • /r/all — top-voted content across all public subreddits.
  • /r/popular — filtered popular feed by geography.
  • /r/{subreddit}/hot, /r/{subreddit}/rising — subreddit-level trends.
  • /r/OutOfTheLoop, /r/memes, /r/technology, /r/wallstreetbets — category proxies.

Key Characteristics

  • Conversations here surface grassroots trends before mainstream media.
  • Rising feed is especially useful for detecting early-stage viral stories.
  • Strong moderation and API authentication requirements since 2023.

1.5 TikTok

What it is: The dominant short-form video platform for Generation Z and Alpha global culture.

Discovery Surfaces

  • TikTok Creative Center — official portal for trending hashtags, songs, creators, videos (ads and organic).
  • Discover page (in-app) — personalized trending hashtags and sounds.
  • TikTok for You Page (FYP) signals — indirectly observable via viral creator aggregators.

Key Characteristics

  • Trends often start on TikTok and migrate to Instagram Reels / YouTube Shorts 24–48 hours later.
  • Creative Center is the only reliable web surface; organic FYP data requires scraping.
  • Strong anti-bot (device fingerprinting, TLS fingerprinting, CAPTCHA).

1.6 BuzzSumo-Style Content Discovery (Alternatives)

What they are: Cross-platform content intelligence tools that show most-shared / top-performing articles and videos.

Notable Free/API-Accessible Alternatives

  • NewsAPI — 100 req/day free; JSON news headlines by keyword, source, date.
  • GNews API — 100 req/day free; curated global news.
  • Exploding Topics — freemium trend spotting; $39/mo Pro (API usually paid).
  • Fe.ly / Mention / Marketing Miner — freemium brand-monitoring or SEO tools with limited or paid-only APIs.

2. APIs and Unofficial Endpoints

There is no official public API for Google Trends. The dominant access layer is the open-source Python library pytrends (GeneralMills/pytrends, archived April 2025 but still the canonical 2024 reference).

How it works

  • TrendReq.build_payload(kw_list, timeframe, geo, gprop, cat) constructs a request mimicking the browser.
  • Under the hood, Google returns a widget JSON payload with tokens.
  • Pytrends then resolves each widget (TIMESERIES, GEO_MAP, RELATED_QUERIES, etc.) into pandas DataFrames.

Endpoint Pattern (pseudo)

  • Exploration: POST https://trends.google.com/trends/api/explore (returns widgets).
  • Widget fetch: GET https://trends.google.com/trends/api/widgetdata/... (tokenized).

2.2 Twitter / X (GraphQL Unofficial Endpoints)

Official API: ~$100+/mo Basic tier (2024) with strict limits. Free tier largely removed.

Unofficial / Scraping Stack

  • vladkens/twscrape — Python library that queries X’s internal GraphQL endpoints with authenticated account rotation.
  • Trends method: api.trends(category)
    e.g. api.trends("news") or api.trends("VGltZWxpbmU6DAC2CwABAAAACHRyZW5kaW5nAAA")
  • How it works:
    • Pools multiple authenticated accounts.
    • Automatically resolves latest GraphQL doc_ids.
    • Handles guest-token and TLS-fingerprinting updates.

Key GraphQL themes (2024)

  • X rotates doc_ids and cookie-validation rules frequently (observed rotations in Jan, Apr, Jul, Oct 2024).
  • Rate limits are hidden; expect throttling after high-volume requests and IP bans without proxy rotation.

2.3 YouTube (Unofficial: InnerTube / Invidious / yt-dlp)

Official API: YouTube Data API v3 videos().list(chart="mostPopular") — quota-limited (~100 units/day baseline free tier is effectively unusable at scale).

Best Unofficial Patterns

A) Invidious Instances (REST API)

  • Invidious is a privacy-friendly YouTube front-end that exposes a clean JSON API.
  • Endpoint: GET /api/v1/trending
  • Example instances (rotate for reliability):
    • https://vid.puffyan.us/api/v1/trending
    • https://y.com.sb/api/v1/trending
  • Parameters:
    • region (ISO country code, e.g. US, GB, JP)
    • type — default, music, gaming, movies
  • Response: Array of video objects with title, videoId, author, authorId, published, viewCount, lengthSeconds, videoThumbnails, description, etc.

B) yt-dlp (Feed Extraction)

  • yt-dlp can directly extract https://www.youtube.com/feed/trending.
  • Under the hood it hits YouTube’s InnerTube POST /youtubei/v1/browse with browseId: "FEtrending".
  • Good for embedding in Python data pipelines (yt_dlp.YoutubeDL class).

2.4 Reddit (Official but OAuth-Required)

Official API: Free read access with OAuth, but unauthenticated traffic is blocked since June 2023.

Public JSON Feeds (with User-Agent)

  • https://www.reddit.com/r/all/hot.json?limit=100
  • https://www.reddit.com/r/all/rising.json?limit=100
  • https://www.reddit.com/r/popular.json?limit=100
  • https://www.reddit.com/r/{subreddit}/top.json?t=day&limit=100

OAuth Rate Limits (2024)

  • 100 queries per minute per OAuth client ID, averaged over a 10-minute window.
  • Headers returned:
    • X-Ratelimit-Used
    • X-Ratelimit-Remaining
    • X-Ratelimit-Reset

Third-Party APIs (paid)

  • apidirect.io — $0.003/request; free tier 50 req/month.
  • Various Reddit scraper marketplaces on RapidAPI.

2.5 TikTok (Unofficial Scrapers)

Official API: TikTok for Developers API is invite-only / restricted.

Unofficial Patterns

  • Apify Actors (best commercial fit):
    • doliz/tiktok-creative-center-scraper — trending hashtags, songs, creators, videos, products.
    • codebyte/tiktok-creative-center-top-ads — top ads and creators.
  • Open-source options:
    • Russell-Newton/TikTokPy — general TikTok profile/video scraper via Playwright.
    • Shorya777/tiktok-data-scraper-rag-recommender — open-source Creative Center scraper.
  • Anti-bot layers: sigi-state hydration, device fingerprinting, TLS fingerprinting, CAPTCHA.

Normalization Pipeline

Every source returns different shapes. A robust trend-scout-global agent should coerce all inputs into a common schema.

Recommended Normalization Steps

  1. Timestamp normalization — all crawl_time or published fields to ISO 8601 UTC.
  2. Geo normalization — map region codes to ISO-3166-1 alpha-2 (US, GB, JP, GLOBAL).
  3. Category inference — use keyword matching or LLM classification into:
    • entertainment
    • technology
    • politics
    • sports
    • gaming
    • finance
    • memes / internet_culture
  4. Engagement scoring — where available, store raw counts; where not available (Google Trends), store the platform’s relative score with a confidence flag.
  5. Deduplication — canonicalize titles/queries by lowercasing, stripping punctuation, and merging near-duplicates via Jaccard or embedding similarity.

Parsing Techniques by Platform

  • Google Trends: Parse the widgetdata JSON; strip the leading ")]}',\n" artifact before json.loads(). Use interest_over_time DataFrame → isPartial bool; related_queries dict of top/rising DataFrames.
  • Twitter / X (twscrape): Returns structured Trend models — extract name, query, url, tweet_volume (may be null), and relative rank.
  • YouTube (Invidious): Parse the /api/v1/trending JSON array directly. Each item has title, videoId, author, published, viewCount, lengthSeconds, description.
  • Reddit: Parse data.children[].data from the .json feed. Key fields: title, subreddit, author, score, upvote_ratio, num_comments, url, created_utc, permalink.
  • TikTok (Apify / scraper): Parse JSON lines from Apify dataset or local HTML/JS state (#SIGI_STATE). Key fields: hashtag, video_count, creator, music_title, view_count, like_count.

Category Platforms of Origin Typical Signals
Entertainment YouTube, TikTok, Twitter/X Music releases, movie trailers, celebrity moments, viral dances
Technology Reddit (r/technology, r/gadgets), Twitter/X, YouTube Product launches, AI announcements, teardowns, security news
Politics Twitter/X, Reddit (r/politics, r/worldnews), Google Trends Election polls, debates, policy announcements, protests
Memes / Internet Culture TikTok, Twitter/X, Reddit (r/memes, r/dankmemes) Image macros, audio trends, ironic formats, copypastas
Sports Twitter/X, YouTube, Reddit (r/sports) Match results, athlete transfers, highlights
Gaming YouTube Gaming, Reddit (r/gaming, r/pcgaming), Twitch Releases, speedruns, esports finals, patch notes
Finance Reddit (r/wallstreetbets), Twitter/X, Google Trends Stock/crypto movements, earnings, macro events

5. Rate Limits, Anti-Bot Measures, and API Key Requirements

5.1 Summary Table

Platform API Key Required Rate Limit Anti-Bot Layers Mitigation Strategy
Google Trends No (unofficial) ~1,400–2,000 req/day/IP IP throttling, 429s 60s backoff on 429; proxy rotation; 5 keywords max per request
Twitter / X No (unofficial) / Yes (official) Hidden / strict on official API TLS fingerprinting, cookie rotation, IP reputation, CAPTCHA Account pools + proxy rotation + active library (twscrape)
YouTube No (Invidious) / Yes (official API v3) 100 units/day free quota (official) Bot detection on direct HTML Use Invidious instances; rotate instances; fallback to yt-dlp
Reddit Yes (OAuth app) 100 QPM (official); 10-min avg window OAuth enforcement, User-Agent blocks OAuth client + custom User-Agent + respect headers
TikTok No (unofficial) Unclear / heavily throttled sigi-state hydration, fingerprinting, CAPTCHA Dedicated scraper (Apify Actor) or Playwright-based tool
NewsAPI Yes 100 req/day (free) API-key auth standard Upgrade or cache aggressively
GNews API Yes 100 req/day (free) API-key auth standard Upgrade or cache aggressively

5.2 Operational Recommendations

  1. Assume every source will break. Abstract each crawler behind a common interface so endpoint changes only affect one module.
  2. Run crawlers asynchronously with bounded concurrency (e.g., asyncio.Semaphore(3)).
  3. Cache aggressively. Trend data has a half-life of minutes to hours; cache feed-level results for 5–15 minutes and aggregate-level results for 1–4 hours.
  4. Rotate proxies / instances. For Google Trends, YouTube Invidious, and Twitter/X, proxy rotation is the difference between a stable pipeline and a banned IP.
  5. Monitor health. Track response codes, latency, and parse-failure rates per source. Disable a source automatically if it fails >30% of requests over 10 minutes.

6. Data Schemas for Trend Entries

Every trend entry should normalize down to the following JSON structure:

{
  "id": "sha256-of-source+title+timestamp",
  "platform": "google_trends | twitter | youtube | reddit | tiktok | newsapi",
  "source_url": "https://...",
  "crawl_time": "2026-04-13T10:00:00Z",
  "published_time": "2026-04-13T09:30:00Z",
  "region": "US",
  "category": "technology",
  "title": "Apple Intelligence",
  "query": "Apple Intelligence",
  "url": "https://trends.google.com/trends/explore?q=Apple+Intelligence",
  "author": "",
  "engagement": {
    "score": 100,
    "score_type": "relative_0_100",
    "raw_views": null,
    "raw_comments": null,
    "raw_shares": null
  },
  "metadata": {
    "rank": 1,
    "is_realtime": false,
    "tags": ["AI", "Apple"]
  }
}

6.2 Platform-Specific Schema Details

  • interest_over_time DataFrame:
    • Index: DatetimeIndex (date)
    • Columns: one int64 per keyword (0–100)
    • Flag: isPartial (bool)
  • related_queries:
    • Dict keys: top, rising
    • Values: DataFrames with query, value (relative relevance)
  • trending_searches:
    • DataFrame with title, formattedTraffic, relatedQueries

Twitter / X (via twscrape)

  • Trend object fields:
    • name (display text)
    • query (search query)
    • url (trend URL)
    • tweet_volume (int or null)
    • rank (implied by list order)

YouTube (via Invidious)

  • Video object fields:
    • title, videoId, author, authorId
    • published (Unix timestamp or ISO)
    • viewCount (int)
    • lengthSeconds (int)
    • videoThumbnails (array of {url, quality, width, height})
    • description (string)

Reddit (via .json feed)

  • children[].data fields:
    • title, subreddit_name_prefixed (r/...)
    • author, score (net upvotes), upvote_ratio
    • num_comments, url, permalink
    • created_utc (Unix timestamp)
    • thumbnail, media, is_video

TikTok (via Apify / scraper)

  • Creative Center trend item fields:
    • hashtag or music_title
    • video_count or posts_count
    • view_count or popularity_score
    • creator (for creator trends)
    • region
    • trend_type (hashtag, sound, video, creator)

7. Data Sources and References

  1. Google Trends / pytrends

    • PyPI: https://pypi.org/project/pytrends/
    • GitHub (archived 2025): https://github.com/GeneralMills/pytrends
    • HU Berlin rate-limit paper (2024): https://edoc.hu-berlin.de/... (working paper estimating ~2,000 queries/day)
    • CRAN R wrapper (2024): https://cran.r-project.org/package=PytrendsLongitudinalR
  2. Twitter / X Scraping

    • vladkens/twscrape: https://github.com/vladkens/twscrape
    • Scrapfly guide: https://scrapfly.io/blog/posts/how-to-scrape-twitter
    • DeepWiki architecture: https://deepwiki.com/vladkens/twscrape
  3. YouTube Unofficial APIs

    • Invidious API docs: https://docs.invidious.io/
    • yt-dlp GitHub: https://github.com/yt-dlp/yt-dlp
    • ScrapeCreators blog: https://scrapecreators.com/blog/unofficial-youtube-api
  4. Reddit API

    • Reddit Data API Wiki: https://support.reddithelp.com/hc/en-us/articles/16160319875092-Reddit-Data-API-Wiki
    • Reddit JSON feed guide: https://rapidevelopers.com/api-automations/how-to-automate-reddit-trend-monitoring-using-the-api
  5. TikTok Scraping

    • Apify TikTok Creative Center Scraper: https://apify.com/doliz/tiktok-creative-center-scraper
    • Russell-Newton/TikTokPy: https://github.com/Russell-Newton/TikTokPy
    • Scrapfly TikTok guide: https://scrapfly.io/blog/posts/how-to-scrape-tiktok-python-json
  6. BuzzSumo Alternatives / Content APIs

    • NewsAPI: https://newsapi.org/
    • GNews API: https://gnews.io/
    • Exploding Topics alternatives article: https://explodingtopics.com/blog/buzzsumo-alternatives

8. Quick-Reference Cheat Sheet

Goal Best Free Tool/API Fallback
Validate search demand pytrends (Google Trends) SERP scraping
Real-time conversation trends twscrape (Twitter/X) Apify X Trends Scraper
Video viral leaderboards Invidous /api/v1/trending yt-dlp
Grassroots early signals Reddit OAuth JSON feeds apidirect.io
Short-form culture trends Apify TikTok Creative Center TikTokPy (Playwright)
Global news headlines NewsAPI GNews API
Cross-platform share counts Exploding Topics (free tier) Marketing Miner (free tier)

End of knowhow_v1.md