youtube-trending-crawler.md 6.29 KB

Edit Raw Blame History Permalink



name: youtube-trending-crawler
description: "Fetch YouTube trending videos using YouTube Data API v3. Get popular videos by region and category with complete metadata. Use when users want to (1) Get trending videos for a region, (2) Fetch popular videos by category, (3) Batch crawl multiple regions, (4) Monitor trending content across markets. Requires YouTube Data API v3 key."
YouTube Trending Crawler
Fetch YouTube trending/popular videos using YouTube Data API v3.
Setup

API Key: Get from Google Cloud Console - enable YouTube Data API v3
Regions: ISO 3166-1 alpha-2 codes (US, JP, KR, GB, DE, FR, IN, BR, etc.)
Categories: Video category IDs (1=Film, 2=Autos, 10=Music, 17=Sports, 20=Gaming, 24=Entertainment, 28=Tech, etc.)

Core API
Primary: videos endpoint (1 quota unit)
# Most Popular Videos (CHART endpoint - 1 unit)
curl "https://www.googleapis.com/youtube/v3/videos?part=snippet,statistics,contentDetails&chart=mostPopular&regionCode=US&videoCategoryId=10&maxResults=10&key=API_KEY"

# Batch video IDs (max 50 per call)
curl "https://www.googleapis.com/youtube/v3/videos?part=snippet,statistics,contentDetails&id=VIDEO_IDS&key=API_KEY"

Fallback: yt-dlp (when API fails)
# Get trending page
yt-dlp "https://www.youtube.com/results?sp=CAA%253D&q=trending" --flat-playlist --print "%(id)s|%(title)s|%(channel_id)s|%(channel)s|%(view_count)s|%(like_count)s|%(comment_count)s|%(upload_date)s|%(description)s|%(thumbnail)s|%(categories)s|%(duration)s"

# Get specific video details
yt-dlp "https://youtube.com/watch?v=VIDEO_ID" --print "%(id)s|%(title)s|%(channel_id)s|%(channel)s|%(view_count)s|%(like_count)s|%(comment_count)s|%(upload_date)s|%(description)s|%(thumbnail)s|%(categories)s|%(duration)s|%(tags)s"

Parameters


Parameter
Type
Required
Description


regionCode
string
No
ISO 3166-1 alpha-2 (default: US)


videoCategoryId
int
No
1-44 category ID


maxResults
int
No
1-50 (default: 10)


pageToken
string
No
Pagination token


Supported Regions
const SUPPORTED_REGIONS = [
  'US', 'JP', 'KR', 'GB', 'DE', 'FR', 'IN', 'BR',  // Primary
  'CA', 'AU', 'RU', 'MX', 'ES', 'IT', 'NL', 'SE'  // Extended
];

Supported Categories
const CATEGORIES = {
  1: 'Film & Animation',
  2: 'Autos & Vehicles',
  10: 'Music',
  15: 'Pets & Animals',
  17: 'Sports',
  18: 'Short Movies',
  19: 'Travel & Events',
  20: 'Gaming',
  21: 'Videoblogging',
  22: 'People & Blogs',
  23: 'Comedy',
  24: 'Entertainment',
  25: 'News & Politics',
  26: 'Howto & Style',
  27: 'Education',
  28: 'Science & Technology',
  29: 'Nonprofits',
  30: 'Movies',
  43: 'Shows',
  44: 'Trailers'
};

Output Fields (17 fields)


Field
Source
Type


videoId
id
string


title
snippet.title
string


channelTitle
snippet.channelTitle
string


channelId
snippet.channelId
string


subscriberCount
channels.list (separate call)
int


viewCount
statistics.viewCount
int


likeCount
statistics.likeCount
int


commentCount
statistics.commentCount
int


publishedAt
snippet.publishedAt
ISO8601


description
snippet.description
string


url
https://youtube.com/watch?v={id}
string


thumbnail
snippet.thumbnails.default.url
string


categoryId
snippet.categoryId
string


categoryName
mapped from CATEGORIES
string


tags
snippet.tags
array


duration
contentDetails.duration (ISO 8601)
string


crawlTime
generated
ISO8601


regionCode
from request
string


Error Handling
403 Quota Exceeded
if (error.code === 403 && error.message.includes('quota')) {
  // Fallback to yt-dlp
  return await fetchWithYtDlp(params);
}

404 Video Not Found
if (error.code === 404) {
  console.log(`Video ${videoId} not found, skipping`);
  return null;
}

500 Server Error
// Exponential backoff retry
const maxRetries = 3;
for (let i = 0; i < maxRetries; i++) {
  try {
    return await fetch(...);
  } catch (e) {
    if (e.code === 500 && i < maxRetries - 1) {
      await sleep(Math.pow(2, i) * 1000);  // 1s, 2s, 4s
    }
  }
}

Rate Limit (429)
if (error.code === 429) {
  // Wait and retry with longer delay
  await sleep(60000); // 1 minute
  return await fetch(...);
}

Quota Management

Use videos?chart=mostPopular endpoint (1 unit)
NEVER use search endpoint (100 units)
Batch video IDs (max 50 per call when extending)
Cache category names locally
Rate limit: min 1000ms between requests

Output Format
{
  "videos": [
    {
      "videoId": "dQw4w9WgXcQ",
      "title": "Video Title",
      "channelTitle": "Channel Name",
      "channelId": "UCxxxx",
      "subscriberCount": 1000000,
      "viewCount": 5000000,
      "likeCount": 250000,
      "commentCount": 50000,
      "publishedAt": "2024-01-15T10:00:00Z",
      "description": "Video description...",
      "url": "https://youtube.com/watch?v=dQw4w9WgXcQ",
      "thumbnail": "https://i.ytimg.com/vi/dQw4w9WgXcQ/default.jpg",
      "categoryId": "10",
      "categoryName": "Music",
      "tags": ["tag1", "tag2"],
      "duration": "PT3M45S",
      "crawlTime": "2024-01-15T12:00:00Z",
      "regionCode": "US"
    }
  ],
  "metadata": {
    "regionCode": "US",
    "videoCategoryId": "10",
    "totalResults": 10,
    "crawlTime": "2024-01-15T12:00:00Z"
  }
}

Example Usage
# Single region
claude --print 'Get US trending videos' \
  -i '{"regionCode": "US", "maxResults": 10}'

# Multi-region
claude --print 'Get JP and KR trending' \
  -i '{"regions": ["JP", "KR"], "maxResults": 10}'

# Category filter
claude --print 'Get US gaming videos' \
  -i '{"regionCode": "US", "videoCategoryId": 20, "maxResults": 25}'

Category Mapping
function getCategoryName(categoryId) {
  const map = {
    '1': 'Film & Animation',
    '2': 'Autos & Vehicles',
    '10': 'Music',
    '15': 'Pets & Animals',
    '17': 'Sports',
    '20': 'Gaming',
    '22': 'People & Blogs',
    '23': 'Comedy',
    '24': 'Entertainment',
    '25': 'News & Politics',
    '27': 'Education',
    '28': 'Science & Technology'
  };
  return map[categoryId] || 'Unknown';
}