---
name: youtube-trending-crawler
description: "Fetch YouTube trending videos using YouTube Data API v3. Get popular videos by region and category with complete metadata. Use when users want to (1) Get trending videos for a region, (2) Fetch popular videos by category, (3) Batch crawl multiple regions, (4) Monitor trending content across markets. Requires YouTube Data API v3 key."
---

# YouTube Trending Crawler

Fetch YouTube trending/popular videos using YouTube Data API v3.

## Setup

1. **API Key**: Get from [Google Cloud Console](https://console.cloud.google.com/apis/credentials) - enable YouTube Data API v3
2. **Regions**: ISO 3166-1 alpha-2 codes (US, JP, KR, GB, DE, FR, IN, BR, etc.)
3. **Categories**: Video category IDs (1=Film, 2=Autos, 10=Music, 17=Sports, 20=Gaming, 24=Entertainment, 28=Tech, etc.)

## Core API

### Primary: videos endpoint (1 quota unit)

```bash
# Most Popular Videos (CHART endpoint - 1 unit)
curl "https://www.googleapis.com/youtube/v3/videos?part=snippet,statistics,contentDetails&chart=mostPopular&regionCode=US&videoCategoryId=10&maxResults=10&key=API_KEY"

# Batch video IDs (max 50 per call)
curl "https://www.googleapis.com/youtube/v3/videos?part=snippet,statistics,contentDetails&id=VIDEO_IDS&key=API_KEY"
```

### Fallback: yt-dlp (when API fails)

```bash
# Get trending page
yt-dlp "https://www.youtube.com/results?sp=CAA%253D&q=trending" --flat-playlist --print "%(id)s|%(title)s|%(channel_id)s|%(channel)s|%(view_count)s|%(like_count)s|%(comment_count)s|%(upload_date)s|%(description)s|%(thumbnail)s|%(categories)s|%(duration)s"

# Get specific video details
yt-dlp "https://youtube.com/watch?v=VIDEO_ID" --print "%(id)s|%(title)s|%(channel_id)s|%(channel)s|%(view_count)s|%(like_count)s|%(comment_count)s|%(upload_date)s|%(description)s|%(thumbnail)s|%(categories)s|%(duration)s|%(tags)s"
```

## Parameters

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| regionCode | string | No | ISO 3166-1 alpha-2 (default: US) |
| videoCategoryId | int | No | 1-44 category ID |
| maxResults | int | No | 1-50 (default: 10) |
| pageToken | string | No | Pagination token |

## Supported Regions

```javascript
const SUPPORTED_REGIONS = [
  'US', 'JP', 'KR', 'GB', 'DE', 'FR', 'IN', 'BR',  // Primary
  'CA', 'AU', 'RU', 'MX', 'ES', 'IT', 'NL', 'SE'  // Extended
];
```

## Supported Categories

```javascript
const CATEGORIES = {
  1: 'Film & Animation',
  2: 'Autos & Vehicles',
  10: 'Music',
  15: 'Pets & Animals',
  17: 'Sports',
  20: 'Gaming',
  22: 'People & Blogs',
  23: 'Comedy',
  24: 'Entertainment',
  25: 'News & Politics',
  27: 'Education',
  28: 'Science & Technology'
};
```

## Output Fields (17 fields)

| Field | Source | Type |
|-------|--------|------|
| videoId | id | string |
| title | snippet.title | string |
| channelTitle | snippet.channelTitle | string |
| channelId | snippet.channelId | string |
| subscriberCount | channels.list (separate call) | int |
| viewCount | statistics.viewCount | int |
| likeCount | statistics.likeCount | int |
| commentCount | statistics.commentCount | int |
| publishedAt | snippet.publishedAt | ISO8601 |
| description | snippet.description | string |
| url | https://youtube.com/watch?v={id} | string |
| thumbnail | snippet.thumbnails.default.url | string |
| categoryId | snippet.categoryId | string |
| categoryName | mapped from CATEGORIES | string |
| tags | snippet.tags | array |
| duration | contentDetails.duration (ISO 8601) | string |
| crawlTime |generated | ISO8601 |
| regionCode | from request | string |

## Error Handling

### 403 Quota Exceeded
```javascript
if (error.code === 403 && error.message.includes('quota')) {
  // Fallback to yt-dlp
  return await fetchWithYtDlp(params);
}
```

### 404 Video Not Found
```javascript
if (error.code === 404) {
  console.log(`Video ${videoId} not found, skipping`);
  return null;
}
```

### 500 Server Error
```javascript
// Exponential backoff retry
const maxRetries = 3;
for (let i = 0; i < maxRetries; i++) {
  try {
    return await fetch(...);
  } catch (e) {
    if (e.code === 500 && i < maxRetries - 1) {
      await sleep(Math.pow(2, i) * 1000);  // 1s, 2s, 4s
    }
  }
}
```

### Rate Limit (429)
```javascript
if (error.code === 429) {
  // Wait and retry with longer delay
  await sleep(60000); // 1 minute
  return await fetch(...);
}
```

## Quota Management

- Use `videos?chart=mostPopular` endpoint (1 unit)
- NEVER use `search` endpoint (100 units)
- Batch video IDs (max 50 per call when extending)
- Cache category names locally
- Rate limit: min 1000ms between requests

## Output Format

```json
{
  "videos": [
    {
      "videoId": "dQw4w9WgXcQ",
      "title": "Video Title",
      "channelTitle": "Channel Name",
      "channelId": "UCxxxx",
      "subscriberCount": 1000000,
      "viewCount": 5000000,
      "likeCount": 250000,
      "commentCount": 50000,
      "publishedAt": "2024-01-15T10:00:00Z",
      "description": "Video description...",
      "url": "https://youtube.com/watch?v=dQw4w9WgXcQ",
      "thumbnail": "https://i.ytimg.com/vi/dQw4w9WgXcQ/default.jpg",
      "categoryId": "10",
      "categoryName": "Music",
      "tags": ["tag1", "tag2"],
      "duration": "PT3M45S",
      "crawlTime": "2024-01-15T12:00:00Z",
      "regionCode": "US"
    }
  ],
  "metadata": {
    "regionCode": "US",
    "videoCategoryId": "10",
    "totalResults": 10,
    "crawlTime": "2024-01-15T12:00:00Z"
  }
}
```

## Category Mapping

```javascript
function getCategoryName(categoryId) {
  const map = {
    '1': 'Film & Animation',
    '2': 'Autos & Vehicles',
    '10': 'Music',
    '15': 'Pets & Animals',
    '17': 'Sports',
    '20': 'Gaming',
    '22': 'People & Blogs',
    '23': 'Comedy',
    '24': 'Entertainment',
    '25': 'News & Politics',
    '27': 'Education',
    '28': 'Science & Technology'
  };
  return map[categoryId] || 'Unknown';
}
```