# Weibo Hot Search / Trending Topics Data Source Research Report

## 1. Executive Summary

This report evaluates data sources for scraping Weibo hot search/trending topics (微博热搜). The primary target is the real-time hot search list on Weibo, which reflects trending topics in China. We evaluate official APIs, web endpoints, and third-party aggregators.

## 2. Data Sources (Detailed Evaluation)

---

### Source 1: Weibo Web Hot Search Board (Official Web Interface)
- **URL**: `https://s.weibo.com/top/summary`
- **Type**: Web Scraping (HTML)
- **Authority**: High (Official Sina Weibo)
- **Update Frequency**: Real-time (every 1-3 minutes during peak hours)
- **Data Quality**: High. Provides rank, keyword, category tag, and heat score (热度).
- **Crawling Difficulty**: Medium. HTML table structure is stable but requires a valid `User-Agent` and cookie/session management for anti-bot. The page returns a static HTML table with ID `#pl_top_realtimehot`.
- **Robots**: No explicit block for this path in `weibo.com/robots.txt`, but aggressive crawling will trigger anti-bot measures.
- **Fields Available**: `rank`, `keyword`, `heat_score`, `tag`, `source_url`

---

### Source 2: Weibo Mobile Hot Search API (m.weibo.cn)
- **URL**: `https://m.weibo.cn/api/container/getIndex?containerid=106003type%3D25%26t%3D3%26disable_hot%3D1%26filter_type%3Drealtimehot`
- **Type**: Unofficial/Mobile API (JSON)
- **Authority**: High (Sina Official Mobile Endpoint)
- **Update Frequency**: Real-time
- **Data Quality**: High. Returns structured JSON with `desc`, `desc_extr` (heat), `pic`.
- **Crawling Difficulty**: Medium to High. Requires valid `User-Agent` (mobile), `Referer`, and sometimes anti-bot tokens. The API is designed for the mobile app/web and is rate-limited.
- **Robots**: `m.weibo.cn/robots.txt` disallows many paths, but `/api/container/getIndex` is a core endpoint used by the official app.
- **Fields Available**: `keyword`, `heat_score`, `rank`, `category`, `pic_url`

---

### Source 3: Weibo Hot Search API via weibo.com/ajax
- **URL**: `https://weibo.com/ajax/side/hotSearch`
- **Type**: Unofficial/AJAX API (JSON)
- **Authority**: High (Official Weibo Frontend)
- **Update Frequency**: Real-time
- **Data Quality**: Very High. Returns a clean JSON array of hot search items with rich metadata: `word` (keyword), `raw_hot` (raw heat), `category`, `rank`, `subject_label`, `icon_desc`, `topic_flag`.
- **Crawling Difficulty**: Medium. Requires authenticated cookies (`SUB`, `SUBP`, `XSRF-TOKEN`). Without a login session, it returns 403 or empty data. Session rotation is recommended for large-scale crawling.
- **Robots**: `weibo.com/robots.txt` is permissive for general crawlers, but aggressive access to `/ajax/` endpoints may trigger IP bans.
- **Fields Available**: `rank`, `word`, `raw_hot`, `category`, `subject_label`, `icon_desc`, `topic_flag`, `source_url`

---

### Source 4: Third-Party Aggregator: 微博热搜 (wrd.cn / unofficial mirrors)
- **URL**: `https://weibo.zhaunkunqiu.com/` (or similar community mirrors)
- **Type**: Third-Party Aggregator (HTML/JSON)
- **Authority**: Low to Medium (Community-maintained)
- **Update Frequency**: Every 5-15 minutes (depends on the mirror)
- **Data Quality**: Medium. Data is often a mirror of official sources, but may have latency, missing fields, or be stale.
- **Crawling Difficulty**: Low. These sites are typically more permissive and built for scraping.
- **Robots**: Usually permissive.
- **Fields Available**: `rank`, `keyword`, `heat_score`

---

### Source 5: Weibo Realtime Hot Search RSS / Feed Providers
- **URL**: Various (e.g., `https://rsshub.app/weibo/search/hot` if available)
- **Type**: RSS / Third-Party Feed
- **Authority**: Low (RSSHub is a community project)
- **Update Frequency**: Every 10-30 minutes (limited by RSS polling)
- **Data Quality**: Medium. Good for simple keyword lists, but lacks heat scores and rich metadata.
- **Crawling Difficulty**: Low. Standard RSS polling.
- **Robots**: Varies by provider.
- **Fields Available**: `title`, `link`, `pubDate`

---

### Source 6: Weibo New Media Hot Board (官方新媒体热搜榜)
- **URL**: `https://s.weibo.com/top/summary?cate=realtimehot`
- **Type**: Web Scraping (HTML)
- **Authority**: High (Official)
- **Update Frequency**: Real-time
- **Data Quality**: High. Same data as Source 1, but explicitly filtered for realtime hot (实时热搜).
- **Crawling Difficulty**: Medium. Same anti-bot measures as Source 1.
- **Robots**: General crawler rules apply.
- **Fields Available**: `rank`, `keyword`, `heat_score`, `tag`, `link`

---

### Source 7: GitHub Open Source Data Snapshots
- **URL**: `https://raw.githubusercontent.com/Various/Weibo-Hot-Search/main/...`
- **Type**: Static Archive / GitHub Data Dump
- **Authority**: Low (Community)
- **Update Frequency**: Hourly to Daily
- **Data Quality**: Medium. Data is stale but historical; useful for seed data.
- **Crawling Difficulty**: Very Low. Static files on GitHub.
- **Robots**: `raw.githubusercontent.com` is generally accessible.
- **Fields Available**: `time`, `rank`, `keyword`, `heat`

## 3. Comparative Matrix

| Source | Authority | Freshness | Difficulty | Risk | Recommendation |
|--------|-----------|-----------|------------|------|----------------|
| s.weibo.com (HTML) | High | Very High | Medium | Medium | **Primary** |
| m.weibo.cn API | High | Very High | Medium-High | Medium | **Secondary** |
| weibo.com/ajax | High | Very High | Medium | High | **Primary (w/ auth)** |
| Third-Party Mirrors | Low | Medium | Low | Low | Fallback |
| RSS Feeds | Low | Low | Low | Low | Supplementary |
| GitHub Snapshots | Low | Low | Very Low | Very Low | Seed/Historical |

## 4. Recommended Strategy

1. **Primary**: Use `https://s.weibo.com/top/summary` (HTML scraping) with respectful headers and 3-second intervals.
2. **Secondary**: Use `https://weibo.com/ajax/side/hotSearch` if authenticated session cookies are available.
3. **Fallback**: Use `https://m.weibo.cn/api/container/getIndex` with mobile headers.
4. **Seed/Historical**: Use GitHub open-source snapshots for initial seed data and testing.
5. **Rate Limiting**: Strictly enforce a 3-second minimum delay between requests to any Weibo domain.
6. **Compliance**: Cache `robots.txt`, respect `Crawl-delay`, and do not crawl during peak hours (12:00-14:00, 20:00-23:00) if possible to reduce server load.