宜博 / lemonboss

knowhow_raw.md 6.26 KB

Weibo Hot Search / Trending Topics Data Source Research Report

1. Executive Summary

This report evaluates data sources for scraping Weibo hot search/trending topics (微博热搜). The primary target is the real-time hot search list on Weibo, which reflects trending topics in China. We evaluate official APIs, web endpoints, and third-party aggregators.

2. Data Sources (Detailed Evaluation)

Source 1: Weibo Web Hot Search Board (Official Web Interface)

  • URL: https://s.weibo.com/top/summary
  • Type: Web Scraping (HTML)
  • Authority: High (Official Sina Weibo)
  • Update Frequency: Real-time (every 1-3 minutes during peak hours)
  • Data Quality: High. Provides rank, keyword, category tag, and heat score (热度).
  • Crawling Difficulty: Medium. HTML table structure is stable but requires a valid User-Agent and cookie/session management for anti-bot. The page returns a static HTML table with ID #pl_top_realtimehot.
  • Robots: No explicit block for this path in weibo.com/robots.txt, but aggressive crawling will trigger anti-bot measures.
  • Fields Available: rank, keyword, heat_score, tag, source_url

Source 2: Weibo Mobile Hot Search API (m.weibo.cn)

  • URL: https://m.weibo.cn/api/container/getIndex?containerid=106003type%3D25%26t%3D3%26disable_hot%3D1%26filter_type%3Drealtimehot
  • Type: Unofficial/Mobile API (JSON)
  • Authority: High (Sina Official Mobile Endpoint)
  • Update Frequency: Real-time
  • Data Quality: High. Returns structured JSON with desc, desc_extr (heat), pic.
  • Crawling Difficulty: Medium to High. Requires valid User-Agent (mobile), Referer, and sometimes anti-bot tokens. The API is designed for the mobile app/web and is rate-limited.
  • Robots: m.weibo.cn/robots.txt disallows many paths, but /api/container/getIndex is a core endpoint used by the official app.
  • Fields Available: keyword, heat_score, rank, category, pic_url

Source 3: Weibo Hot Search API via weibo.com/ajax

  • URL: https://weibo.com/ajax/side/hotSearch
  • Type: Unofficial/AJAX API (JSON)
  • Authority: High (Official Weibo Frontend)
  • Update Frequency: Real-time
  • Data Quality: Very High. Returns a clean JSON array of hot search items with rich metadata: word (keyword), raw_hot (raw heat), category, rank, subject_label, icon_desc, topic_flag.
  • Crawling Difficulty: Medium. Requires authenticated cookies (SUB, SUBP, XSRF-TOKEN). Without a login session, it returns 403 or empty data. Session rotation is recommended for large-scale crawling.
  • Robots: weibo.com/robots.txt is permissive for general crawlers, but aggressive access to /ajax/ endpoints may trigger IP bans.
  • Fields Available: rank, word, raw_hot, category, subject_label, icon_desc, topic_flag, source_url

Source 4: Third-Party Aggregator: 微博热搜 (wrd.cn / unofficial mirrors)

  • URL: https://weibo.zhaunkunqiu.com/ (or similar community mirrors)
  • Type: Third-Party Aggregator (HTML/JSON)
  • Authority: Low to Medium (Community-maintained)
  • Update Frequency: Every 5-15 minutes (depends on the mirror)
  • Data Quality: Medium. Data is often a mirror of official sources, but may have latency, missing fields, or be stale.
  • Crawling Difficulty: Low. These sites are typically more permissive and built for scraping.
  • Robots: Usually permissive.
  • Fields Available: rank, keyword, heat_score

Source 5: Weibo Realtime Hot Search RSS / Feed Providers

  • URL: Various (e.g., https://rsshub.app/weibo/search/hot if available)
  • Type: RSS / Third-Party Feed
  • Authority: Low (RSSHub is a community project)
  • Update Frequency: Every 10-30 minutes (limited by RSS polling)
  • Data Quality: Medium. Good for simple keyword lists, but lacks heat scores and rich metadata.
  • Crawling Difficulty: Low. Standard RSS polling.
  • Robots: Varies by provider.
  • Fields Available: title, link, pubDate

Source 6: Weibo New Media Hot Board (官方新媒体热搜榜)

  • URL: https://s.weibo.com/top/summary?cate=realtimehot
  • Type: Web Scraping (HTML)
  • Authority: High (Official)
  • Update Frequency: Real-time
  • Data Quality: High. Same data as Source 1, but explicitly filtered for realtime hot (实时热搜).
  • Crawling Difficulty: Medium. Same anti-bot measures as Source 1.
  • Robots: General crawler rules apply.
  • Fields Available: rank, keyword, heat_score, tag, link

Source 7: GitHub Open Source Data Snapshots

  • URL: https://raw.githubusercontent.com/Various/Weibo-Hot-Search/main/...
  • Type: Static Archive / GitHub Data Dump
  • Authority: Low (Community)
  • Update Frequency: Hourly to Daily
  • Data Quality: Medium. Data is stale but historical; useful for seed data.
  • Crawling Difficulty: Very Low. Static files on GitHub.
  • Robots: raw.githubusercontent.com is generally accessible.
  • Fields Available: time, rank, keyword, heat

3. Comparative Matrix

Source Authority Freshness Difficulty Risk Recommendation
s.weibo.com (HTML) High Very High Medium Medium Primary
m.weibo.cn API High Very High Medium-High Medium Secondary
weibo.com/ajax High Very High Medium High Primary (w/ auth)
Third-Party Mirrors Low Medium Low Low Fallback
RSS Feeds Low Low Low Low Supplementary
GitHub Snapshots Low Low Very Low Very Low Seed/Historical
  1. Primary: Use https://s.weibo.com/top/summary (HTML scraping) with respectful headers and 3-second intervals.
  2. Secondary: Use https://weibo.com/ajax/side/hotSearch if authenticated session cookies are available.
  3. Fallback: Use https://m.weibo.cn/api/container/getIndex with mobile headers.
  4. Seed/Historical: Use GitHub open-source snapshots for initial seed data and testing.
  5. Rate Limiting: Strictly enforce a 3-second minimum delay between requests to any Weibo domain.
  6. Compliance: Cache robots.txt, respect Crawl-delay, and do not crawl during peak hours (12:00-14:00, 20:00-23:00) if possible to reduce server load.