bulk-summarize-async.md 2.59 KB

Bulk Async Article Summarization

When you need to summarize many articles from the TGO pipeline (or any group) and do not want to wait for each one sequentially, submit them all in parallel against POST /api/articles/summarize and collect the async task IDs. The endpoint is single-article and asynchronous, so it returns immediately for each URL.

Endpoint contract

POST http://127.0.0.1:5001/api/articles/summarize
Content-Type: application/json

{"url": "https://mp.weixin.qq.com/s?...", "title": "..."}

Responses:

  • {"ok": true, "data": {"status": "cached", "content": "...", "report_id": N}} — already summarized
  • {"ok": true, "data": {"status": "started", "task_id": "..."}} — Hermes job spawned; poll /api/job/<task_id> if you want to wait

Common mistakes

  • ❌ Calling POST /api/articles/summarize without Content-Type: application/json → 415 Unsupported Media Type.
  • ❌ Calling it with an empty body or without url → 400 Bad Request or {"ok": false, "error": "需要 url"}.
  • ❌ Expecting a batch API — there is none; the endpoint is one article per call.
  • ❌ Using python3 -c in a Hermes session to script the loop — this may be blocked by the script execution via -e/-c flag security policy. Use bash + jq or write a .py/.sh file and run it via terminal.

Quick one-off bulk submit (bash + jq)

API=http://127.0.0.1:5001
OUT=$HOME/.tmp/bulk_summaries
mkdir -p "$OUT"

# 1. Pull the unsummarized list
curl -s "$API/api/articles/unsummarized" > "$OUT/unsummarized.json"

# 2. Submit each URL in parallel; write response to a task file
while IFS=$'\t' read -r id url; do
  payload=$(jq -n --arg url "$url" '{url:$url}')
  curl -s -X POST -H "Content-Type: application/json" \
       -d "$payload" "$API/api/articles/summarize" \
       > "$OUT/task_${id}.json" &
done < <(jq -r '.data[] | [.id, .url] | @tsv' "$OUT/unsummarized.json")

wait

# 3. Inspect results
jq -s 'map(.data) | {cached: map(select(.status=="cached")) | length, started: map(select(.status=="started")) | length}' "$OUT"/task_*.json

Concurrency guard

The underlying /api/articles/summarize spawns a Hermes CLI subprocess per article. Do not fire 50+ requests simultaneously — it overloads the device and can trigger WeChat CAPTCHA / web-scraping limits. Keep parallel submits to 2–4 at a time, or use the existing scan_articles.py (sequential, zero concurrency) for cron runs.

Files from this session

  • /data/data/com.termux/files/home/tgo_summarize_batch.sh — full working script that extracts, fetches unsummarized, and bulk-submits summaries in parallel.