bulk-summarize-async.md
2.59 KB
Bulk Async Article Summarization
When you need to summarize many articles from the TGO pipeline (or any group) and do not want to wait for each one sequentially, submit them all in parallel against POST /api/articles/summarize and collect the async task IDs. The endpoint is single-article and asynchronous, so it returns immediately for each URL.
Endpoint contract
POST http://127.0.0.1:5001/api/articles/summarize
Content-Type: application/json
{"url": "https://mp.weixin.qq.com/s?...", "title": "..."}
Responses:
{"ok": true, "data": {"status": "cached", "content": "...", "report_id": N}}— already summarized{"ok": true, "data": {"status": "started", "task_id": "..."}}— Hermes job spawned; poll/api/job/<task_id>if you want to wait
Common mistakes
- ❌ Calling
POST /api/articles/summarizewithoutContent-Type: application/json→415 Unsupported Media Type. - ❌ Calling it with an empty body or without
url→400 Bad Requestor{"ok": false, "error": "需要 url"}. - ❌ Expecting a batch API — there is none; the endpoint is one article per call.
- ❌ Using
python3 -cin a Hermes session to script the loop — this may be blocked by thescript execution via -e/-c flagsecurity policy. Usebash + jqor write a.py/.shfile and run it viaterminal.
Quick one-off bulk submit (bash + jq)
API=http://127.0.0.1:5001
OUT=$HOME/.tmp/bulk_summaries
mkdir -p "$OUT"
# 1. Pull the unsummarized list
curl -s "$API/api/articles/unsummarized" > "$OUT/unsummarized.json"
# 2. Submit each URL in parallel; write response to a task file
while IFS=$'\t' read -r id url; do
payload=$(jq -n --arg url "$url" '{url:$url}')
curl -s -X POST -H "Content-Type: application/json" \
-d "$payload" "$API/api/articles/summarize" \
> "$OUT/task_${id}.json" &
done < <(jq -r '.data[] | [.id, .url] | @tsv' "$OUT/unsummarized.json")
wait
# 3. Inspect results
jq -s 'map(.data) | {cached: map(select(.status=="cached")) | length, started: map(select(.status=="started")) | length}' "$OUT"/task_*.json
Concurrency guard
The underlying /api/articles/summarize spawns a Hermes CLI subprocess per article. Do not fire 50+ requests simultaneously — it overloads the device and can trigger WeChat CAPTCHA / web-scraping limits. Keep parallel submits to 2–4 at a time, or use the existing scan_articles.py (sequential, zero concurrency) for cron runs.
Files from this session
/data/data/com.termux/files/home/tgo_summarize_batch.sh— full working script that extracts, fetches unsummarized, and bulk-submits summaries in parallel.