diff --git a/CLAUDE.md b/CLAUDE.md index b60757b..2f176c4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -9,6 +9,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co | 子项目 | 路径 | 形态 | 状态 | |---|---|---|---| | **lifelog-recorder** | `lifelog-recorder/` | Pixel 原生 Android 应用（Kotlin + Compose） | **主线 v0.8** | +| **phoneuse** | `phoneuse/` | Hermes Agent 的 Android MCP 工具层（Python/Termux + Kotlin APK 双实现） | 并行实验，多数工具未验证 | | **wechat_plain** | `wechat_plain/` | Pixel 上微信 bot 的 Xposed/A11y 模块（端上） | 辅助 | | **lemonPCRM** | `server/` + `frontend/` + `packages/` | Node.js + Express 后端 + 静态前端 | **已废弃** | @@ -34,6 +35,44 @@ Pixel 上微信发送/跟踪的落地，纯设备内运行（不依赖 Mac）： 详见 `wechat_plain/` 内文档。 +## phoneuse（Hermes Agent 的 Android MCP 工具层，并行实验） + +> 完整架构见 `phoneuse/ARCHITECTURE.md`，Sprint 任务 + 真机审计记录见 `phoneuse/TASKS.md`，产品 vision 原始稿见 `phoneuse/phoneuse.md`。本节只给定位 + 别踩的坑，细节进文件看。 + +### 定位 + +把 Pixel 上**所有 App** 虚拟化成 **Hermes Agent**（独立 LLM agent runtime，跑在设备 Termux 里）可调用的数据源/能力源：App Runtime Access → Semantic Extraction → Memory Graph → Agent Planning，目标是替代"截图 + OCR + 点击"那种第一代 phone use。跟 lifelog-recorder 是**不同的项目脉络**，2026-06-23 才拍平合并进本仓库，目前两边没有打通。 + +> [!IMPORTANT] 不是零 Mac +> lifelog-recorder 的"完全本地零云端"约束**不适用于 phoneuse 的 Frida 层**：`frida_bridge.py` 跑在 Mac 上，设备端 MCP 工具通过 `adb reverse tcp:8767 tcp:8767` 调用 Mac 上的 frida Python 库去 attach 设备进程。判断这部分能力时不要套用 lifelog-recorder 那套"零网络"假设。 + +### 架构（两套并行实现，未合并） + +- **实现 A（Python/Termux，主链路已跑通）**：`notification_monitor.py`(root dumpsys 轮询) → `event_server.py`(FastAPI+SQLite 队列，:8765) → `android_mcp.py`(FastMCP stdio，Hermes 直连)，独立工具模块各自 `register_xxx_tools(mcp)`：`ui_control_tools`/`device_tools`/`telephony_tools`/`ocr_tools`/`frida_inspect_tools`。 +- **实现 B（Kotlin APK，大量 stub）**：`AgentAccessibilityService`/`AgentNotificationListenerService` 实时事件 → Unix socket → MCP HTTP/SSE(:8080)；`frida-bridge` 模块是 Frida REST client 骨架。 + +### MCP 工具现状：不要假设"写了就能用" + +`android_mcp.py` 注册 ~60 个工具。`TASKS.md` 2026-06-16 真机逐个审计结论：**真可用 11 / 降级 1 / 失效 23**，且明确写了"之前版本所有任务全标 `[x]` 是错的"。典型坑：`EVENT_BUS_DIR` 未定义连带 connector/persona/cloud 11 个工具 `NameError`；`_run_sqlcipher` 不支持参数化导致部分微信查询工具直接 `datatype mismatch`。改这块前先读 `TASKS.md` 的审计记录，别被勾选状态骗。 + +### Frida 深度运行时层（Layer 5，"特种部队"，只打高价值 App） + +通用能力（**不需要逆向任何具体 App**）： +- `frida_call(package, script)` — 注入任意调用方自写的 JS。 +- `inspect_foreground_intent(package)` — 枚举目标进程里所有存活 Activity，按 `hasWindowFocus()` 识别前台的那个，dump 它的 Intent（action/data/component/extras）。 +- `dump_live_bitmaps(package, max_export)` — 枚举目标进程堆里所有存活 Bitmap，导出最大的 N 张为 base64 PNG。 + +字符串解密 / `sendMessage()` 之类具体方法 hook 是**per-app 逆向**，不通用——微信 8.x 重度混淆已经卡死过一次（`TASKS.md` S3-05/S3-06），明确排除在这层通用能力之外。 + +> [!IMPORTANT] `frida_bridge.py` 版本锁定 + 已知匹配 bug +> Mac 端跑 `frida_bridge.py` 必须用 **`frida==17.9.10` + `frida-tools==14.8.0`**（`phoneuse/.venv`，`start_frida_bridge.sh` 一键起 `adb reverse` + 起桥）——更新版本的 `frida-tools` 对 `frida` 有严格版本 range 依赖，配不上设备上跑的 `frida-server 17.9.10` 会直接报错。 +> 现有进程匹配（`req.package in p.name`）按子串匹配 frida 报告的人类可读进程标签，不是包名——微信主进程标签是 `"WeChat"` 不含包名子串，会回退匹配到无 UI 的 `:push` 子进程，导致**静默空结果**（不报错，看起来像目标 App 没有 Activity/Bitmap）。`inspect_foreground_intent`/`dump_live_bitmaps` 已对空结果加诊断提示，根因（`frida_bridge.py` 本身）未修——影响面会牵连 `frida_call` 其它调用方，留作后续单独修。 + +### 已知阻塞 + +- 真机连续性差：开发时常掉 USB，涉及真机的验证要先确认 `adb devices` 能看到设备。 +- 微信相关的 Frida/A11y 经验跟 `wechat_plain` 是分开积累的——`wechat_plain` 已验证过 Frida daemon 注入微信大 bundle 会触发反调试崩溃（见上文 §wechat_plain）；`phoneuse` 走的是 USB attach 而非常驻 daemon，崩溃模式不一定一样，但动手前先看那段记录，别重新踩同一个坑。 + ## lifelog-recorder（Pixel 8 Pro 24/7 录音 + 本地中文 ASR + 本地 LLM，v0.8） > 完整接手地图、踩坑清单、命令速查见 `lifelog-recorder/HANDOFF.md`。本节只勒住整个项目的**架构骨架 + 关键决策来源**，不重复细节。 diff --git a/phoneuse/TASKS.md b/phoneuse/TASKS.md index 7fc6646..a1c3667 100644 --- a/phoneuse/TASKS.md +++ b/phoneuse/TASKS.md @@ -170,6 +170,27 @@ - 修两处真机暴露的缺陷：① 发送前 `_wake_and_unlock()` 唤醒+解非安全锁屏（否则设备息屏时全部 tap 落空——已验证息屏起跑仍 `sent` msgId 4464）； ② `_capture_restore_ime()` 持久化用户真实 IME，防止上一轮中断把 ADBKeyboard 残留为默认输入法后被错误"还原"（修前实测残留过，已修复并回归）。 +## 通用 Frida 深度检视（Intent / Bitmap，2026-06-24） + +> 目标：把"注入目标进程拿完整 Java 对象图"封装成两个**通用**（不需要逆向任何具体 App）MCP 工具， +> 跟 S3-05/S3-06 的 per-app hook 路线区分开。字符串解密能力明确排除（per-app 逆向，超出本次范围）。 + +- [x] **MAC-01** 修通 Mac 端 Frida 链路（此前 S3-03 因 Mac 缺 frida pip 包而完全不可达） + - 工作：`phoneuse/.venv` + `pip install frida==17.9.10 frida-tools==14.8.0 fastapi uvicorn`；新增 `start_frida_bridge.sh`（`adb reverse tcp:8767 tcp:8767` + 起 `frida_bridge.py`） + - 🔬验证 2026-06-24：版本锁定经实测确认——`frida-tools` 最新版对 `frida` 有严格版本范围依赖，与设备上跑的 `frida-server 17.9.10` 不匹配会直接报错；`frida==17.9.10`+`frida-tools==14.8.0` 这组干净安装且匹配。venv 内 `import frida, frida_tools, fastapi, uvicorn` 通过。 + +- [!] **S3-08** 新增 `inspect_foreground_intent(package, timeout)` 工具 + - 工作：新模块 `frida_inspect_tools.py`，`Java.choose('android.app.Activity', ...)` 枚举目标进程内所有存活 Activity，按 `hasWindowFocus()` 识别真正前台的那个，dump `getIntent()` 的 action/dataString/component/extras。 + - 🔬JS 脚本本身的正确性已对一个真实在跑进程验证过（`hasWindowFocus` 准确识别出唯一前台 Activity，`extras` 正确提取出非空字段，Parcelable 类型 extras 按预期落到 Java 默认 `toString()`）。 + - [!] 阻塞：本次会话设备从 `adb devices` 掉线，**MCP 工具本身（经 `deploy.sh` 推送 + 真实 `adb reverse` 隧道 + `android_mcp.py` 注册）的端到端验证未完成**，待设备重新连接后用 `com.android.settings`/Calculator 之类无隐私数据的单进程 App 补测。本地 `_build_server()` 已确认能正常注册此工具（60 个工具总数，无注册期异常）。 + - 已知限制：`frida_bridge.py` 现有的进程名子串匹配对微信这类多进程 App 会匹配错进程（主进程标签是 `"WeChat"` 不含包名子串，回退逻辑会捡到无 UI 的 `:push` 子进程），导致静默空结果——工具已对空结果附加 `note` 诊断提示，但根因留作后续修复（不在本次范围内，怕影响 `frida_call` 其它调用方）。 + +- [!] **S3-09** 新增 `dump_live_bitmaps(package, max_export, timeout)` 工具 + - 工作：同模块，`Java.choose('android.graphics.Bitmap', ...)` 枚举目标进程堆里所有存活 Bitmap，单次 `Java.choose` 内直接持有引用、按 `allocationByteCount` 排序，对最大的 `max_export` 张调 `compress(CompressFormat.PNG.value, 100, baos)` + `Base64.encodeToString(..., NO_WRAP)` 经 `send()` 回传。 + - 🔬JS 脚本本身的正确性已对一个真实在跑进程验证过（导出的 base64 解码后是合法 PNG，magic bytes 正确，尺寸跟 `width`/`height` 字段一致）；同时验证了两处易错点：`CompressFormat.PNG.value`（去掉 `.value` 会报 overload 不匹配）、回收过的 Bitmap 要整段跳过元数据读取（而非只在 `compress()` 处兜底，否则会有垂悬 0 值污染排序）。 + - [!] 阻塞：同 S3-08，设备掉线，MCP 工具端到端验证待补。 + - 已知限制：同 S3-08 的多进程匹配问题；另外单次调用只导出"当下堆里恰好有什么"，不保证是调用者想要的那一张。 + --- # Sprint 4 — Memory Graph（中优先级，3-4 周） diff --git a/phoneuse/android_mcp.py b/phoneuse/android_mcp.py index 601994d..25727da 100644 --- a/phoneuse/android_mcp.py +++ b/phoneuse/android_mcp.py @@ -1572,6 +1572,13 @@ def _build_server(): except Exception as exc: # pragma: no cover - keep server up if module/models missing logger.warning("ocr_tools not registered: %s", exc) + # ── E6: Frida deep-runtime inspection (Intent/Bitmap) — independent module ── + try: + from frida_inspect_tools import register_frida_inspect_tools + register_frida_inspect_tools(mcp) + except Exception as exc: # pragma: no cover - keep server up if module missing + logger.warning("frida_inspect_tools not registered: %s", exc) + # ── E4: Code deploy tool (Mac → Pixel sync) ───────────────────────────────── @mcp.tool() def deploy_phoneuse_code(restart: bool = True) -> str: diff --git a/phoneuse/deploy.sh b/phoneuse/deploy.sh index 67cf851..c43059c 100755 --- a/phoneuse/deploy.sh +++ b/phoneuse/deploy.sh @@ -50,6 +50,7 @@ push_files() { mitm_proxy.py notification_monitor.py frida_bridge.py + frida_inspect_tools.py ) log_info "Pushing ${#files[@]} files to Pixel..." diff --git a/phoneuse/frida_inspect_tools.py b/phoneuse/frida_inspect_tools.py new file mode 100644 index 0000000..0c2f9d1 --- /dev/null +++ b/phoneuse/frida_inspect_tools.py @@ -0,0 +1,241 @@ +""" +Frida deep-runtime inspection MCP tools — generic, no per-app reverse engineering. + +Independent module (E6 / AgentPhone parity). Registered into the existing +FastMCP instance by android_mcp._build_server() via: + + from frida_inspect_tools import register_frida_inspect_tools + register_frida_inspect_tools(mcp) + +This is phoneuse.md's Layer 5 "Deep Runtime" (Frida), scoped deliberately +narrow: only the two capabilities that work on ANY app without first +reverse-engineering it. String-decryption hooking (the third capability +discussed alongside these two) is NOT here — it requires locating a specific +method in a specific app version, which is per-app work (see TASKS.md S3-05/ +S3-06: WeChat's obfuscation already defeated that approach once). + + inspect_foreground_intent — dump the focused Activity's Intent (action/ + data/component/extras) in a target process. + dump_live_bitmaps — enumerate every live Bitmap in a target + process's heap; export the largest N as PNG. + +Both inject a fixed, audited JS payload (not caller-supplied script) into the +target process via the same Mac-side bridge the existing frida_call MCP tool +uses (frida_bridge.py, POST http://127.0.0.1:8767/frida_call). On-device this +reaches the Mac through `adb reverse tcp:8767 tcp:8767` (see +start_frida_bridge.sh); requires frida-server running as root on the device +and frida_bridge.py running on the Mac. + +Known limitation (in frida_bridge.py, not fixed here — out of scope for this +module): its process-name matching does substring search against the +human-readable process label frida reports, not the package name, so +multi-process apps whose main process is labeled differently (e.g. WeChat's +main process shows as "WeChat", not "com.tencent.mm") can resolve to a +sub-process with no UI (e.g. "com.tencent.mm:push") instead of the real +foreground process. Both tools below detect the resulting empty result and +attach a `note` explaining this, instead of silently implying the target app +has no Activities/Bitmaps. +""" + +from __future__ import annotations + +import json +import urllib.request +from typing import Any + +BRIDGE_URL = "http://127.0.0.1:8767/frida_call" + +_INTENT_SCRIPT = """ +Java.perform(function () { + var results = []; + Java.choose('android.app.Activity', { + onMatch: function (instance) { + try { + var info = {}; + info.className = instance.getClass().getName(); + info.hasWindowFocus = instance.hasWindowFocus(); + info.isFinishing = instance.isFinishing(); + var intent = instance.getIntent(); + if (intent !== null) { + info.action = intent.getAction(); + info.dataString = intent.getDataString(); + var comp = intent.getComponent(); + info.component = comp !== null ? comp.toString() : null; + var extrasObj = {}; + var bundle = intent.getExtras(); + if (bundle !== null) { + var iterator = bundle.keySet().iterator(); + while (iterator.hasNext()) { + var key = iterator.next(); + try { + var val = bundle.get(key); + extrasObj[key] = val !== null ? val.toString() : null; + } catch (innerErr) { + extrasObj[key] = ""; + } + } + } + info.extras = extrasObj; + } else { + info.action = null; + } + results.push(info); + } catch (e) { + results.push({ error: e.toString() }); + } + }, + onComplete: function () { + send({ activities: results, count: results.length }); + } + }); +}); +""" + +_BITMAP_SCRIPT = """ +Java.perform(function () { + var MAX_EXPORT = __MAX_EXPORT__; + var CompressFormat = Java.use('android.graphics.Bitmap$CompressFormat'); + var ByteArrayOutputStream = Java.use('java.io.ByteArrayOutputStream'); + var Base64 = Java.use('android.util.Base64'); + var BASE64_NO_WRAP = 2; + var candidates = []; + + Java.choose('android.graphics.Bitmap', { + onMatch: function (bmp) { + try { + if (bmp.isRecycled()) return; + candidates.push({ + bmp: bmp, + width: bmp.getWidth(), + height: bmp.getHeight(), + config: bmp.getConfig() !== null ? bmp.getConfig().toString() : null, + allocationByteCount: bmp.getAllocationByteCount() + }); + } catch (e) {} + }, + onComplete: function () { + candidates.sort(function (a, b) { return b.allocationByteCount - a.allocationByteCount; }); + var top = candidates.slice(0, MAX_EXPORT); + var exported = []; + for (var i = 0; i < top.length; i++) { + var c = top[i]; + var item = { width: c.width, height: c.height, config: c.config, + allocationByteCount: c.allocationByteCount }; + try { + var baos = ByteArrayOutputStream.$new(); + item.compressOk = c.bmp.compress(CompressFormat.PNG.value, 100, baos); + var bytes = baos.toByteArray(); + item.pngByteLength = bytes.length; + item.png_base64 = Base64.encodeToString(bytes, BASE64_NO_WRAP); + } catch (e) { + item.exportError = e.toString(); + } + exported.push(item); + } + send({ totalLiveBitmaps: candidates.length, exportedCount: exported.length, bitmaps: exported }); + } + }); +}); +""" + +_EMPTY_RESULT_NOTE = ( + "Empty result. The target process may be a no-UI helper process rather " + "than the app's main process — frida_bridge.py matches by substring " + "against frida's process label (not the package name), which can pick " + "the wrong process for multi-process apps (e.g. WeChat's main process is " + "labeled 'WeChat', not 'com.tencent.mm', so the substring match falls " + "back to a sub-process like 'com.tencent.mm:push'). This does not " + "necessarily mean the app has no foreground Activity/Bitmaps." +) + + +def _frida_call(package: str, script: str, timeout: int) -> dict[str, Any]: + """POST a JS payload to frida_bridge.py (Mac) via the adb-reverse tunnel.""" + payload = json.dumps({"package": package, "script": script, "timeout": timeout}).encode() + try: + req = urllib.request.Request( + BRIDGE_URL, data=payload, headers={"Content-Type": "application/json"} + ) + with urllib.request.urlopen(req, timeout=timeout + 10) as resp: + return json.loads(resp.read().decode()) + except Exception as e: + return { + "error": f"frida_bridge unreachable: {e}. Ensure start_frida_bridge.sh is " + f"running on the Mac and adb reverse tcp:8767 tcp:8767 is set." + } + + +def register_frida_inspect_tools(mcp) -> None: + """Register the generic Frida deep-runtime inspection tools onto an existing FastMCP.""" + + @mcp.tool() + def inspect_foreground_intent(package: str, timeout: int = 10) -> str: + """ + Inspect the focused Activity's Intent in a running app via Frida. + + Injects a fixed, generic script (no per-app reverse engineering) that + enumerates every live android.app.Activity instance in the target + process and returns each one's class name, hasWindowFocus/isFinishing + state, and Intent (action, data, component, all Bundle extras + stringified). The entry with hasWindowFocus=true is the true + foreground Activity; others are backgrounded-but-alive. + + Requires root + frida-server on device, and frida_bridge.py running + on the Mac (start_frida_bridge.sh) reachable via adb reverse + tcp:8767 tcp:8767. + + Note: complex (Parcelable/Serializable) extras only show Java's + default toString() (e.g. "SomeClass@a1b2c3"), not a deep dump. + Multi-process apps may attach to the wrong sub-process — see this + module's docstring. + + Args: + package: Target app package name, e.g. 'com.android.settings'. + timeout: Max seconds to wait for script output (default 10). + """ + result = _frida_call(package, _INTENT_SCRIPT, timeout) + if "error" in result: + return json.dumps(result, ensure_ascii=False) + output = result.get("output") or [] + activities = output[0].get("activities", []) if output else [] + if not activities: + result["note"] = _EMPTY_RESULT_NOTE + return json.dumps(result, ensure_ascii=False) + + @mcp.tool() + def dump_live_bitmaps(package: str, max_export: int = 3, timeout: int = 15) -> str: + """ + Enumerate live (non-recycled) Bitmap objects in a running app's heap + via Frida, and export the largest few as base64-encoded PNGs. + + Injects a fixed, generic script (no per-app reverse engineering) that + walks the target process's heap for every live android.graphics.Bitmap, + collects width/height/config/allocationByteCount for all of them, then + PNG-compresses and base64-encodes only the `max_export` largest ones + (compression happens inside the target app's own process — keep this + small; it is slow and memory-heavy for large bitmaps). + + Requires root + frida-server on device, and frida_bridge.py running + on the Mac (start_frida_bridge.sh) reachable via adb reverse + tcp:8767 tcp:8767. + + Note: this only captures whatever happens to be allocated in memory + at the moment of the call — not a guarantee of "the photo you + wanted," just a snapshot. Multi-process apps may attach to the wrong + sub-process — see this module's docstring. + + Args: + package: Target app package name, e.g. 'com.android.settings'. + max_export: How many of the largest live bitmaps to export as PNG + (default 3, keep small — see note above). + timeout: Max seconds to wait for script output (default 15). + """ + script = _BITMAP_SCRIPT.replace("__MAX_EXPORT__", str(int(max_export))) + result = _frida_call(package, script, timeout) + if "error" in result: + return json.dumps(result, ensure_ascii=False) + output = result.get("output") or [] + total = output[0].get("totalLiveBitmaps", 0) if output else 0 + if not total: + result["note"] = _EMPTY_RESULT_NOTE + return json.dumps(result, ensure_ascii=False) diff --git a/phoneuse/start_frida_bridge.sh b/phoneuse/start_frida_bridge.sh new file mode 100755 index 0000000..320650b --- /dev/null +++ b/phoneuse/start_frida_bridge.sh @@ -0,0 +1,10 @@ +#!/bin/bash +# Start frida_bridge.py on Mac: tunnels device->Mac for android_mcp.py's +# frida_call / frida_inspect_tools, then runs the bridge in this venv +# (pinned frida==17.9.10 to match the on-device frida-server version). +set -euo pipefail +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ADB="${SCRIPT_DIR}/../platform-tools/adb" + +"$ADB" reverse tcp:8767 tcp:8767 +exec "${SCRIPT_DIR}/.venv/bin/python3" "${SCRIPT_DIR}/frida_bridge.py"