Commit 3dfabb19b2c3ac1db00473f12f4d9bae5325d244

Authored by yibo
1 parent d79aff6c

feat(lifelog/crm+chat): 多服务商LLM + tool calling + 语义搜图embed落地

## 多服务商 & 任务槽(ModelConfig + SettingsScreen)
- 新增 ModelConfig:LlmProvider 注册表(本地/Hermes + 用户自定义远程)+ ModelSlot(VL/EMBED 槽绑定),SettingsScreen 可视化管理
- ChatConfig 加 ChatTargetKind.REMOTE;targets() 动态合并本地+Hermes+用户服务商
- VisionClient / EmbedClient / PhotoVlWorker 统一走 ModelConfig.slotEndpoint 取端点

## Tool Calling & Agentic Loop(LifelogTools + ChatViewModel)
- LifelogTools:6 个本地工具(search_transcripts/persons/photos/timeline/summaries/followups),OpenAI function schema + 端上 Room 执行
- ChatViewModel:agenticLoop 检测 tool_calls → 调 LifelogTools.execute() → tool role 追回 → 多轮直到无 tool call

## 语义搜图 embed 落地(Pixel 设备侧 + 补算)
- 下载 bge-small-zh-v1.5-q8_0.gguf(25MB,512维中文)到 Pixel ~/models/
- llama-swap config.yaml 新增 "embed" 条目(--embedding --pooling mean,ttl 600)
- 补算向量:PhotoDao.withCaptionNoEmbedding/updateCaptionEmbedding + PhotologViewModel.backfillCaptionEmbeddings()
- AlbumTab ProgressHeader 加「🔢 补算语义向量」按钮(有 caption 无 embedding 时显示,带进度)

## 文档
- HANDOFF.md:P0/P1/P2 全部打 ✅,补 v0.9 已完成列表
- CLAUDE.md:v0.8→v0.9,⭐待做项更新为已完成

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
CLAUDE.md
@@ -140,12 +140,12 @@ Foreground Service (foregroundServiceType=microphone, WAKE_LOCK partial) @@ -140,12 +140,12 @@ Foreground Service (foregroundServiceType=microphone, WAKE_LOCK partial)
140 140
141 完整待办、估时、技术路线见 `lifelog-recorder/HANDOFF.md` §五。 141 完整待办、估时、技术路线见 `lifelog-recorder/HANDOFF.md` §五。
142 142
143 -✅ **v0.2–v0.8 已完成**:Opus 编码 / BOOT 自启 + 电池白名单 / FTS4 中文搜索 / 段播放 / **声纹在线聚类** / **会话级纪要(按时间间隔切会话)** / **底部 4-tab + 子 tab UI** / **App 内对话(本地 2B/4B + Hermes)** / **本地 LLM 栈(llama-swap)** / **Room 迁移(去 destructive)** / **block-based diarization-first 录音管线(v0.7,攒块→整块分轮→逐 turn 转写)** / **声纹纠偏 L1 冻结 anchor + L2 多 anchor 重建** / **设置页(`SettingsScreen`:纪要模型后端 / 聚类阈值 / llama-swap 端点 / 电池豁免)** / **photolog 人脉 CRM 层(人物档案 + 相册人脸 + 微信通讯录/朋友圈/AI 印象)** / **存储 TTL(超期删 .opus,文字永留)** / **全相册自动索引(v0.8:MediaStore 全量扫描 + Tier-0 人脸/Tier-1 VL 分层管线 + 关键词/语义搜图 + VL 暂停/停止控制,Room v14)**。 143 +✅ **v0.2–v0.9 已完成**:Opus 编码 / BOOT 自启 + 电池白名单 / FTS4 中文搜索 / 段播放 / **声纹在线聚类** / **会话级纪要(按时间间隔切会话)** / **底部 4-tab + 子 tab UI** / **App 内对话(本地 2B/4B + Hermes)** / **本地 LLM 栈(llama-swap)** / **Room 迁移(去 destructive)** / **block-based diarization-first 录音管线(v0.7,攒块→整块分轮→逐 turn 转写)** / **声纹纠偏 L1 冻结 anchor + L2 多 anchor 重建** / **设置页(`SettingsScreen`:纪要模型后端 / 聚类阈值 / llama-swap 端点 / 电池豁免)** / **photolog 人脉 CRM 层(人物档案 + 相册人脸 + 微信通讯录/朋友圈/AI 印象)** / **存储 TTL(超期删 .opus,文字永留)** / **全相册自动索引(v0.8:MediaStore 全量扫描 + Tier-0 人脸/Tier-1 VL 分层管线 + 关键词/语义搜图 + VL 暂停/停止控制,Room v14)** / **微信抽取完全端上化(v0.8 P2 #6:SQLCipher key 端上推导 + 联系人/头像/聊天/朋友圈全不依赖 Mac)** / **Transcript→Person 提及关系(v0.9 P0 #5:词典反查 + MentionScanner 实时扫 + mentions 表 + 「被提到」tab)** / **人↔人共现边(v0.9 P1 #1:RelationBuilder + relations 表 + 「相关」tab + CommunitiesScreen)** / **待跟进 & 话题实体(FollowupExtractor + TopicExtractor,4B LLM 抽取)**。
144 144
145 剩余摘要: 145 剩余摘要:
146 -- ⭐ **语义搜图落地**(v0.8 已接 `EmbedClient`,但需在 Pixel llama-swap 配 `embed` 模型才生效;中文 embedding GGUF)  
147 -- ⭐ **内部 Transcript 实体 → Person 表**(端上 NER + 共现,取代旧「回链 lemonPCRM」;见下文 §下一步)  
148 -- ⭐ **微信抽取端上化**(消除 photolog 仅剩的 Mac 依赖;见下文差距分析 #6) 146 +- ⭐ **语义搜图落地**(代码已完成:`EmbedClient` + `SettingsScreen` embed 字段;**需在 Pixel llama-swap config.yaml 加 embed 模型**才生效,中文 GGUF 如 bge-small-zh,参见 HANDOFF.md §七 调试陷阱)
  147 +- ✅ **内部 Transcript 实体 → Person 表**(已完成:`crm/MentionExtractor.kt` + `MentionScanner` 词典反查 + `BlockProcessor` 实时扫 + `PersonProfileScreen` 「被提到」tab + `RelationBuilder` 共现边)
  148 +- ✅ **微信抽取端上化**(已完成 2026-06-10:SQLCipher 端上解密 + 联系人/头像/朋友圈全端上,不依赖 Mac)
149 149
150 ### App 即 Personal CRM:与旧 lemonPCRM 的能力对比 150 ### App 即 Personal CRM:与旧 lemonPCRM 的能力对比
151 151
lifelog-recorder/HANDOFF.md
@@ -289,33 +289,29 @@ key = MD5(imei + uin).hex()[:7].lower() @@ -289,33 +289,29 @@ key = MD5(imei + uin).hex()[:7].lower()
289 - v0.5 (2026-06):hermes api_server 端点 + SummaryWorker + 纪要 UI section 289 - v0.5 (2026-06):hermes api_server 端点 + SummaryWorker + 纪要 UI section
290 - v0.6 (2026-06-04):**声纹在线聚类**(SpeakerClusterer,自动「说话人N」+ reclusterAll)/ **5-tab UI**(录音/对话/时间轴/声纹/纪要,时间轴全 CRUD + 批量删导出 + 按说话人/时间/未匹配筛选 + 排序)/ **真暂停**(停采集留模型)/ **会话级纪要**(按时间间隔切会话、可调阈值、要点带 [HH:mm:ss] 点击跳回时间轴、折叠头显示说话人名单)/ **App 内对话**(本地 2B/4B + Hermes 多端点流式)/ **本地 LLM 栈**(llama.cpp HEAD + llama-swap 8080,纪要+对话全本地,见 §六)/ **Room v3 迁移**(去 destructive 改 fail-loud) 290 - v0.6 (2026-06-04):**声纹在线聚类**(SpeakerClusterer,自动「说话人N」+ reclusterAll)/ **5-tab UI**(录音/对话/时间轴/声纹/纪要,时间轴全 CRUD + 批量删导出 + 按说话人/时间/未匹配筛选 + 排序)/ **真暂停**(停采集留模型)/ **会话级纪要**(按时间间隔切会话、可调阈值、要点带 [HH:mm:ss] 点击跳回时间轴、折叠头显示说话人名单)/ **App 内对话**(本地 2B/4B + Hermes 多端点流式)/ **本地 LLM 栈**(llama.cpp HEAD + llama-swap 8080,纪要+对话全本地,见 §六)/ **Room v3 迁移**(去 destructive 改 fail-loud)
291 - v0.7 (2026-06-07):**block-based diarization-first 录音管线**(攒块→整块 pyannote 分轮→逐 turn 转写+嵌入+认人,turn=时间轴一行;去实时字幕;BlockBuffer/BlockProcessor;崩溃 listPending 续跑;commit 07f3d22)/ **声纹纠偏 L1+L2**(L1=reclusterAll & assign 冻结命名 anchor 不漂移止串人;L2=重命名/合并时 rebuildAnchor 用该人全部段均值重建多 anchor)/ 阈值默认 0.45、设置页 0.35-0.65 可调 / **photolog 人脉 CRM 层**(人物档案 + 相册人脸聚类 + 微信通讯录/朋友圈/AI 印象) 291 - v0.7 (2026-06-07):**block-based diarization-first 录音管线**(攒块→整块 pyannote 分轮→逐 turn 转写+嵌入+认人,turn=时间轴一行;去实时字幕;BlockBuffer/BlockProcessor;崩溃 listPending 续跑;commit 07f3d22)/ **声纹纠偏 L1+L2**(L1=reclusterAll & assign 冻结命名 anchor 不漂移止串人;L2=重命名/合并时 rebuildAnchor 用该人全部段均值重建多 anchor)/ 阈值默认 0.45、设置页 0.35-0.65 可调 / **photolog 人脉 CRM 层**(人物档案 + 相册人脸聚类 + 微信通讯录/朋友圈/AI 印象)
292 -- v0.8 (2026-06-09):**全相册自动索引 + 分层管线**(GalleryScanner 扫 MediaStore 全量、content uri 原地登记不拷文件;Tier-0 人脸/时间秒级全量铺、Tier-1 VL 充电时排队补;Room v14;详见 §四 photolog 段)/ **搜图**(关键词 caption+人名 LIKE / 语义 caption 向量余弦,embed 走 llama-swap)/ **VL 控制**(PhotoVlState:暂停/继续/停止切换 + 单张转圈 + 格子状态角标) 292 +- v0.8 (2026-06-09):**全相册自动索引 + 分层管线**(GalleryScanner 扫 MediaStore 全量、content uri 原地登记不拷文件;Tier-0 人脸/时间秒级全量铺、Tier-1 VL 充电时排队补;Room v14;详见 §四 photolog 段)/ **搜图**(关键词 caption+人名 LIKE / 语义 caption 向量余弦,embed 走 llama-swap)/ **VL 控制**(PhotoVlState:暂停/继续/停止切换 + 单张转圈 + 格子状态角标)/ **微信抽取完全端上化(P2 #6)**(SQLCipher key 端上推导 + WeChatChatImporter + 联系人/头像端上解密;好友/群友区分)
  293 +- v0.9 (2026-06-10):**Transcript→Person 提及关系(P0 #5)**(`crm/MentionExtractor.kt` 词典反查;`BlockProcessor` 实时扫;`MentionScanner.rescanAll()` 全量重建;Room v15 `mentions` 表;`PersonProfileScreen` 「被提到」tab)/ **人↔人共现边(P1 #1)**(`RelationBuilder.rebuild()` 按会话在场+被提到配对;Room v16 `relations` 表;「相关」tab + `CommunitiesScreen` 圈子可视化)/ **待跟进(P3 #3 部分)**(`FollowupExtractor` 4B LLM 抽待办;Room v17 `followups` 表;`FollowupsScreen`)/ **话题实体(P3 #2 部分)**(`TopicExtractor` 4B 抽 org/project/event;Room v18/v19 `topics`/`topic_mentions`;`TopicsScreen`)
293 294
294 剩下: 295 剩下:
295 296
296 1. ✅ **存储 TTL**(2026-06-08 完成):`StorageManager.prune` 按天目录删超期 .opus、文字永留;每天 periodic worker;设置页「存储」段(用量 + 保留时长 + 立即清理),默认永久保留。 297 1. ✅ **存储 TTL**(2026-06-08 完成):`StorageManager.prune` 按天目录删超期 .opus、文字永留;每天 periodic worker;设置页「存储」段(用量 + 保留时长 + 立即清理),默认永久保留。
297 2. ✅ **设置页**(基本完成):`SettingsScreen` 已含 纪要模型后端 / 聚类·合并·最短识别阈值 / llama-swap 端点 / 分段间隔 / **定时不录(quiet hours)** / 存储用量+TTL / 电池豁免。 298 2. ✅ **设置页**(基本完成):`SettingsScreen` 已含 纪要模型后端 / 聚类·合并·最短识别阈值 / llama-swap 端点 / 分段间隔 / **定时不录(quiet hours)** / 存储用量+TTL / 电池豁免。
298 3. ✅ **多 anchor 平均**(2026-06-07 完成,L2):重命名/合并某说话人时 `rebuildAnchor` 用该人名下全部段的均值重建 anchor,取代单段注册。 299 3. ✅ **多 anchor 平均**(2026-06-07 完成,L2):重命名/合并某说话人时 `rebuildAnchor` 用该人名下全部段的均值重建 anchor,取代单段注册。
299 -4. **内部 Transcript 实体 → Person 表**(取代旧「回链 lemonPCRM」):在 **app 内部**从 `Transcript.text`/`Summary` 抽实体(人名/项目,端上 LLM 或规则 NER)→ 匹配本地 `Person` → 建「提及」关系(新表 `mentions(segmentId, personId)`),让人物档案显示「TA 被哪些录音提到」(现仅显示「TA 自己说的」=声纹段)。同段共现的人 → 人↔人边。**不再回链 Mac 端 lemonPCRM**(已废弃)。 300 +4. ✅ **内部 Transcript 实体 → Person 表**(已完成):`crm/MentionExtractor.kt`(词典反查:managed 人物的 name/realName/nickname/remark/alias)+ `MentionScanner`(单段实时 & 全量重扫);`BlockProcessor` 每块处理时实时扫 → `mentions(segmentId, personId, matchedText)`;`PersonProfileScreen` 加「被提到」tab;`rescanMentions()` 结束后自动调 `RelationBuilder.rebuild()`。
300 5. ✅ **hermes gateway 自启**(2026-06-04 完成):用 Magisk `service.d` 脚本开机拉起(Termux:Boot 未装,root 方案更稳),见下方「§ 自启(Magisk service.d)」。 301 5. ✅ **hermes gateway 自启**(2026-06-04 完成):用 Magisk `service.d` 脚本开机拉起(Termux:Boot 未装,root 方案更稳),见下方「§ 自启(Magisk service.d)」。
301 -6. **微信抽取端上化**:消除 photolog 仅剩的 Mac 依赖(微信联系人/头像/聊天/朋友圈现由 Mac 一次性抽取推到 `filesDir/wx_enroll/`)。 302 +6. ✅ **微信抽取端上化**(2026-06-10 完成,P2 #6):`WeChatChatImporter`(SQLCipher key 端上推导 + 聊天读取)+ `WeChatSnsImporter`(朋友圈明文库)+ 联系人/头像端上解密(`rcontact`/`img_flag`);好友 vs 群友区分(`wechatFriend` 字段);通讯录「🔄更新」一次性整库解密。不再依赖 Mac 导出器。
302 303
303 ### 端上 CRM 补齐 — TODO(对齐并超越旧 lemonPCRM) 304 ### 端上 CRM 补齐 — TODO(对齐并超越旧 lemonPCRM)
304 305
305 > 旧 Mac 端 lemonPCRM 已废弃,能力在 app 内重建。下表是「app 当前缺、旧 lemonPCRM 有」的 6 项, 306 > 旧 Mac 端 lemonPCRM 已废弃,能力在 app 内重建。下表是「app 当前缺、旧 lemonPCRM 有」的 6 项,
306 > 按优先级排(P0 先做)。能力对比详见 CLAUDE.md §「App 即 Personal CRM」。 307 > 按优先级排(P0 先做)。能力对比详见 CLAUDE.md §「App 即 Personal CRM」。
307 308
308 -- [ ] **P0 · #5 Transcript 内容 → Person「提及」关系**(最高价值,最贴 app 形态)  
309 - - 现状:录音只按**声纹**连到人(谁说的);不从转写文本抽人名/项目。  
310 - - 做法:块处理后(或事后 worker)对 `Transcript.text` 抽实体(端上 LLM 4B 或规则/词典 NER)→ 匹配本地 `Person`(name/realName/nickname/alias,拼音兜底)→ 写新表 `mentions(segmentId, personId, name, confidence)`(Room 迁移 +1)。  
311 - - 出口:`PersonProfileScreen` 加「被提到」tab(现只有「TA 自己说的」=声纹段)。  
312 - - 依赖:端上 4B 已就绪(llama-swap 8080)。  
313 -- [ ] **P1 · #1 人↔人关系(共现边)**  
314 - - 做法:基于 #5,同一会话/时间段被同时提到 + 同时在场(声纹)的人 → `relations(personA, personB, kind, weight)` 共现/认识边,weight 随共现次数累加。  
315 - - 出口:人物档案「相关的人」;后续可做关系图可视化。  
316 -- [ ] **P2 · #6 微信抽取端上化**(消除最后的 Mac 依赖,工程债)  
317 - - 现状:微信联系人/头像/聊天/朋友圈由 Mac 解密 `EnMicroMsg.db`(SQLCipher)+抠头像,推到 `filesDir/wx_enroll/`。  
318 - - 做法:把解密+抽取搬到端上(root + SQLCipher key 端上取),`WeChatEnrollWorker` 直接读微信库;配方见 memory `wechat-android-decrypt.md` / `wechat-avatar-face-enrollment.md`。 309 +- [x] **P0 · #5 Transcript 内容 → Person「提及」关系**(已完成)
  310 + - 词典反查(managed 人物 name/realName/nickname/remark/alias 多别名);`BlockProcessor` 每块实时扫;`MentionScanner.rescanAll()` 全量重扫;`mentions` 表 + `PersonProfileScreen` 「被提到」tab。
  311 +- [x] **P1 · #1 人↔人关系(共现边)**(已完成)
  312 + - `RelationBuilder.rebuild()`(rescanAll 后自动调)按 5min 静音切会话,在场(声纹)+ 被提到(mentions)的人两两配对 → `relations` 表;`PersonProfileScreen` 「相关」tab + `CommunitiesScreen` 圈子。
  313 +- [x] **P2 · #6 微信抽取端上化**(已完成,2026-06-10)
  314 + - SQLCipher key 端上推导(MD5(imei+uin)[:7])+ `WeChatChatImporter` + 联系人/头像/朋友圈全端上。
319 - [ ] **P3 · #2 Organization / Project / Event 实体**(产品决策:要不要从「联系人表」长成「图谱」) 315 - [ ] **P3 · #2 Organization / Project / Event 实体**(产品决策:要不要从「联系人表」长成「图谱」)
320 - 新增节点类型 + 与 Person 的边(属于/参与);可由 #5 的实体抽取顺带产出(项目名/公司名/事件)。 316 - 新增节点类型 + 与 Person 的边(属于/参与);可由 #5 的实体抽取顺带产出(项目名/公司名/事件)。
321 - [ ] **P3 · #3 关系分析**:关系路径(两人间最短链)/ 社群发现(聚类)/ 待跟进(followups:从录音承诺「下周给他发X」抽待办)。依赖 #1 的关系边。 317 - [ ] **P3 · #3 关系分析**:关系路径(两人间最短链)/ 社群发现(聚类)/ 待跟进(followups:从录音承诺「下周给他发X」抽待办)。依赖 #1 的关系边。
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/ChatScreen.kt
@@ -192,7 +192,7 @@ internal fun ChatTab(repo: Repository) { @@ -192,7 +192,7 @@ internal fun ChatTab(repo: Repository) {
192 } 192 }
193 } 193 }
194 194
195 - // lifelog 模式开关。 195 + // 模式开关行
196 Row( 196 Row(
197 modifier = Modifier.fillMaxWidth().padding(horizontal = 12.dp), 197 modifier = Modifier.fillMaxWidth().padding(horizontal = 12.dp),
198 verticalAlignment = Alignment.CenterVertically, 198 verticalAlignment = Alignment.CenterVertically,
@@ -201,12 +201,20 @@ internal fun ChatTab(repo: Repository) { @@ -201,12 +201,20 @@ internal fun ChatTab(repo: Repository) {
201 FilterChip( 201 FilterChip(
202 selected = vm.lifelogMode, 202 selected = vm.lifelogMode,
203 onClick = { vm.setLifelog(!vm.lifelogMode) }, 203 onClick = { vm.setLifelog(!vm.lifelogMode) },
204 - label = { Text("🧠 lifelog 模式") }, 204 + label = { Text("🧠 lifelog") },
205 ) 205 )
206 - Text(  
207 - if (vm.lifelogMode) "回答会带上你的纪要/转写检索" else "纯聊天,不读 lifelog",  
208 - style = MaterialTheme.typography.labelSmall, 206 + FilterChip(
  207 + selected = vm.toolMode,
  208 + onClick = { vm.setTool(!vm.toolMode) },
  209 + label = { Text("🔧 工具") },
209 ) 210 )
  211 + val statusHint = when {
  212 + vm.toolMode && vm.toolStatus.isNotBlank() -> vm.toolStatus
  213 + vm.toolMode -> "AI 可自动查录音/联系人/相册"
  214 + vm.lifelogMode -> "回答带纪要/转写检索"
  215 + else -> "纯聊天"
  216 + }
  217 + Text(statusHint, style = MaterialTheme.typography.labelSmall, modifier = Modifier.weight(1f))
210 } 218 }
211 219
212 HorizontalDivider() 220 HorizontalDivider()
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/MainActivity.kt
@@ -122,9 +122,9 @@ class MainActivity : ComponentActivity() { @@ -122,9 +122,9 @@ class MainActivity : ComponentActivity() {
122 } 122 }
123 123
124 private enum class Tab(val label: String, val icon: String) { 124 private enum class Tab(val label: String, val icon: String) {
125 - People("人物", "👤"), // 通讯录 + 联系人档案(CRM 核心,放第一位)  
126 - Recording("录音", "⏺"), // 顶部子页:录音 / 时间轴 / 声纹 / 纪要  
127 - Chat("对话", "💬"), 125 + Chat("对话", "💬"), // 默认首页:AI 对话 + 工具模式
  126 + People("人物", "👤"),
  127 + Recording("录音", "⏺"),
128 Album("相册", "🖼"), 128 Album("相册", "🖼"),
129 Settings("设置", "⚙"), 129 Settings("设置", "⚙"),
130 } 130 }
@@ -144,7 +144,7 @@ enum class TimeFilter(val label: String) { @@ -144,7 +144,7 @@ enum class TimeFilter(val label: String) {
144 144
145 @Composable 145 @Composable
146 private fun MainScreen() { 146 private fun MainScreen() {
147 - var tab by remember { mutableStateOf(Tab.People) } 147 + var tab by remember { mutableStateOf(Tab.Chat) }
148 val activity = androidx.compose.ui.platform.LocalContext.current 148 val activity = androidx.compose.ui.platform.LocalContext.current
149 val repo = remember(activity) { Repository(activity) } 149 val repo = remember(activity) { Repository(activity) }
150 150
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/PhotologScreen.kt
@@ -161,6 +161,7 @@ internal fun AlbumTab(repo: Repository) { @@ -161,6 +161,7 @@ internal fun AlbumTab(repo: Repository) {
161 val vlRunning by PhotoVlState.running.collectAsState() 161 val vlRunning by PhotoVlState.running.collectAsState()
162 val vlPaused by PhotoVlState.paused.collectAsState() 162 val vlPaused by PhotoVlState.paused.collectAsState()
163 val vlCurrentId by PhotoVlState.currentPhotoId.collectAsState() 163 val vlCurrentId by PhotoVlState.currentPhotoId.collectAsState()
  164 + val embBackfill by vm.embBackfill.collectAsState()
164 165
165 // 读相册权限(13+ READ_MEDIA_IMAGES;12 及以下 READ_EXTERNAL_STORAGE)。授予即扫全相册。 166 // 读相册权限(13+ READ_MEDIA_IMAGES;12 及以下 READ_EXTERNAL_STORAGE)。授予即扫全相册。
166 val mediaPerm = if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.TIRAMISU) 167 val mediaPerm = if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.TIRAMISU)
@@ -221,6 +222,7 @@ internal fun AlbumTab(repo: Repository) { @@ -221,6 +222,7 @@ internal fun AlbumTab(repo: Repository) {
221 } else { 222 } else {
222 ProgressHeader( 223 ProgressHeader(
223 p = progress, running = vlRunning, paused = vlPaused, 224 p = progress, running = vlRunning, paused = vlPaused,
  225 + embBackfill = embBackfill,
224 onRun = { 226 onRun = {
225 vm.scanGallery(); vm.runVlNow() 227 vm.scanGallery(); vm.runVlNow()
226 android.widget.Toast.makeText( 228 android.widget.Toast.makeText(
@@ -229,6 +231,7 @@ internal fun AlbumTab(repo: Repository) { @@ -229,6 +231,7 @@ internal fun AlbumTab(repo: Repository) {
229 }, 231 },
230 onPause = { vm.pauseVl() }, 232 onPause = { vm.pauseVl() },
231 onStop = { vm.stopVl() }, 233 onStop = { vm.stopVl() },
  234 + onBackfillEmbed = { vm.backfillCaptionEmbeddings() },
232 ) 235 )
233 } 236 }
234 // 搜索框 + 语义按钮。 237 // 搜索框 + 语义按钮。
@@ -312,9 +315,11 @@ private fun ProgressHeader( @@ -312,9 +315,11 @@ private fun ProgressHeader(
312 p: PhotoProgress, 315 p: PhotoProgress,
313 running: Boolean, 316 running: Boolean,
314 paused: Boolean, 317 paused: Boolean,
  318 + embBackfill: Pair<Int, Int>?,
315 onRun: () -> Unit, 319 onRun: () -> Unit,
316 onPause: () -> Unit, 320 onPause: () -> Unit,
317 onStop: () -> Unit, 321 onStop: () -> Unit,
  322 + onBackfillEmbed: () -> Unit,
318 ) { 323 ) {
319 if (p.total == 0) { 324 if (p.total == 0) {
320 Row( 325 Row(
@@ -359,6 +364,23 @@ private fun ProgressHeader( @@ -359,6 +364,23 @@ private fun ProgressHeader(
359 } else { 364 } else {
360 Text("✓ 全部已理解", style = MaterialTheme.typography.labelSmall) 365 Text("✓ 全部已理解", style = MaterialTheme.typography.labelSmall)
361 } 366 }
  367 + // 补算语义向量(有 caption 但无 embedding 时显示)。
  368 + if (embBackfill != null) {
  369 + val (done, total) = embBackfill
  370 + Row(
  371 + verticalAlignment = Alignment.CenterVertically,
  372 + horizontalArrangement = Arrangement.spacedBy(8.dp),
  373 + modifier = Modifier.padding(top = 4.dp),
  374 + ) {
  375 + CircularProgressIndicator(modifier = Modifier.size(14.dp), strokeWidth = 2.dp)
  376 + Text("语义向量 $done/$total", style = MaterialTheme.typography.labelSmall)
  377 + }
  378 + } else if (p.captionDone > 0) {
  379 + TextButton(
  380 + onClick = onBackfillEmbed,
  381 + contentPadding = androidx.compose.foundation.layout.PaddingValues(horizontal = 0.dp, vertical = 0.dp),
  382 + ) { Text("🔢 补算语义向量", style = MaterialTheme.typography.labelSmall) }
  383 + }
362 } 384 }
363 } 385 }
364 386
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/PhotologViewModel.kt
@@ -116,7 +116,8 @@ class PhotologViewModel(app: Application) : AndroidViewModel(app) { @@ -116,7 +116,8 @@ class PhotologViewModel(app: Application) : AndroidViewModel(app) {
116 val app = getApplication<Application>() 116 val app = getApplication<Application>()
117 try { 117 try {
118 val rows = withContext(Dispatchers.IO) { 118 val rows = withContext(Dispatchers.IO) {
119 - val embed = EmbedClient(ChatConfig.localBase(app), ChatConfig.modelEmbed(app)) 119 + val (embedBase, embedKey, embedModel) = com.lemon.lifelog.chat.ModelConfig.slotEndpoint(app, com.lemon.lifelog.chat.ModelSlot.EMBED)
  120 + val embed = EmbedClient(embedBase, embedModel, embedKey)
120 val q = embed.embed(query.trim()) 121 val q = embed.embed(query.trim())
121 val vecs = repo.photos.captionVecs() 122 val vecs = repo.photos.captionVecs()
122 if (vecs.isEmpty()) return@withContext emptyList<PhotoRow>() 123 if (vecs.isEmpty()) return@withContext emptyList<PhotoRow>()
@@ -138,6 +139,35 @@ class PhotologViewModel(app: Application) : AndroidViewModel(app) { @@ -138,6 +139,35 @@ class PhotologViewModel(app: Application) : AndroidViewModel(app) {
138 } 139 }
139 } 140 }
140 141
  142 + /**
  143 + * 补算 captionEmbedding:对已有 caption 但缺 embedding 的照片(embed 模型后配时回填),
  144 + * 只调 embed 端点,不重跑 VL。返回成功补算数量。
  145 + */
  146 + private val _embBackfill = MutableStateFlow<Pair<Int, Int>?>(null) // (done, total) | null=idle
  147 + val embBackfill: StateFlow<Pair<Int, Int>?> = _embBackfill
  148 +
  149 + fun backfillCaptionEmbeddings() {
  150 + if (_embBackfill.value != null) return
  151 + viewModelScope.launch(Dispatchers.IO) {
  152 + val app = getApplication<Application>()
  153 + val pending = repo.photos.withCaptionNoEmbedding()
  154 + if (pending.isEmpty()) { _embBackfill.value = null; return@launch }
  155 + val (embedBase, embedKey, embedModel) = com.lemon.lifelog.chat.ModelConfig.slotEndpoint(app, com.lemon.lifelog.chat.ModelSlot.EMBED)
  156 + val client = EmbedClient(embedBase, embedModel, embedKey)
  157 + var done = 0
  158 + _embBackfill.value = 0 to pending.size
  159 + for (p in pending) {
  160 + runCatching {
  161 + val emb = client.embed(p.caption)
  162 + repo.photos.updateCaptionEmbedding(p.id, com.lemon.lifelog.vision.ImageUtil.packEmbedding(emb))
  163 + done++
  164 + }
  165 + _embBackfill.value = done to pending.size
  166 + }
  167 + _embBackfill.value = null
  168 + }
  169 + }
  170 +
141 fun createPersonAndTag(faceId: Long, name: String) { 171 fun createPersonAndTag(faceId: Long, name: String) {
142 viewModelScope.launch { 172 viewModelScope.launch {
143 // 手动建的人 = 我关心的 → 直接 managed。 173 // 手动建的人 = 我关心的 → 直接 managed。
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/SettingsScreen.kt
@@ -17,6 +17,7 @@ import androidx.compose.foundation.layout.height @@ -17,6 +17,7 @@ import androidx.compose.foundation.layout.height
17 import androidx.compose.foundation.layout.padding 17 import androidx.compose.foundation.layout.padding
18 import androidx.compose.foundation.rememberScrollState 18 import androidx.compose.foundation.rememberScrollState
19 import androidx.compose.foundation.verticalScroll 19 import androidx.compose.foundation.verticalScroll
  20 +import androidx.compose.material3.AlertDialog
20 import androidx.compose.material3.Button 21 import androidx.compose.material3.Button
21 import androidx.compose.material3.DropdownMenu 22 import androidx.compose.material3.DropdownMenu
22 import androidx.compose.material3.DropdownMenuItem 23 import androidx.compose.material3.DropdownMenuItem
@@ -26,6 +27,7 @@ import androidx.compose.material3.HorizontalDivider @@ -26,6 +27,7 @@ import androidx.compose.material3.HorizontalDivider
26 import androidx.compose.material3.MaterialTheme 27 import androidx.compose.material3.MaterialTheme
27 import androidx.compose.material3.OutlinedTextField 28 import androidx.compose.material3.OutlinedTextField
28 import androidx.compose.material3.Text 29 import androidx.compose.material3.Text
  30 +import androidx.compose.material3.TextButton
29 import androidx.compose.runtime.Composable 31 import androidx.compose.runtime.Composable
30 import androidx.compose.runtime.LaunchedEffect 32 import androidx.compose.runtime.LaunchedEffect
31 import androidx.compose.runtime.getValue 33 import androidx.compose.runtime.getValue
@@ -40,6 +42,9 @@ import com.lemon.lifelog.asr.SpeakerClusterer @@ -40,6 +42,9 @@ import com.lemon.lifelog.asr.SpeakerClusterer
40 import com.lemon.lifelog.service.BlockBuffer 42 import com.lemon.lifelog.service.BlockBuffer
41 import com.lemon.lifelog.service.QuietHours 43 import com.lemon.lifelog.service.QuietHours
42 import com.lemon.lifelog.chat.ChatConfig 44 import com.lemon.lifelog.chat.ChatConfig
  45 +import com.lemon.lifelog.chat.LlmProvider
  46 +import com.lemon.lifelog.chat.ModelConfig
  47 +import com.lemon.lifelog.chat.ModelSlot
43 import com.lemon.lifelog.storage.StorageManager 48 import com.lemon.lifelog.storage.StorageManager
44 import com.lemon.lifelog.summary.SummaryScheduler 49 import com.lemon.lifelog.summary.SummaryScheduler
45 import com.lemon.lifelog.summary.SummaryWorker 50 import com.lemon.lifelog.summary.SummaryWorker
@@ -71,14 +76,29 @@ fun SettingsScreen(activity: Context) { @@ -71,14 +76,29 @@ fun SettingsScreen(activity: Context) {
71 var pruneMsg by remember { mutableStateOf("") } 76 var pruneMsg by remember { mutableStateOf("") }
72 val scope = rememberCoroutineScope() 77 val scope = rememberCoroutineScope()
73 LaunchedEffect(Unit) { audioUsage = StorageManager.audioUsageBytes(activity.filesDir) } 78 LaunchedEffect(Unit) { audioUsage = StorageManager.audioUsageBytes(activity.filesDir) }
74 - val backendTargets = remember { ChatConfig.targets(activity) } 79 + var configRev by remember { mutableStateOf(0) }
  80 + val backendTargets = remember(configRev) { ChatConfig.targets(activity) }
75 81
76 var localBase by remember { mutableStateOf(ChatConfig.localBase(activity)) } 82 var localBase by remember { mutableStateOf(ChatConfig.localBase(activity)) }
77 var m2b by remember { mutableStateOf(ChatConfig.model2b(activity)) } 83 var m2b by remember { mutableStateOf(ChatConfig.model2b(activity)) }
78 var m4b by remember { mutableStateOf(ChatConfig.model4b(activity)) } 84 var m4b by remember { mutableStateOf(ChatConfig.model4b(activity)) }
79 - var mEmbed by remember { mutableStateOf(ChatConfig.modelEmbed(activity)) }  
80 var endpointMsg by remember { mutableStateOf("") } 85 var endpointMsg by remember { mutableStateOf("") }
81 86
  87 + var hermesUrl by remember(configRev) { mutableStateOf(ModelConfig.hermesProvider(activity).baseUrl) }
  88 + var hermesKey by remember(configRev) { mutableStateOf(ModelConfig.hermesProvider(activity).apiKey) }
  89 + var hermesModel by remember(configRev) { mutableStateOf(ModelConfig.hermesProvider(activity).chatModel) }
  90 + var hermesSaveMsg by remember { mutableStateOf("") }
  91 +
  92 + var editingProvider by remember { mutableStateOf<LlmProvider?>(null) }
  93 + var isNewProvider by remember { mutableStateOf(false) }
  94 +
  95 + var vlPid by remember(configRev) { mutableStateOf(ModelConfig.slotProviderId(activity, ModelSlot.VL)) }
  96 + var vlModel by remember(configRev) { mutableStateOf(ModelConfig.slotModel(activity, ModelSlot.VL)) }
  97 + var vlSaveMsg by remember { mutableStateOf("") }
  98 + var embedPid by remember(configRev) { mutableStateOf(ModelConfig.slotProviderId(activity, ModelSlot.EMBED)) }
  99 + var embedModel by remember(configRev) { mutableStateOf(ModelConfig.slotModel(activity, ModelSlot.EMBED)) }
  100 + var embedSaveMsg by remember { mutableStateOf("") }
  101 +
82 var batteryExempt by remember { mutableStateOf(isBatteryExempt(activity)) } 102 var batteryExempt by remember { mutableStateOf(isBatteryExempt(activity)) }
83 103
84 Column( 104 Column(
@@ -125,7 +145,7 @@ fun SettingsScreen(activity: Context) { @@ -125,7 +145,7 @@ fun SettingsScreen(activity: Context) {
125 style = MaterialTheme.typography.labelSmall) 145 style = MaterialTheme.typography.labelSmall)
126 146
127 HorizontalDivider() 147 HorizontalDivider()
128 - Text("本地模型端点(对话 + 纪要共用 llama-swap)", style = MaterialTheme.typography.titleMedium) 148 + Text("本地 llama-swap", style = MaterialTheme.typography.titleMedium)
129 OutlinedTextField(value = localBase, onValueChange = { localBase = it }, 149 OutlinedTextField(value = localBase, onValueChange = { localBase = it },
130 label = { Text("baseUrl") }, singleLine = true, modifier = Modifier.fillMaxWidth()) 150 label = { Text("baseUrl") }, singleLine = true, modifier = Modifier.fillMaxWidth())
131 Row(horizontalArrangement = Arrangement.spacedBy(8.dp)) { 151 Row(horizontalArrangement = Arrangement.spacedBy(8.dp)) {
@@ -134,18 +154,101 @@ fun SettingsScreen(activity: Context) { @@ -134,18 +154,101 @@ fun SettingsScreen(activity: Context) {
134 OutlinedTextField(value = m4b, onValueChange = { m4b = it }, 154 OutlinedTextField(value = m4b, onValueChange = { m4b = it },
135 label = { Text("4B model 名") }, singleLine = true, modifier = Modifier.weight(1f)) 155 label = { Text("4B model 名") }, singleLine = true, modifier = Modifier.weight(1f))
136 } 156 }
137 - OutlinedTextField(value = mEmbed, onValueChange = { mEmbed = it },  
138 - label = { Text("embed model 名(语义搜图,llama-swap 里配 --embedding)") },  
139 - singleLine = true, modifier = Modifier.fillMaxWidth())  
140 Row(verticalAlignment = Alignment.CenterVertically, horizontalArrangement = Arrangement.spacedBy(8.dp)) { 157 Row(verticalAlignment = Alignment.CenterVertically, horizontalArrangement = Arrangement.spacedBy(8.dp)) {
141 Button(onClick = { 158 Button(onClick = {
142 - ChatConfig.save(activity, localBase, m2b, m4b, mEmbed) 159 + ChatConfig.save(activity, localBase, m2b, m4b)
143 endpointMsg = "已保存" 160 endpointMsg = "已保存"
144 - }) { Text("保存端点") } 161 + configRev++
  162 + }) { Text("保存") }
145 if (endpointMsg.isNotEmpty()) Text(endpointMsg, style = MaterialTheme.typography.labelSmall) 163 if (endpointMsg.isNotEmpty()) Text(endpointMsg, style = MaterialTheme.typography.labelSmall)
146 } 164 }
147 165
148 HorizontalDivider() 166 HorizontalDivider()
  167 + Text("Hermes 网关", style = MaterialTheme.typography.titleMedium)
  168 + OutlinedTextField(value = hermesUrl, onValueChange = { hermesUrl = it },
  169 + label = { Text("接口地址") }, singleLine = true, modifier = Modifier.fillMaxWidth())
  170 + OutlinedTextField(value = hermesKey, onValueChange = { hermesKey = it },
  171 + label = { Text("API Key") }, singleLine = true, modifier = Modifier.fillMaxWidth())
  172 + OutlinedTextField(value = hermesModel, onValueChange = { hermesModel = it },
  173 + label = { Text("路由模型名(如 kimi-for-coding / qwen-plus)") },
  174 + singleLine = true, modifier = Modifier.fillMaxWidth())
  175 + Row(verticalAlignment = Alignment.CenterVertically, horizontalArrangement = Arrangement.spacedBy(8.dp)) {
  176 + Button(onClick = {
  177 + ModelConfig.saveHermes(activity, hermesUrl, hermesKey, hermesModel)
  178 + hermesSaveMsg = "已保存"
  179 + configRev++
  180 + }) { Text("保存 Hermes") }
  181 + if (hermesSaveMsg.isNotEmpty()) Text(hermesSaveMsg, style = MaterialTheme.typography.labelSmall)
  182 + }
  183 +
  184 + HorizontalDivider()
  185 + Text("远程服务商", style = MaterialTheme.typography.titleMedium)
  186 + Text("添加任意 OpenAI 兼容 API(通义千问 / Kimi / MiniMax 等),对话模型名不为空则出现在聊天列表。",
  187 + style = MaterialTheme.typography.labelSmall)
  188 + val userProviders = remember(configRev) { ModelConfig.userProviders(activity) }
  189 + userProviders.forEach { p ->
  190 + Row(
  191 + modifier = Modifier.fillMaxWidth(),
  192 + verticalAlignment = Alignment.CenterVertically,
  193 + horizontalArrangement = Arrangement.spacedBy(8.dp),
  194 + ) {
  195 + Column(modifier = Modifier.weight(1f)) {
  196 + Text(p.label, style = MaterialTheme.typography.bodyMedium)
  197 + Text(
  198 + buildString {
  199 + append(p.baseUrl.take(40))
  200 + if (p.chatModel.isNotBlank()) append(" ${p.chatModel}")
  201 + },
  202 + style = MaterialTheme.typography.labelSmall,
  203 + )
  204 + }
  205 + TextButton(onClick = { editingProvider = p; isNewProvider = false }) { Text("编辑") }
  206 + TextButton(onClick = { ModelConfig.deleteProvider(activity, p.id); configRev++ }) { Text("删除") }
  207 + }
  208 + }
  209 + Button(onClick = {
  210 + editingProvider = LlmProvider(ModelConfig.generateId(), "", "", "", "")
  211 + isNewProvider = true
  212 + }) { Text("+ 添加服务商") }
  213 +
  214 + HorizontalDivider()
  215 + Text("模型分配", style = MaterialTheme.typography.titleMedium)
  216 + Text("指定 VL(图像理解)和语义向量各用哪个服务商 + 模型。",
  217 + style = MaterialTheme.typography.labelSmall)
  218 + val allProviders = remember(configRev) { ModelConfig.allProviders(activity) }
  219 + // VL slot
  220 + Text("图像理解(VL)", style = MaterialTheme.typography.labelLarge)
  221 + DropdownSetting(
  222 + "服务商", allProviders.firstOrNull { it.id == vlPid }?.label ?: vlPid,
  223 + allProviders.map { it.label to it.id },
  224 + ) { vlPid = it }
  225 + Row(verticalAlignment = Alignment.CenterVertically, horizontalArrangement = Arrangement.spacedBy(8.dp)) {
  226 + OutlinedTextField(value = vlModel, onValueChange = { vlModel = it },
  227 + label = { Text("模型名") }, singleLine = true, modifier = Modifier.weight(1f))
  228 + Button(onClick = {
  229 + ModelConfig.setSlot(activity, ModelSlot.VL, vlPid, vlModel)
  230 + vlSaveMsg = "已保存"
  231 + }) { Text("保存") }
  232 + }
  233 + if (vlSaveMsg.isNotEmpty()) Text(vlSaveMsg, style = MaterialTheme.typography.labelSmall)
  234 + // EMBED slot
  235 + Text("语义向量", style = MaterialTheme.typography.labelLarge)
  236 + DropdownSetting(
  237 + "服务商", allProviders.firstOrNull { it.id == embedPid }?.label ?: embedPid,
  238 + allProviders.map { it.label to it.id },
  239 + ) { embedPid = it }
  240 + Row(verticalAlignment = Alignment.CenterVertically, horizontalArrangement = Arrangement.spacedBy(8.dp)) {
  241 + OutlinedTextField(value = embedModel, onValueChange = { embedModel = it },
  242 + label = { Text("模型名(llama-swap 里配 --embedding 的 GGUF 名)") },
  243 + singleLine = true, modifier = Modifier.weight(1f))
  244 + Button(onClick = {
  245 + ModelConfig.setSlot(activity, ModelSlot.EMBED, embedPid, embedModel)
  246 + embedSaveMsg = "已保存"
  247 + }) { Text("保存") }
  248 + }
  249 + if (embedSaveMsg.isNotEmpty()) Text(embedSaveMsg, style = MaterialTheme.typography.labelSmall)
  250 +
  251 + HorizontalDivider()
149 Text("录音 / 电池", style = MaterialTheme.typography.titleMedium) 252 Text("录音 / 电池", style = MaterialTheme.typography.titleMedium)
150 DropdownSetting("分段间隔(静音多久收尾一段)", "$blockSilenceMin 分钟", 253 DropdownSetting("分段间隔(静音多久收尾一段)", "$blockSilenceMin 分钟",
151 listOf(1, 2, 3, 5, 10, 15).map { "$it 分钟" to it }) { 254 listOf(1, 2, 3, 5, 10, 15).map { "$it 分钟" to it }) {
@@ -209,6 +312,65 @@ fun SettingsScreen(activity: Context) { @@ -209,6 +312,65 @@ fun SettingsScreen(activity: Context) {
209 312
210 Spacer(Modifier.height(24.dp)) 313 Spacer(Modifier.height(24.dp))
211 } 314 }
  315 +
  316 + editingProvider?.let { p ->
  317 + ProviderEditDialog(
  318 + initial = p,
  319 + isNew = isNewProvider,
  320 + onConfirm = { updated ->
  321 + if (isNewProvider) ModelConfig.addProvider(activity, updated)
  322 + else ModelConfig.updateProvider(activity, updated)
  323 + editingProvider = null
  324 + configRev++
  325 + },
  326 + onDismiss = { editingProvider = null },
  327 + )
  328 + }
  329 +}
  330 +
  331 +@Composable
  332 +private fun ProviderEditDialog(
  333 + initial: LlmProvider,
  334 + isNew: Boolean,
  335 + onConfirm: (LlmProvider) -> Unit,
  336 + onDismiss: () -> Unit,
  337 +) {
  338 + var label by remember { mutableStateOf(initial.label) }
  339 + var baseUrl by remember { mutableStateOf(initial.baseUrl) }
  340 + var apiKey by remember { mutableStateOf(initial.apiKey) }
  341 + var chatModel by remember { mutableStateOf(initial.chatModel) }
  342 + AlertDialog(
  343 + onDismissRequest = onDismiss,
  344 + title = { Text(if (isNew) "添加服务商" else "编辑服务商") },
  345 + text = {
  346 + Column(verticalArrangement = Arrangement.spacedBy(8.dp)) {
  347 + OutlinedTextField(value = label, onValueChange = { label = it },
  348 + label = { Text("名称") }, singleLine = true, modifier = Modifier.fillMaxWidth())
  349 + OutlinedTextField(value = baseUrl, onValueChange = { baseUrl = it },
  350 + label = { Text("接口地址 (baseUrl)") }, singleLine = true, modifier = Modifier.fillMaxWidth())
  351 + OutlinedTextField(value = apiKey, onValueChange = { apiKey = it },
  352 + label = { Text("API Key") }, singleLine = true, modifier = Modifier.fillMaxWidth())
  353 + OutlinedTextField(value = chatModel, onValueChange = { chatModel = it },
  354 + label = { Text("对话模型名(留空则不出现在聊天列表)") },
  355 + singleLine = true, modifier = Modifier.fillMaxWidth())
  356 + Text("接口地址需兼容 OpenAI /v1/chat/completions 协议。",
  357 + style = MaterialTheme.typography.labelSmall)
  358 + }
  359 + },
  360 + confirmButton = {
  361 + TextButton(onClick = {
  362 + if (label.isNotBlank() && baseUrl.isNotBlank()) {
  363 + onConfirm(initial.copy(
  364 + label = label.trim(),
  365 + baseUrl = baseUrl.trimEnd('/'),
  366 + apiKey = apiKey.trim(),
  367 + chatModel = chatModel.trim(),
  368 + ))
  369 + }
  370 + }) { Text("保存") }
  371 + },
  372 + dismissButton = { TextButton(onClick = onDismiss) { Text("取消") } },
  373 + )
212 } 374 }
213 375
214 /** 字节数 → 人类可读(KB/MB/GB)。 */ 376 /** 字节数 → 人类可读(KB/MB/GB)。 */
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/chat/ChatClient.kt
@@ -12,6 +12,18 @@ import org.json.JSONObject @@ -12,6 +12,18 @@ import org.json.JSONObject
12 import java.io.IOException 12 import java.io.IOException
13 import java.util.concurrent.TimeUnit 13 import java.util.concurrent.TimeUnit
14 14
  15 +data class ToolCall(
  16 + val id: String,
  17 + val name: String,
  18 + val arguments: String,
  19 +)
  20 +
  21 +data class ToolChatResult(
  22 + val text: String,
  23 + val toolCalls: List<ToolCall>?,
  24 + val rawToolCallsJson: JSONArray?,
  25 +)
  26 +
15 /** 27 /**
16 * OpenAI 兼容的聊天客户端,支持 SSE 流式(stream:true)增量回复。 28 * OpenAI 兼容的聊天客户端,支持 SSE 流式(stream:true)增量回复。
17 * 29 *
@@ -124,4 +136,65 @@ class ChatClient { @@ -124,4 +136,65 @@ class ChatClient {
124 if (choices.length() == 0) return "" 136 if (choices.length() == 0) return ""
125 return choices.getJSONObject(0).optJSONObject("message")?.optString("content", "").orEmpty() 137 return choices.getJSONObject(0).optJSONObject("message")?.optString("content", "").orEmpty()
126 } 138 }
  139 +
  140 + /**
  141 + * 非流式 function calling 请求。
  142 + * 传入 [messages](JSONObject 列表,支持 tool/assistant with tool_calls 等 role)和
  143 + * [tools](LifelogTools.SCHEMAS),返回 [ToolChatResult]:
  144 + * - 若模型给出 tool_calls → toolCalls 非 null,调用方执行工具后再次调用本方法
  145 + * - 若模型给出最终文字 → toolCalls == null,text 是答案
  146 + */
  147 + @Throws(IOException::class)
  148 + suspend fun chatWithTools(
  149 + target: ChatTarget,
  150 + messages: List<JSONObject>,
  151 + tools: JSONArray,
  152 + maxTokens: Int = 1024,
  153 + ): ToolChatResult = withContext(Dispatchers.IO) {
  154 + val payload = JSONObject().apply {
  155 + put("model", target.model)
  156 + put("stream", false)
  157 + put("max_tokens", maxTokens)
  158 + put("tools", tools)
  159 + put("tool_choice", "auto")
  160 + if (target.kind == ChatTargetKind.LOCAL) {
  161 + put("chat_template_kwargs", JSONObject().put("enable_thinking", false))
  162 + }
  163 + put("messages", JSONArray().apply { messages.forEach { put(it) } })
  164 + }
  165 + val builder = Request.Builder()
  166 + .url("${target.baseUrl}/v1/chat/completions")
  167 + .header("Content-Type", "application/json")
  168 + .post(payload.toString().toRequestBody(JSON))
  169 + if (target.apiKey.isNotBlank()) builder.header("Authorization", "Bearer ${target.apiKey}")
  170 +
  171 + http.newCall(builder.build()).execute().use { resp ->
  172 + val body = resp.body?.string().orEmpty()
  173 + if (!resp.isSuccessful) throw IOException("${target.label} ${resp.code}: ${body.take(300)}")
  174 + parseToolChatResponse(body)
  175 + }
  176 + }
  177 +
  178 + private fun parseToolChatResponse(body: String): ToolChatResult {
  179 + val json = JSONObject(body)
  180 + val choices = json.optJSONArray("choices") ?: throw IOException("no choices")
  181 + if (choices.length() == 0) throw IOException("empty choices")
  182 + val message = choices.getJSONObject(0).optJSONObject("message")
  183 + ?: throw IOException("no message")
  184 + val text = if (message.isNull("content")) "" else message.optString("content", "")
  185 + val rawCalls = message.optJSONArray("tool_calls")
  186 + if (rawCalls != null && rawCalls.length() > 0) {
  187 + val calls = (0 until rawCalls.length()).map { i ->
  188 + val tc = rawCalls.getJSONObject(i)
  189 + val fn = tc.getJSONObject("function")
  190 + ToolCall(
  191 + id = tc.getString("id"),
  192 + name = fn.getString("name"),
  193 + arguments = fn.optString("arguments", "{}"),
  194 + )
  195 + }
  196 + return ToolChatResult(text, calls, rawCalls)
  197 + }
  198 + return ToolChatResult(text, null, null)
  199 + }
127 } 200 }
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/chat/ChatConfig.kt
@@ -3,7 +3,7 @@ package com.lemon.lifelog.chat @@ -3,7 +3,7 @@ package com.lemon.lifelog.chat
3 import android.content.Context 3 import android.content.Context
4 import com.lemon.lifelog.BuildConfig 4 import com.lemon.lifelog.BuildConfig
5 5
6 -enum class ChatTargetKind { LOCAL, HERMES } 6 +enum class ChatTargetKind { LOCAL, HERMES, REMOTE }
7 7
8 /** 8 /**
9 * 一个可聊天的对象 = 一个 OpenAI 兼容端点 + 一个 model 名。 9 * 一个可聊天的对象 = 一个 OpenAI 兼容端点 + 一个 model 名。
@@ -34,6 +34,7 @@ object ChatConfig { @@ -34,6 +34,7 @@ object ChatConfig {
34 /** 语义搜索的文本 embedding model 名(llama-swap 里配 --embedding 的 GGUF,如 bge-small-zh)。 */ 34 /** 语义搜索的文本 embedding model 名(llama-swap 里配 --embedding 的 GGUF,如 bge-small-zh)。 */
35 const val DEFAULT_MODEL_EMBED = "embed" 35 const val DEFAULT_MODEL_EMBED = "embed"
36 36
  37 + const val ID_LOCAL = "local-swap"
37 const val ID_2B = "local-2b" 38 const val ID_2B = "local-2b"
38 const val ID_4B = "local-4b" 39 const val ID_4B = "local-4b"
39 const val ID_HERMES = "hermes" 40 const val ID_HERMES = "hermes"
@@ -61,21 +62,20 @@ object ChatConfig { @@ -61,21 +62,20 @@ object ChatConfig {
61 .apply() 62 .apply()
62 } 63 }
63 64
64 - /** 当前可选的聊天对象列表。 */ 65 + /** 当前可选的聊天对象列表(本地 2B/4B + Hermes + 用户添加的远程服务商)。 */
65 fun targets(ctx: Context): List<ChatTarget> { 66 fun targets(ctx: Context): List<ChatTarget> {
66 val local = localBase(ctx) 67 val local = localBase(ctx)
67 - return listOf(  
68 - ChatTarget(ID_2B, "本地 2B", local, "", model2b(ctx), ChatTargetKind.LOCAL),  
69 - ChatTarget(ID_4B, "本地 4B", local, "", model4b(ctx), ChatTargetKind.LOCAL),  
70 - ChatTarget(  
71 - ID_HERMES,  
72 - "Hermes Agent",  
73 - BuildConfig.HERMES_API_URL.trimEnd('/'),  
74 - BuildConfig.HERMES_API_KEY,  
75 - BuildConfig.HERMES_MODEL,  
76 - ChatTargetKind.HERMES,  
77 - ),  
78 - ) 68 + val hermes = ModelConfig.hermesProvider(ctx)
  69 + return buildList {
  70 + add(ChatTarget(ID_2B, "本地 2B", local, "", model2b(ctx), ChatTargetKind.LOCAL))
  71 + add(ChatTarget(ID_4B, "本地 4B", local, "", model4b(ctx), ChatTargetKind.LOCAL))
  72 + add(ChatTarget(ID_HERMES, hermes.label, hermes.baseUrl, hermes.apiKey, hermes.chatModel, ChatTargetKind.HERMES))
  73 + ModelConfig.userProviders(ctx)
  74 + .filter { it.chatModel.isNotBlank() }
  75 + .forEach { p ->
  76 + add(ChatTarget(p.id, p.label, p.baseUrl, p.apiKey, p.chatModel, ChatTargetKind.REMOTE))
  77 + }
  78 + }
79 } 79 }
80 80
81 fun targetById(ctx: Context, id: String?): ChatTarget { 81 fun targetById(ctx: Context, id: String?): ChatTarget {
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/chat/ChatViewModel.kt
@@ -10,6 +10,8 @@ import androidx.lifecycle.viewModelScope @@ -10,6 +10,8 @@ import androidx.lifecycle.viewModelScope
10 import com.lemon.lifelog.data.ChatConversation 10 import com.lemon.lifelog.data.ChatConversation
11 import com.lemon.lifelog.data.ChatMessage 11 import com.lemon.lifelog.data.ChatMessage
12 import com.lemon.lifelog.data.Repository 12 import com.lemon.lifelog.data.Repository
  13 +import org.json.JSONArray
  14 +import org.json.JSONObject
13 import kotlinx.coroutines.Dispatchers 15 import kotlinx.coroutines.Dispatchers
14 import kotlinx.coroutines.launch 16 import kotlinx.coroutines.launch
15 import kotlinx.coroutines.withContext 17 import kotlinx.coroutines.withContext
@@ -33,6 +35,10 @@ class ChatViewModel(app: Application) : AndroidViewModel(app) { @@ -33,6 +35,10 @@ class ChatViewModel(app: Application) : AndroidViewModel(app) {
33 private set 35 private set
34 var lifelogMode by mutableStateOf(prefs.getBoolean(K_LIFELOG, false)) 36 var lifelogMode by mutableStateOf(prefs.getBoolean(K_LIFELOG, false))
35 private set 37 private set
  38 + var toolMode by mutableStateOf(prefs.getBoolean(K_TOOL, false))
  39 + private set
  40 + var toolStatus by mutableStateOf("")
  41 + private set
36 42
37 var input by mutableStateOf("") 43 var input by mutableStateOf("")
38 var streaming by mutableStateOf(false) 44 var streaming by mutableStateOf(false)
@@ -47,6 +53,13 @@ class ChatViewModel(app: Application) : AndroidViewModel(app) { @@ -47,6 +53,13 @@ class ChatViewModel(app: Application) : AndroidViewModel(app) {
47 prefs.edit().putBoolean(K_LIFELOG, on).apply() 53 prefs.edit().putBoolean(K_LIFELOG, on).apply()
48 } 54 }
49 55
  56 + fun setTool(on: Boolean) {
  57 + toolMode = on
  58 + prefs.edit().putBoolean(K_TOOL, on).apply()
  59 + // 工具模式和 lifelog 模式互斥
  60 + if (on && lifelogMode) setLifelog(false)
  61 + }
  62 +
50 fun selectTarget(id: String) { 63 fun selectTarget(id: String) {
51 selectedTargetId = id 64 selectedTargetId = id
52 prefs.edit().putString(K_TARGET, id).apply() 65 prefs.edit().putString(K_TARGET, id).apply()
@@ -90,8 +103,8 @@ class ChatViewModel(app: Application) : AndroidViewModel(app) { @@ -90,8 +103,8 @@ class ChatViewModel(app: Application) : AndroidViewModel(app) {
90 input = "" 103 input = ""
91 val app = getApplication<Application>() 104 val app = getApplication<Application>()
92 val target = ChatConfig.targetById(app, selectedTargetId) 105 val target = ChatConfig.targetById(app, selectedTargetId)
93 - val useLifelog = lifelogMode  
94 viewModelScope.launch { 106 viewModelScope.launch {
  107 + // ── 公共准备:建会话 + 存 user 消息 + 取历史 ──────────────────
95 val now = System.currentTimeMillis() 108 val now = System.currentTimeMillis()
96 var convId = currentConvId 109 var convId = currentConvId
97 if (convId == null) { 110 if (convId == null) {
@@ -108,48 +121,128 @@ class ChatViewModel(app: Application) : AndroidViewModel(app) { @@ -108,48 +121,128 @@ class ChatViewModel(app: Application) : AndroidViewModel(app) {
108 repo.chats.insertMessage(ChatMessage(conversationId = cid, role = "user", content = text, createdAt = now)) 121 repo.chats.insertMessage(ChatMessage(conversationId = cid, role = "user", content = text, createdAt = now))
109 repo.chats.touch(cid, now) 122 repo.chats.touch(cid, now)
110 } 123 }
111 - val history = withContext(Dispatchers.IO) { repo.chats.messagesOf(cid) }.map { it.role to it.content } 124 + val history = withContext(Dispatchers.IO) { repo.chats.messagesOf(cid) }
112 125
113 streaming = true 126 streaming = true
114 streamingText = "" 127 streamingText = ""
115 try { 128 try {
116 - val messagesToSend = if (useLifelog) {  
117 - val c = LifelogRetriever.buildContext(repo, text)  
118 - if (c != null) listOf("system" to c) + history else history 129 + if (toolMode) {
  130 + agenticLoop(target, cid, history)
119 } else { 131 } else {
120 - history  
121 - }  
122 - val full = client.stream(target, messagesToSend) { delta -> streamingText += delta }  
123 - withContext(Dispatchers.IO) {  
124 - repo.chats.insertMessage(  
125 - ChatMessage(  
126 - conversationId = cid, role = "assistant",  
127 - content = full.ifBlank { "(空回复)" }, model = target.model,  
128 - createdAt = System.currentTimeMillis(),  
129 - ),  
130 - )  
131 - repo.chats.touch(cid, System.currentTimeMillis()) 132 + normalChat(text, target, cid, history.map { it.role to it.content })
132 } 133 }
133 } catch (t: Throwable) { 134 } catch (t: Throwable) {
134 withContext(Dispatchers.IO) { 135 withContext(Dispatchers.IO) {
135 - repo.chats.insertMessage(  
136 - ChatMessage(  
137 - conversationId = cid, role = "assistant",  
138 - content = "⚠ 出错:${t.message ?: t.toString()}", model = target.model,  
139 - createdAt = System.currentTimeMillis(),  
140 - ),  
141 - ) 136 + repo.chats.insertMessage(ChatMessage(
  137 + conversationId = cid, role = "assistant",
  138 + content = "⚠ 出错:${t.message ?: t.toString()}", model = target.model,
  139 + createdAt = System.currentTimeMillis(),
  140 + ))
142 } 141 }
143 } finally { 142 } finally {
144 streaming = false 143 streaming = false
145 streamingText = "" 144 streamingText = ""
  145 + toolStatus = ""
  146 + }
  147 + }
  148 + }
  149 +
  150 + private suspend fun normalChat(
  151 + text: String,
  152 + target: ChatTarget,
  153 + cid: Long,
  154 + history: List<Pair<String, String>>,
  155 + ) {
  156 + val messagesToSend = if (lifelogMode) {
  157 + val c = LifelogRetriever.buildContext(repo, text)
  158 + if (c != null) listOf("system" to c) + history else history
  159 + } else {
  160 + history
  161 + }
  162 + val full = client.stream(target, messagesToSend) { delta -> streamingText += delta }
  163 + withContext(Dispatchers.IO) {
  164 + repo.chats.insertMessage(ChatMessage(
  165 + conversationId = cid, role = "assistant",
  166 + content = full.ifBlank { "(空回复)" }, model = target.model,
  167 + createdAt = System.currentTimeMillis(),
  168 + ))
  169 + repo.chats.touch(cid, System.currentTimeMillis())
  170 + }
  171 + }
  172 +
  173 + /**
  174 + * 工具模式 agentic loop:
  175 + * 1. 发带 tools 的请求 → 若 LLM 返回 tool_calls,本地执行 → 追加结果 → 再请求
  176 + * 2. 直到 LLM 不再调工具(给出最终文字)或达到最大轮数
  177 + * 3. 只把 user 消息和最终 assistant 文字存 Room(中间工具消息不持久化)
  178 + */
  179 + private suspend fun agenticLoop(
  180 + target: ChatTarget,
  181 + cid: Long,
  182 + history: List<ChatMessage>,
  183 + ) {
  184 + // 构建初始 messages(system prompt + 全部 Room 历史,已含刚存的 user 消息)
  185 + val messages: MutableList<JSONObject> = mutableListOf()
  186 + messages.add(JSONObject().put("role", "system").put("content", TOOL_SYSTEM_PROMPT))
  187 + history.forEach { m ->
  188 + messages.add(JSONObject().put("role", m.role).put("content", m.content))
  189 + }
  190 +
  191 + var finalText = ""
  192 + var rounds = 0
  193 + toolStatus = "思考中…"
  194 +
  195 + while (rounds < MAX_TOOL_ROUNDS) {
  196 + val result = withContext(Dispatchers.IO) {
  197 + client.chatWithTools(target, messages, LifelogTools.SCHEMAS)
  198 + }
  199 + if (result.toolCalls.isNullOrEmpty()) {
  200 + finalText = result.text
  201 + break
  202 + }
  203 + // assistant 消息(含 tool_calls,API 要求原样回传)
  204 + val assistantMsg = JSONObject().put("role", "assistant").put("content", result.text.ifBlank { "" })
  205 + result.rawToolCallsJson?.let { assistantMsg.put("tool_calls", it) }
  206 + messages.add(assistantMsg)
  207 +
  208 + // 逐个执行工具
  209 + for (call in result.toolCalls) {
  210 + toolStatus = "⚙ ${call.name}"
  211 + streamingText = "调用工具:${call.name}\n"
  212 + val args = runCatching { JSONObject(call.arguments) }.getOrElse { JSONObject() }
  213 + val toolResult = LifelogTools.execute(repo, call.name, args)
  214 + streamingText += toolResult.take(200)
  215 + messages.add(JSONObject()
  216 + .put("role", "tool")
  217 + .put("tool_call_id", call.id)
  218 + .put("content", toolResult))
146 } 219 }
  220 + rounds++
  221 + toolStatus = "整理回复中(第 $rounds 轮)…"
  222 + streamingText = toolStatus
  223 + }
  224 +
  225 + if (finalText.isBlank()) finalText = "(未获得有效回复,请重试或检查服务商是否支持 function calling)"
  226 + withContext(Dispatchers.IO) {
  227 + repo.chats.insertMessage(ChatMessage(
  228 + conversationId = cid, role = "assistant",
  229 + content = finalText, model = target.model,
  230 + createdAt = System.currentTimeMillis(),
  231 + ))
  232 + repo.chats.touch(cid, System.currentTimeMillis())
147 } 233 }
148 } 234 }
149 235
150 companion object { 236 companion object {
151 private const val K_LIFELOG = "lifelog_mode" 237 private const val K_LIFELOG = "lifelog_mode"
  238 + private const val K_TOOL = "tool_mode"
152 private const val K_TARGET = "selected_target" 239 private const val K_TARGET = "selected_target"
153 - private const val K_CONV = "current_conv" 240 + private const val K_CONV = "current_conv"
  241 + private const val MAX_TOOL_ROUNDS = 8
  242 + private val TOOL_SYSTEM_PROMPT = """
  243 + 你是用户的 lifelog AI 助手,可以访问用户 24/7 录音转写、会话纪要、联系人档案和相册。
  244 + 当用户询问记忆、联系人或照片时,请主动调用相应工具获取真实数据后再回答。
  245 + 多步骤问题可多次调用工具。最终回复用中文,简洁清晰。
  246 + """.trimIndent()
154 } 247 }
155 } 248 }
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/chat/LifelogTools.kt 0 → 100644
  1 +package com.lemon.lifelog.chat
  2 +
  3 +import com.lemon.lifelog.data.Repository
  4 +import kotlinx.coroutines.Dispatchers
  5 +import kotlinx.coroutines.flow.first
  6 +import kotlinx.coroutines.withContext
  7 +import org.json.JSONArray
  8 +import org.json.JSONObject
  9 +import java.text.SimpleDateFormat
  10 +import java.util.Date
  11 +import java.util.Locale
  12 +
  13 +/**
  14 + * lifelog 工具集:OpenAI function calling schema + 本地执行。
  15 + *
  16 + * 所有工具直接查 Room DB,在 app 进程内执行,无需网络/额外服务。
  17 + * agenticLoop(ChatViewModel)检测到 tool_calls → 调 execute() → 把结果以 tool role 追回给 LLM。
  18 + *
  19 + * 当前 6 个工具:
  20 + * search_transcripts 搜录音转写片段
  21 + * get_person 查联系人档案
  22 + * list_persons 列 CRM 人物
  23 + * get_summaries 取会话纪要
  24 + * search_photos 关键词搜相册
  25 + * get_followups 待跟进事项
  26 + */
  27 +object LifelogTools {
  28 +
  29 + private val TS = SimpleDateFormat("MM-dd HH:mm", Locale.US)
  30 + private val TSL = SimpleDateFormat("MM-dd HH:mm:ss", Locale.US)
  31 +
  32 + // ── Schema(OpenAI function calling 格式)────────────────────────────────
  33 +
  34 + val SCHEMAS: JSONArray by lazy {
  35 + JSONArray().apply {
  36 + put(schema(
  37 + name = "search_transcripts",
  38 + desc = "在录音转写片段中搜索关键词,返回匹配段落(时间、说话人、内容)",
  39 + props = mapOf(
  40 + "query" to prop("string", "搜索关键词"),
  41 + "speaker" to prop("string", "限定说话人姓名,可选"),
  42 + "limit" to prop("integer", "返回条数,默认 12"),
  43 + ),
  44 + required = listOf("query"),
  45 + ))
  46 + put(schema(
  47 + name = "get_person",
  48 + desc = "按姓名或备注查找联系人档案(基本资料、AI 印象、简历、关联声纹/人脸)",
  49 + props = mapOf("name" to prop("string", "人物姓名或备注名")),
  50 + required = listOf("name"),
  51 + ))
  52 + put(schema(
  53 + name = "list_persons",
  54 + desc = "列出 CRM 中的联系人(默认只列我管理的人)",
  55 + props = mapOf(
  56 + "managed_only" to prop("boolean", "true=只看管理的人(默认),false=全部通讯录"),
  57 + ),
  58 + required = emptyList(),
  59 + ))
  60 + put(schema(
  61 + name = "get_summaries",
  62 + desc = "获取最近的会话纪要(LLM 自动生成的录音摘要,时间倒序)",
  63 + props = mapOf("limit" to prop("integer", "返回篇数,默认 5,最多 20")),
  64 + required = emptyList(),
  65 + ))
  66 + put(schema(
  67 + name = "search_photos",
  68 + desc = "在相册中按关键词或人名搜索照片(需要 VL 索引完成后才有描述内容)",
  69 + props = mapOf(
  70 + "query" to prop("string", "关键词或人名"),
  71 + "limit" to prop("integer", "返回张数,默认 8"),
  72 + ),
  73 + required = listOf("query"),
  74 + ))
  75 + put(schema(
  76 + name = "get_followups",
  77 + desc = "获取待跟进事项列表(从录音中 AI 抽取的 TODO/行动项,未完成在前)",
  78 + props = emptyMap(),
  79 + required = emptyList(),
  80 + ))
  81 + }
  82 + }
  83 +
  84 + private fun schema(name: String, desc: String, props: Map<String, JSONObject>, required: List<String>) =
  85 + JSONObject().apply {
  86 + put("type", "function")
  87 + put("function", JSONObject().apply {
  88 + put("name", name)
  89 + put("description", desc)
  90 + put("parameters", JSONObject().apply {
  91 + put("type", "object")
  92 + put("properties", JSONObject().apply { props.forEach { (k, v) -> put(k, v) } })
  93 + put("required", JSONArray().apply { required.forEach { put(it) } })
  94 + })
  95 + })
  96 + }
  97 +
  98 + private fun prop(type: String, description: String) =
  99 + JSONObject().put("type", type).put("description", description)
  100 +
  101 + // ── 执行(在 IO 线程调)──────────────────────────────────────────────────
  102 +
  103 + suspend fun execute(repo: Repository, name: String, args: JSONObject): String =
  104 + withContext(Dispatchers.IO) {
  105 + try {
  106 + when (name) {
  107 + "search_transcripts" -> searchTranscripts(repo, args)
  108 + "get_person" -> getPerson(repo, args)
  109 + "list_persons" -> listPersons(repo, args)
  110 + "get_summaries" -> getSummaries(repo, args)
  111 + "search_photos" -> searchPhotos(repo, args)
  112 + "get_followups" -> getFollowups(repo)
  113 + else -> "未知工具:$name"
  114 + }
  115 + } catch (t: Throwable) {
  116 + "工具执行出错($name):${t.message?.take(120)}"
  117 + }
  118 + }
  119 +
  120 + // ── 各工具实现 ────────────────────────────────────────────────────────────
  121 +
  122 + private suspend fun searchTranscripts(repo: Repository, args: JSONObject): String {
  123 + val query = args.optString("query", "").ifBlank { return "请提供搜索关键词" }
  124 + val speaker = args.optString("speaker", "").ifBlank { null }
  125 + val limit = args.optInt("limit", 12).coerceIn(1, 40)
  126 + val rows = repo.segments.searchTextLike(query, limit * 3)
  127 + .filter { it.text != null }
  128 + .let { list -> if (speaker != null) list.filter { it.speakerName?.contains(speaker) == true } else list }
  129 + .take(limit)
  130 + if (rows.isEmpty()) return "未找到包含「$query」的录音片段。"
  131 + return buildString {
  132 + appendLine("找到 ${rows.size} 条片段(关键词:「$query」):")
  133 + rows.forEach { r ->
  134 + val who = r.speakerName ?: r.personName ?: "未知"
  135 + appendLine("[${TSL.format(Date(r.startTs))}] $who:${r.text?.trim()}")
  136 + }
  137 + }
  138 + }
  139 +
  140 + private suspend fun getPerson(repo: Repository, args: JSONObject): String {
  141 + val q = args.optString("name", "").ifBlank { return "请提供人物姓名" }
  142 + val persons = repo.persons.all().filter { p ->
  143 + p.name.contains(q) || p.realName?.contains(q) == true ||
  144 + p.nickname?.contains(q) == true || p.remark?.contains(q) == true
  145 + }
  146 + if (persons.isEmpty()) return "未找到与「$q」匹配的联系人。"
  147 + return buildString {
  148 + persons.take(3).forEach { p ->
  149 + appendLine("=== ${p.name} ===")
  150 + p.realName?.takeIf { it.isNotBlank() }?.let { appendLine("真实姓名:$it") }
  151 + p.nickname?.takeIf { it.isNotBlank() }?.let { appendLine("微信昵称:$it") }
  152 + p.remark?.takeIf { it.isNotBlank() }?.let { appendLine("备注:$it") }
  153 + p.phone?.takeIf { it.isNotBlank() }?.let { appendLine("手机:$it") }
  154 + if (p.wechatFriend == true) appendLine("微信好友:是")
  155 + p.impression?.takeIf { it.isNotBlank() }?.let { appendLine("AI 印象:${it.take(400)}") }
  156 + p.resume?.takeIf { it.isNotBlank() }?.let { appendLine("简历:${it.take(300)}") }
  157 + val tags = buildList {
  158 + if (p.managed) add("CRM 管理中")
  159 + if (p.voiceSpeakerId != null) add("有声纹")
  160 + if (p.faceEmbedding != null) add("有人脸")
  161 + }.joinToString(" / ")
  162 + if (tags.isNotBlank()) appendLine("标签:$tags")
  163 + appendLine()
  164 + }
  165 + }
  166 + }
  167 +
  168 + private suspend fun listPersons(repo: Repository, args: JSONObject): String {
  169 + val managedOnly = args.optBoolean("managed_only", true)
  170 + val persons = repo.persons.all().let { if (managedOnly) it.filter { p -> p.managed } else it }
  171 + if (persons.isEmpty()) return if (managedOnly) "CRM 中暂无管理的人物。" else "通讯录为空。"
  172 + return buildString {
  173 + appendLine("共 ${persons.size} 位${if (managedOnly) "(管理中)" else ""}:")
  174 + persons.forEach { p ->
  175 + val tags = buildList {
  176 + if (p.wechatFriend == true) add("微信")
  177 + if (p.voiceSpeakerId != null) add("声纹")
  178 + if (p.faceEmbedding != null) add("人脸")
  179 + if (!p.phone.isNullOrBlank()) add("手机")
  180 + }.joinToString("/")
  181 + appendLine("- ${p.name}${if (tags.isNotEmpty()) "($tags)" else ""}")
  182 + }
  183 + }
  184 + }
  185 +
  186 + private suspend fun getSummaries(repo: Repository, args: JSONObject): String {
  187 + val limit = args.optInt("limit", 5).coerceIn(1, 20)
  188 + val list = repo.summaries.recentList(limit)
  189 + if (list.isEmpty()) return "暂无会话纪要(需先生成)。"
  190 + return buildString {
  191 + appendLine("最近 ${list.size} 篇纪要:")
  192 + list.forEach { s ->
  193 + val range = "${TS.format(Date(s.windowStartTs))}–${TS.format(Date(s.windowEndTs))}"
  194 + appendLine("\n【$range】")
  195 + appendLine(s.markdown.trim().take(700))
  196 + }
  197 + }
  198 + }
  199 +
  200 + private suspend fun searchPhotos(repo: Repository, args: JSONObject): String {
  201 + val query = args.optString("query", "").ifBlank { return "请提供搜索关键词" }
  202 + val limit = args.optInt("limit", 8).coerceIn(1, 20)
  203 + val photos = repo.photos.searchByKeyword(query, limit)
  204 + if (photos.isEmpty()) return "未找到与「$query」相关的照片(VL 描述索引可能尚未完成)。"
  205 + return buildString {
  206 + appendLine("找到 ${photos.size} 张相关照片(关键词:「$query」):")
  207 + photos.forEach { p ->
  208 + val date = TS.format(Date(p.takenTs))
  209 + val caption = p.caption?.take(100) ?: "(暂无场景描述)"
  210 + val faces = if (p.faceCount > 0) "${p.faceCount}人" else "无人脸"
  211 + appendLine("[$date $faces] $caption")
  212 + }
  213 + }
  214 + }
  215 +
  216 + private suspend fun getFollowups(repo: Repository): String {
  217 + val items = repo.followups.observeAll().first()
  218 + if (items.isEmpty()) return "暂无待跟进事项。"
  219 + val open = items.filter { !it.done }
  220 + val done = items.filter { it.done }
  221 + return buildString {
  222 + if (open.isNotEmpty()) {
  223 + appendLine("未完成(${open.size} 条):")
  224 + open.forEach { f ->
  225 + val who = f.personName?.let { "($it)" } ?: ""
  226 + appendLine("- ${f.text}$who [${TS.format(Date(f.sourceTs))}]")
  227 + }
  228 + }
  229 + if (done.isNotEmpty()) {
  230 + appendLine("\n已完成(${done.size} 条,最近 5 条):")
  231 + done.take(5).forEach { f -> appendLine("- ✓ ${f.text}") }
  232 + }
  233 + }
  234 + }
  235 +}
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/chat/ModelConfig.kt 0 → 100644
  1 +package com.lemon.lifelog.chat
  2 +
  3 +import android.content.Context
  4 +import com.lemon.lifelog.BuildConfig
  5 +import org.json.JSONArray
  6 +import org.json.JSONObject
  7 +import java.util.UUID
  8 +
  9 +data class LlmProvider(
  10 + val id: String,
  11 + val label: String,
  12 + val baseUrl: String,
  13 + val apiKey: String,
  14 + val chatModel: String,
  15 + val builtIn: Boolean = false,
  16 +)
  17 +
  18 +enum class ModelSlot(val key: String, val label: String, val defaultModel: String) {
  19 + VL("slot_vl", "图像理解(VL)", "vl"),
  20 + EMBED("slot_embed", "语义向量", "embed"),
  21 +}
  22 +
  23 +/**
  24 + * 统一服务商注册表 + 任务槽绑定。
  25 + *
  26 + * 服务商(LlmProvider)= 一个 OpenAI 兼容端点(baseUrl + apiKey)+ 对话默认模型名。
  27 + * 内置两个:本地 llama-swap (ID_LOCAL) 和 本地 Hermes (ID_HERMES),不可删除。
  28 + * 用户可添加任意远程服务商(通义千问 / Kimi / MiniMax 等)。
  29 + *
  30 + * 任务槽(ModelSlot)= 某项能力用哪个 (provider, model):
  31 + * - VL:图像理解,PhotoVlWorker 读取
  32 + * - EMBED:语义向量,PhotologViewModel / PhotoVlWorker 读取
  33 + * - CHAT:用哪个 target 聊天 → 由 ChatConfig.targets() + ChatViewModel.selectedTargetId 管理,不在此
  34 + * - SUMMARY:纪要用哪个 target → 同上,存 SummaryWorker.KEY_BACKEND(ChatTarget.id)
  35 + *
  36 + * 持久化:SharedPreferences "model_config",JSON 数组 + 独立 key,无 Room 依赖。
  37 + */
  38 +object ModelConfig {
  39 + const val PREFS = "model_config"
  40 + private const val K_PROVIDERS = "user_providers"
  41 + private const val K_HERMES_URL = "hermes_url"
  42 + private const val K_HERMES_KEY = "hermes_key"
  43 + private const val K_HERMES_MODEL = "hermes_model"
  44 +
  45 + private fun prefs(ctx: Context) = ctx.getSharedPreferences(PREFS, Context.MODE_PRIVATE)
  46 +
  47 + // ── 内置:Hermes 网关 ──────────────────────────────────────────────────────
  48 +
  49 + fun hermesProvider(ctx: Context): LlmProvider {
  50 + val p = prefs(ctx)
  51 + return LlmProvider(
  52 + id = ChatConfig.ID_HERMES,
  53 + label = "本地 Hermes",
  54 + baseUrl = p.getString(K_HERMES_URL, null)?.ifBlank { null } ?: BuildConfig.HERMES_API_URL,
  55 + apiKey = p.getString(K_HERMES_KEY, null) ?: BuildConfig.HERMES_API_KEY,
  56 + chatModel = p.getString(K_HERMES_MODEL, null)?.ifBlank { null } ?: BuildConfig.HERMES_MODEL,
  57 + builtIn = true,
  58 + )
  59 + }
  60 +
  61 + fun saveHermes(ctx: Context, url: String, key: String, model: String) {
  62 + prefs(ctx).edit()
  63 + .putString(K_HERMES_URL, url.trim())
  64 + .putString(K_HERMES_KEY, key.trim())
  65 + .putString(K_HERMES_MODEL, model.trim())
  66 + .apply()
  67 + }
  68 +
  69 + // ── 内置:本地 llama-swap ─────────────────────────────────────────────────
  70 +
  71 + fun localSwapProvider(ctx: Context) = LlmProvider(
  72 + id = ChatConfig.ID_LOCAL,
  73 + label = "本地 llama-swap",
  74 + baseUrl = ChatConfig.localBase(ctx),
  75 + apiKey = "",
  76 + chatModel = "",
  77 + builtIn = true,
  78 + )
  79 +
  80 + // ── 用户添加的远程服务商 ───────────────────────────────────────────────────
  81 +
  82 + fun userProviders(ctx: Context): List<LlmProvider> {
  83 + val json = prefs(ctx).getString(K_PROVIDERS, null) ?: return emptyList()
  84 + return try {
  85 + val arr = JSONArray(json)
  86 + (0 until arr.length()).map { arr.getJSONObject(it).toProvider() }
  87 + } catch (t: Throwable) {
  88 + emptyList()
  89 + }
  90 + }
  91 +
  92 + fun allProviders(ctx: Context): List<LlmProvider> = buildList {
  93 + add(localSwapProvider(ctx))
  94 + add(hermesProvider(ctx))
  95 + addAll(userProviders(ctx))
  96 + }
  97 +
  98 + fun addProvider(ctx: Context, p: LlmProvider) {
  99 + saveUserProviders(ctx, userProviders(ctx) + p)
  100 + }
  101 +
  102 + fun updateProvider(ctx: Context, p: LlmProvider) {
  103 + if (p.id == ChatConfig.ID_HERMES) {
  104 + saveHermes(ctx, p.baseUrl, p.apiKey, p.chatModel)
  105 + return
  106 + }
  107 + if (p.builtIn) return
  108 + val list = userProviders(ctx).toMutableList()
  109 + val idx = list.indexOfFirst { it.id == p.id }
  110 + if (idx >= 0) list[idx] = p else list.add(p)
  111 + saveUserProviders(ctx, list)
  112 + }
  113 +
  114 + fun deleteProvider(ctx: Context, id: String) {
  115 + if (id == ChatConfig.ID_LOCAL || id == ChatConfig.ID_HERMES) return
  116 + saveUserProviders(ctx, userProviders(ctx).filter { it.id != id })
  117 + // 如果某个 slot 引用了被删的 provider,重置回默认
  118 + for (slot in ModelSlot.entries) {
  119 + if (slotProviderId(ctx, slot) == id) {
  120 + setSlot(ctx, slot, ChatConfig.ID_LOCAL, slot.defaultModel)
  121 + }
  122 + }
  123 + }
  124 +
  125 + private fun saveUserProviders(ctx: Context, list: List<LlmProvider>) {
  126 + val arr = JSONArray()
  127 + list.forEach { arr.put(it.toJson()) }
  128 + prefs(ctx).edit().putString(K_PROVIDERS, arr.toString()).apply()
  129 + }
  130 +
  131 + // ── 任务槽 ────────────────────────────────────────────────────────────────
  132 +
  133 + fun slotProviderId(ctx: Context, slot: ModelSlot): String =
  134 + prefs(ctx).getString("${slot.key}_pid", null)?.ifBlank { null } ?: ChatConfig.ID_LOCAL
  135 +
  136 + fun slotModel(ctx: Context, slot: ModelSlot): String =
  137 + prefs(ctx).getString("${slot.key}_model", null)?.ifBlank { null } ?: slot.defaultModel
  138 +
  139 + fun setSlot(ctx: Context, slot: ModelSlot, providerId: String, model: String) {
  140 + prefs(ctx).edit()
  141 + .putString("${slot.key}_pid", providerId)
  142 + .putString("${slot.key}_model", model.trim())
  143 + .apply()
  144 + }
  145 +
  146 + /** (baseUrl, apiKey, modelName) 三元组,供 VisionClient / EmbedClient 直接使用。 */
  147 + fun slotEndpoint(ctx: Context, slot: ModelSlot): Triple<String, String, String> {
  148 + val pid = slotProviderId(ctx, slot)
  149 + val model = slotModel(ctx, slot)
  150 + val provider = allProviders(ctx).firstOrNull { it.id == pid } ?: localSwapProvider(ctx)
  151 + return Triple(provider.baseUrl, provider.apiKey, model)
  152 + }
  153 +
  154 + fun generateId(): String = "p_${UUID.randomUUID().toString().take(8)}"
  155 +
  156 + // ── JSON 序列化 ────────────────────────────────────────────────────────────
  157 +
  158 + private fun JSONObject.toProvider() = LlmProvider(
  159 + id = getString("id"),
  160 + label = getString("label"),
  161 + baseUrl = getString("baseUrl"),
  162 + apiKey = optString("apiKey", ""),
  163 + chatModel = optString("chatModel", ""),
  164 + )
  165 +
  166 + private fun LlmProvider.toJson() = JSONObject().apply {
  167 + put("id", id)
  168 + put("label", label)
  169 + put("baseUrl", baseUrl)
  170 + put("apiKey", apiKey)
  171 + put("chatModel", chatModel)
  172 + }
  173 +}
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/data/Daos.kt
@@ -501,6 +501,9 @@ data class PhotoVec( @@ -501,6 +501,9 @@ data class PhotoVec(
501 val captionEmbedding: ByteArray, 501 val captionEmbedding: ByteArray,
502 ) 502 )
503 503
  504 +/** 补算 embedding 用的轻投影:已有 caption 但缺 embedding 的照片。 */
  505 +data class CaptionOnly(val id: Long, val caption: String)
  506 +
504 /** 某人物名下照片数(相册「人物文件夹」封面 + 计数)。 */ 507 /** 某人物名下照片数(相册「人物文件夹」封面 + 计数)。 */
505 data class PersonPhotoCount( 508 data class PersonPhotoCount(
506 val personId: Long, 509 val personId: Long,
@@ -749,6 +752,14 @@ interface PhotoDao { @@ -749,6 +752,14 @@ interface PhotoDao {
749 /** 按 path 模糊查 id(清理旧版误导入的朋友圈图用,pat 形如 "%/moment_<wxid>_%")。 */ 752 /** 按 path 模糊查 id(清理旧版误导入的朋友圈图用,pat 形如 "%/moment_<wxid>_%")。 */
750 @Query("SELECT id FROM photos WHERE path LIKE :pat") 753 @Query("SELECT id FROM photos WHERE path LIKE :pat")
751 suspend fun idsByPathLike(pat: String): List<Long> 754 suspend fun idsByPathLike(pat: String): List<Long>
  755 +
  756 + /** 补算向量:已有 caption 但还没算 embedding 的照片(embed 模型后配时回填用)。 */
  757 + @Query("SELECT id, caption FROM photos WHERE caption IS NOT NULL AND caption != '' AND captionEmbedding IS NULL")
  758 + suspend fun withCaptionNoEmbedding(): List<CaptionOnly>
  759 +
  760 + /** 只写 captionEmbedding,不改其他字段(backfill 用,不重跑 VL)。 */
  761 + @Query("UPDATE photos SET captionEmbedding = :emb WHERE id = :id")
  762 + suspend fun updateCaptionEmbedding(id: Long, emb: ByteArray)
752 } 763 }
753 764
754 @Dao 765 @Dao
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/vision/EmbedClient.kt
@@ -22,6 +22,7 @@ import kotlin.math.sqrt @@ -22,6 +22,7 @@ import kotlin.math.sqrt
22 class EmbedClient( 22 class EmbedClient(
23 private val baseUrl: String, 23 private val baseUrl: String,
24 private val model: String = "embed", 24 private val model: String = "embed",
  25 + private val apiKey: String = "",
25 ) { 26 ) {
26 companion object { 27 companion object {
27 private val JSON = "application/json; charset=utf-8".toMediaType() 28 private val JSON = "application/json; charset=utf-8".toMediaType()
@@ -38,11 +39,12 @@ class EmbedClient( @@ -38,11 +39,12 @@ class EmbedClient(
38 put("model", model) 39 put("model", model)
39 put("input", text) 40 put("input", text)
40 } 41 }
41 - val req = Request.Builder() 42 + val reqBuilder = Request.Builder()
42 .url("$baseUrl/v1/embeddings") 43 .url("$baseUrl/v1/embeddings")
43 .header("Content-Type", "application/json") 44 .header("Content-Type", "application/json")
44 .post(payload.toString().toRequestBody(JSON)) 45 .post(payload.toString().toRequestBody(JSON))
45 - .build() 46 + if (apiKey.isNotBlank()) reqBuilder.header("Authorization", "Bearer $apiKey")
  47 + val req = reqBuilder.build()
46 http.newCall(req).execute().use { resp -> 48 http.newCall(req).execute().use { resp ->
47 val body = resp.body?.string().orEmpty() 49 val body = resp.body?.string().orEmpty()
48 if (!resp.isSuccessful) throw IOException("embed ${resp.code}: ${body.take(200)}") 50 if (!resp.isSuccessful) throw IOException("embed ${resp.code}: ${body.take(200)}")
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/vision/PhotoVlWorker.kt
@@ -4,7 +4,8 @@ import android.content.Context @@ -4,7 +4,8 @@ import android.content.Context
4 import android.util.Log 4 import android.util.Log
5 import androidx.work.CoroutineWorker 5 import androidx.work.CoroutineWorker
6 import androidx.work.WorkerParameters 6 import androidx.work.WorkerParameters
7 -import com.lemon.lifelog.chat.ChatConfig 7 +import com.lemon.lifelog.chat.ModelConfig
  8 +import com.lemon.lifelog.chat.ModelSlot
8 import com.lemon.lifelog.data.Repository 9 import com.lemon.lifelog.data.Repository
9 import java.util.concurrent.atomic.AtomicBoolean 10 import java.util.concurrent.atomic.AtomicBoolean
10 11
@@ -28,9 +29,10 @@ class PhotoVlWorker(ctx: Context, params: WorkerParameters) : CoroutineWorker(ct @@ -28,9 +29,10 @@ class PhotoVlWorker(ctx: Context, params: WorkerParameters) : CoroutineWorker(ct
28 val pending = repo.photos.pendingCaption() 29 val pending = repo.photos.pendingCaption()
29 if (pending.isEmpty()) return Result.success() 30 if (pending.isEmpty()) return Result.success()
30 31
31 - val base = ChatConfig.localBase(applicationContext)  
32 - val vision = VisionClient(base)  
33 - val embed = EmbedClient(base, ChatConfig.modelEmbed(applicationContext)) 32 + val (vlBase, vlKey, vlModel) = ModelConfig.slotEndpoint(applicationContext, ModelSlot.VL)
  33 + val (embedBase, embedKey, embedModel) = ModelConfig.slotEndpoint(applicationContext, ModelSlot.EMBED)
  34 + val vision = VisionClient(vlBase, vlModel, vlKey)
  35 + val embed = EmbedClient(embedBase, embedModel, embedKey)
34 val proc = PhotoProcessor(applicationContext, repo) 36 val proc = PhotoProcessor(applicationContext, repo)
35 val deadline = System.currentTimeMillis() + RUN_BUDGET_MS 37 val deadline = System.currentTimeMillis() + RUN_BUDGET_MS
36 var done = 0 38 var done = 0
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/vision/VisionClient.kt
@@ -17,6 +17,7 @@ import java.util.concurrent.TimeUnit @@ -17,6 +17,7 @@ import java.util.concurrent.TimeUnit
17 class VisionClient( 17 class VisionClient(
18 private val baseUrl: String, 18 private val baseUrl: String,
19 private val model: String = "vl", 19 private val model: String = "vl",
  20 + private val apiKey: String = "",
20 ) { 21 ) {
21 companion object { 22 companion object {
22 private val JSON = "application/json; charset=utf-8".toMediaType() 23 private val JSON = "application/json; charset=utf-8".toMediaType()
@@ -48,11 +49,12 @@ class VisionClient( @@ -48,11 +49,12 @@ class VisionClient(
48 }) 49 })
49 })) 50 }))
50 } 51 }
51 - val req = Request.Builder() 52 + val reqBuilder = Request.Builder()
52 .url("$baseUrl/v1/chat/completions") 53 .url("$baseUrl/v1/chat/completions")
53 .header("Content-Type", "application/json") 54 .header("Content-Type", "application/json")
54 .post(payload.toString().toRequestBody(JSON)) 55 .post(payload.toString().toRequestBody(JSON))
55 - .build() 56 + if (apiKey.isNotBlank()) reqBuilder.header("Authorization", "Bearer $apiKey")
  57 + val req = reqBuilder.build()
56 http.newCall(req).execute().use { resp -> 58 http.newCall(req).execute().use { resp ->
57 val body = resp.body?.string().orEmpty() 59 val body = resp.body?.string().orEmpty()
58 if (!resp.isSuccessful) throw IOException("vl ${resp.code}: ${body.take(200)}") 60 if (!resp.isSuccessful) throw IOException("vl ${resp.code}: ${body.take(200)}")