diff --git a/CLAUDE.md b/CLAUDE.md index 8dfdf21..b60757b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -140,12 +140,12 @@ Foreground Service (foregroundServiceType=microphone, WAKE_LOCK partial) 完整待办、估时、技术路线见 `lifelog-recorder/HANDOFF.md` §五。 -✅ **v0.2–v0.8 已完成**:Opus 编码 / BOOT 自启 + 电池白名单 / FTS4 中文搜索 / 段播放 / **声纹在线聚类** / **会话级纪要(按时间间隔切会话)** / **底部 4-tab + 子 tab UI** / **App 内对话(本地 2B/4B + Hermes)** / **本地 LLM 栈(llama-swap)** / **Room 迁移(去 destructive)** / **block-based diarization-first 录音管线(v0.7,攒块→整块分轮→逐 turn 转写)** / **声纹纠偏 L1 冻结 anchor + L2 多 anchor 重建** / **设置页(`SettingsScreen`:纪要模型后端 / 聚类阈值 / llama-swap 端点 / 电池豁免)** / **photolog 人脉 CRM 层(人物档案 + 相册人脸 + 微信通讯录/朋友圈/AI 印象)** / **存储 TTL(超期删 .opus,文字永留)** / **全相册自动索引(v0.8:MediaStore 全量扫描 + Tier-0 人脸/Tier-1 VL 分层管线 + 关键词/语义搜图 + VL 暂停/停止控制,Room v14)**。 +✅ **v0.2–v0.9 已完成**:Opus 编码 / BOOT 自启 + 电池白名单 / FTS4 中文搜索 / 段播放 / **声纹在线聚类** / **会话级纪要(按时间间隔切会话)** / **底部 4-tab + 子 tab UI** / **App 内对话(本地 2B/4B + Hermes)** / **本地 LLM 栈(llama-swap)** / **Room 迁移(去 destructive)** / **block-based diarization-first 录音管线(v0.7,攒块→整块分轮→逐 turn 转写)** / **声纹纠偏 L1 冻结 anchor + L2 多 anchor 重建** / **设置页(`SettingsScreen`:纪要模型后端 / 聚类阈值 / llama-swap 端点 / 电池豁免)** / **photolog 人脉 CRM 层(人物档案 + 相册人脸 + 微信通讯录/朋友圈/AI 印象)** / **存储 TTL(超期删 .opus,文字永留)** / **全相册自动索引(v0.8:MediaStore 全量扫描 + Tier-0 人脸/Tier-1 VL 分层管线 + 关键词/语义搜图 + VL 暂停/停止控制,Room v14)** / **微信抽取完全端上化(v0.8 P2 #6:SQLCipher key 端上推导 + 联系人/头像/聊天/朋友圈全不依赖 Mac)** / **Transcript→Person 提及关系(v0.9 P0 #5:词典反查 + MentionScanner 实时扫 + mentions 表 + 「被提到」tab)** / **人↔人共现边(v0.9 P1 #1:RelationBuilder + relations 表 + 「相关」tab + CommunitiesScreen)** / **待跟进 & 话题实体(FollowupExtractor + TopicExtractor,4B LLM 抽取)**。 剩余摘要: -- ⭐ **语义搜图落地**(v0.8 已接 `EmbedClient`,但需在 Pixel llama-swap 配 `embed` 模型才生效;中文 embedding GGUF) -- ⭐ **内部 Transcript 实体 → Person 表**(端上 NER + 共现,取代旧「回链 lemonPCRM」;见下文 §下一步) -- ⭐ **微信抽取端上化**(消除 photolog 仅剩的 Mac 依赖;见下文差距分析 #6) +- ⭐ **语义搜图落地**(代码已完成:`EmbedClient` + `SettingsScreen` embed 字段;**需在 Pixel llama-swap config.yaml 加 embed 模型**才生效,中文 GGUF 如 bge-small-zh,参见 HANDOFF.md §七 调试陷阱) +- ✅ **内部 Transcript 实体 → Person 表**(已完成:`crm/MentionExtractor.kt` + `MentionScanner` 词典反查 + `BlockProcessor` 实时扫 + `PersonProfileScreen` 「被提到」tab + `RelationBuilder` 共现边) +- ✅ **微信抽取端上化**(已完成 2026-06-10:SQLCipher 端上解密 + 联系人/头像/朋友圈全端上,不依赖 Mac) ### App 即 Personal CRM:与旧 lemonPCRM 的能力对比 diff --git a/lifelog-recorder/HANDOFF.md b/lifelog-recorder/HANDOFF.md index 4551708..f8f0c58 100644 --- a/lifelog-recorder/HANDOFF.md +++ b/lifelog-recorder/HANDOFF.md @@ -289,33 +289,29 @@ key = MD5(imei + uin).hex()[:7].lower() - v0.5 (2026-06):hermes api_server 端点 + SummaryWorker + 纪要 UI section - v0.6 (2026-06-04):**声纹在线聚类**(SpeakerClusterer,自动「说话人N」+ reclusterAll)/ **5-tab UI**(录音/对话/时间轴/声纹/纪要,时间轴全 CRUD + 批量删导出 + 按说话人/时间/未匹配筛选 + 排序)/ **真暂停**(停采集留模型)/ **会话级纪要**(按时间间隔切会话、可调阈值、要点带 [HH:mm:ss] 点击跳回时间轴、折叠头显示说话人名单)/ **App 内对话**(本地 2B/4B + Hermes 多端点流式)/ **本地 LLM 栈**(llama.cpp HEAD + llama-swap 8080,纪要+对话全本地,见 §六)/ **Room v3 迁移**(去 destructive 改 fail-loud) - v0.7 (2026-06-07):**block-based diarization-first 录音管线**(攒块→整块 pyannote 分轮→逐 turn 转写+嵌入+认人,turn=时间轴一行;去实时字幕;BlockBuffer/BlockProcessor;崩溃 listPending 续跑;commit 07f3d22)/ **声纹纠偏 L1+L2**(L1=reclusterAll & assign 冻结命名 anchor 不漂移止串人;L2=重命名/合并时 rebuildAnchor 用该人全部段均值重建多 anchor)/ 阈值默认 0.45、设置页 0.35-0.65 可调 / **photolog 人脉 CRM 层**(人物档案 + 相册人脸聚类 + 微信通讯录/朋友圈/AI 印象) -- v0.8 (2026-06-09):**全相册自动索引 + 分层管线**(GalleryScanner 扫 MediaStore 全量、content uri 原地登记不拷文件;Tier-0 人脸/时间秒级全量铺、Tier-1 VL 充电时排队补;Room v14;详见 §四 photolog 段)/ **搜图**(关键词 caption+人名 LIKE / 语义 caption 向量余弦,embed 走 llama-swap)/ **VL 控制**(PhotoVlState:暂停/继续/停止切换 + 单张转圈 + 格子状态角标) +- v0.8 (2026-06-09):**全相册自动索引 + 分层管线**(GalleryScanner 扫 MediaStore 全量、content uri 原地登记不拷文件;Tier-0 人脸/时间秒级全量铺、Tier-1 VL 充电时排队补;Room v14;详见 §四 photolog 段)/ **搜图**(关键词 caption+人名 LIKE / 语义 caption 向量余弦,embed 走 llama-swap)/ **VL 控制**(PhotoVlState:暂停/继续/停止切换 + 单张转圈 + 格子状态角标)/ **微信抽取完全端上化(P2 #6)**(SQLCipher key 端上推导 + WeChatChatImporter + 联系人/头像端上解密;好友/群友区分) +- v0.9 (2026-06-10):**Transcript→Person 提及关系(P0 #5)**(`crm/MentionExtractor.kt` 词典反查;`BlockProcessor` 实时扫;`MentionScanner.rescanAll()` 全量重建;Room v15 `mentions` 表;`PersonProfileScreen` 「被提到」tab)/ **人↔人共现边(P1 #1)**(`RelationBuilder.rebuild()` 按会话在场+被提到配对;Room v16 `relations` 表;「相关」tab + `CommunitiesScreen` 圈子可视化)/ **待跟进(P3 #3 部分)**(`FollowupExtractor` 4B LLM 抽待办;Room v17 `followups` 表;`FollowupsScreen`)/ **话题实体(P3 #2 部分)**(`TopicExtractor` 4B 抽 org/project/event;Room v18/v19 `topics`/`topic_mentions`;`TopicsScreen`) 剩下: 1. ✅ **存储 TTL**(2026-06-08 完成):`StorageManager.prune` 按天目录删超期 .opus、文字永留;每天 periodic worker;设置页「存储」段(用量 + 保留时长 + 立即清理),默认永久保留。 2. ✅ **设置页**(基本完成):`SettingsScreen` 已含 纪要模型后端 / 聚类·合并·最短识别阈值 / llama-swap 端点 / 分段间隔 / **定时不录(quiet hours)** / 存储用量+TTL / 电池豁免。 3. ✅ **多 anchor 平均**(2026-06-07 完成,L2):重命名/合并某说话人时 `rebuildAnchor` 用该人名下全部段的均值重建 anchor,取代单段注册。 -4. **内部 Transcript 实体 → Person 表**(取代旧「回链 lemonPCRM」):在 **app 内部**从 `Transcript.text`/`Summary` 抽实体(人名/项目,端上 LLM 或规则 NER)→ 匹配本地 `Person` → 建「提及」关系(新表 `mentions(segmentId, personId)`),让人物档案显示「TA 被哪些录音提到」(现仅显示「TA 自己说的」=声纹段)。同段共现的人 → 人↔人边。**不再回链 Mac 端 lemonPCRM**(已废弃)。 +4. ✅ **内部 Transcript 实体 → Person 表**(已完成):`crm/MentionExtractor.kt`(词典反查:managed 人物的 name/realName/nickname/remark/alias)+ `MentionScanner`(单段实时 & 全量重扫);`BlockProcessor` 每块处理时实时扫 → `mentions(segmentId, personId, matchedText)`;`PersonProfileScreen` 加「被提到」tab;`rescanMentions()` 结束后自动调 `RelationBuilder.rebuild()`。 5. ✅ **hermes gateway 自启**(2026-06-04 完成):用 Magisk `service.d` 脚本开机拉起(Termux:Boot 未装,root 方案更稳),见下方「§ 自启(Magisk service.d)」。 -6. **微信抽取端上化**:消除 photolog 仅剩的 Mac 依赖(微信联系人/头像/聊天/朋友圈现由 Mac 一次性抽取推到 `filesDir/wx_enroll/`)。 +6. ✅ **微信抽取端上化**(2026-06-10 完成,P2 #6):`WeChatChatImporter`(SQLCipher key 端上推导 + 聊天读取)+ `WeChatSnsImporter`(朋友圈明文库)+ 联系人/头像端上解密(`rcontact`/`img_flag`);好友 vs 群友区分(`wechatFriend` 字段);通讯录「🔄更新」一次性整库解密。不再依赖 Mac 导出器。 ### 端上 CRM 补齐 — TODO(对齐并超越旧 lemonPCRM) > 旧 Mac 端 lemonPCRM 已废弃,能力在 app 内重建。下表是「app 当前缺、旧 lemonPCRM 有」的 6 项, > 按优先级排(P0 先做)。能力对比详见 CLAUDE.md §「App 即 Personal CRM」。 -- [ ] **P0 · #5 Transcript 内容 → Person「提及」关系**(最高价值,最贴 app 形态) - - 现状:录音只按**声纹**连到人(谁说的);不从转写文本抽人名/项目。 - - 做法:块处理后(或事后 worker)对 `Transcript.text` 抽实体(端上 LLM 4B 或规则/词典 NER)→ 匹配本地 `Person`(name/realName/nickname/alias,拼音兜底)→ 写新表 `mentions(segmentId, personId, name, confidence)`(Room 迁移 +1)。 - - 出口:`PersonProfileScreen` 加「被提到」tab(现只有「TA 自己说的」=声纹段)。 - - 依赖:端上 4B 已就绪(llama-swap 8080)。 -- [ ] **P1 · #1 人↔人关系(共现边)** - - 做法:基于 #5,同一会话/时间段被同时提到 + 同时在场(声纹)的人 → `relations(personA, personB, kind, weight)` 共现/认识边,weight 随共现次数累加。 - - 出口:人物档案「相关的人」;后续可做关系图可视化。 -- [ ] **P2 · #6 微信抽取端上化**(消除最后的 Mac 依赖,工程债) - - 现状:微信联系人/头像/聊天/朋友圈由 Mac 解密 `EnMicroMsg.db`(SQLCipher)+抠头像,推到 `filesDir/wx_enroll/`。 - - 做法:把解密+抽取搬到端上(root + SQLCipher key 端上取),`WeChatEnrollWorker` 直接读微信库;配方见 memory `wechat-android-decrypt.md` / `wechat-avatar-face-enrollment.md`。 +- [x] **P0 · #5 Transcript 内容 → Person「提及」关系**(已完成) + - 词典反查(managed 人物 name/realName/nickname/remark/alias 多别名);`BlockProcessor` 每块实时扫;`MentionScanner.rescanAll()` 全量重扫;`mentions` 表 + `PersonProfileScreen` 「被提到」tab。 +- [x] **P1 · #1 人↔人关系(共现边)**(已完成) + - `RelationBuilder.rebuild()`(rescanAll 后自动调)按 5min 静音切会话,在场(声纹)+ 被提到(mentions)的人两两配对 → `relations` 表;`PersonProfileScreen` 「相关」tab + `CommunitiesScreen` 圈子。 +- [x] **P2 · #6 微信抽取端上化**(已完成,2026-06-10) + - SQLCipher key 端上推导(MD5(imei+uin)[:7])+ `WeChatChatImporter` + 联系人/头像/朋友圈全端上。 - [ ] **P3 · #2 Organization / Project / Event 实体**(产品决策:要不要从「联系人表」长成「图谱」) - 新增节点类型 + 与 Person 的边(属于/参与);可由 #5 的实体抽取顺带产出(项目名/公司名/事件)。 - [ ] **P3 · #3 关系分析**:关系路径(两人间最短链)/ 社群发现(聚类)/ 待跟进(followups:从录音承诺「下周给他发X」抽待办)。依赖 #1 的关系边。 diff --git a/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/ChatScreen.kt b/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/ChatScreen.kt index 6be7099..2e737e0 100644 --- a/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/ChatScreen.kt +++ b/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/ChatScreen.kt @@ -192,7 +192,7 @@ internal fun ChatTab(repo: Repository) { } } - // lifelog 模式开关。 + // 模式开关行 Row( modifier = Modifier.fillMaxWidth().padding(horizontal = 12.dp), verticalAlignment = Alignment.CenterVertically, @@ -201,12 +201,20 @@ internal fun ChatTab(repo: Repository) { FilterChip( selected = vm.lifelogMode, onClick = { vm.setLifelog(!vm.lifelogMode) }, - label = { Text("🧠 lifelog 模式") }, + label = { Text("🧠 lifelog") }, ) - Text( - if (vm.lifelogMode) "回答会带上你的纪要/转写检索" else "纯聊天,不读 lifelog", - style = MaterialTheme.typography.labelSmall, + FilterChip( + selected = vm.toolMode, + onClick = { vm.setTool(!vm.toolMode) }, + label = { Text("🔧 工具") }, ) + val statusHint = when { + vm.toolMode && vm.toolStatus.isNotBlank() -> vm.toolStatus + vm.toolMode -> "AI 可自动查录音/联系人/相册" + vm.lifelogMode -> "回答带纪要/转写检索" + else -> "纯聊天" + } + Text(statusHint, style = MaterialTheme.typography.labelSmall, modifier = Modifier.weight(1f)) } HorizontalDivider() diff --git a/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/MainActivity.kt b/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/MainActivity.kt index 4050ed7..42dfd38 100644 --- a/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/MainActivity.kt +++ b/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/MainActivity.kt @@ -122,9 +122,9 @@ class MainActivity : ComponentActivity() { } private enum class Tab(val label: String, val icon: String) { - People("人物", "👤"), // 通讯录 + 联系人档案(CRM 核心,放第一位) - Recording("录音", "⏺"), // 顶部子页:录音 / 时间轴 / 声纹 / 纪要 - Chat("对话", "💬"), + Chat("对话", "💬"), // 默认首页:AI 对话 + 工具模式 + People("人物", "👤"), + Recording("录音", "⏺"), Album("相册", "🖼"), Settings("设置", "⚙"), } @@ -144,7 +144,7 @@ enum class TimeFilter(val label: String) { @Composable private fun MainScreen() { - var tab by remember { mutableStateOf(Tab.People) } + var tab by remember { mutableStateOf(Tab.Chat) } val activity = androidx.compose.ui.platform.LocalContext.current val repo = remember(activity) { Repository(activity) } diff --git a/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/PhotologScreen.kt b/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/PhotologScreen.kt index b744793..d6ed6cf 100644 --- a/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/PhotologScreen.kt +++ b/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/PhotologScreen.kt @@ -161,6 +161,7 @@ internal fun AlbumTab(repo: Repository) { val vlRunning by PhotoVlState.running.collectAsState() val vlPaused by PhotoVlState.paused.collectAsState() val vlCurrentId by PhotoVlState.currentPhotoId.collectAsState() + val embBackfill by vm.embBackfill.collectAsState() // 读相册权限(13+ READ_MEDIA_IMAGES;12 及以下 READ_EXTERNAL_STORAGE)。授予即扫全相册。 val mediaPerm = if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.TIRAMISU) @@ -221,6 +222,7 @@ internal fun AlbumTab(repo: Repository) { } else { ProgressHeader( p = progress, running = vlRunning, paused = vlPaused, + embBackfill = embBackfill, onRun = { vm.scanGallery(); vm.runVlNow() android.widget.Toast.makeText( @@ -229,6 +231,7 @@ internal fun AlbumTab(repo: Repository) { }, onPause = { vm.pauseVl() }, onStop = { vm.stopVl() }, + onBackfillEmbed = { vm.backfillCaptionEmbeddings() }, ) } // 搜索框 + 语义按钮。 @@ -312,9 +315,11 @@ private fun ProgressHeader( p: PhotoProgress, running: Boolean, paused: Boolean, + embBackfill: Pair?, onRun: () -> Unit, onPause: () -> Unit, onStop: () -> Unit, + onBackfillEmbed: () -> Unit, ) { if (p.total == 0) { Row( @@ -359,6 +364,23 @@ private fun ProgressHeader( } else { Text("✓ 全部已理解", style = MaterialTheme.typography.labelSmall) } + // 补算语义向量(有 caption 但无 embedding 时显示)。 + if (embBackfill != null) { + val (done, total) = embBackfill + Row( + verticalAlignment = Alignment.CenterVertically, + horizontalArrangement = Arrangement.spacedBy(8.dp), + modifier = Modifier.padding(top = 4.dp), + ) { + CircularProgressIndicator(modifier = Modifier.size(14.dp), strokeWidth = 2.dp) + Text("语义向量 $done/$total", style = MaterialTheme.typography.labelSmall) + } + } else if (p.captionDone > 0) { + TextButton( + onClick = onBackfillEmbed, + contentPadding = androidx.compose.foundation.layout.PaddingValues(horizontal = 0.dp, vertical = 0.dp), + ) { Text("🔢 补算语义向量", style = MaterialTheme.typography.labelSmall) } + } } } diff --git a/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/PhotologViewModel.kt b/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/PhotologViewModel.kt index 000fe85..c5570bc 100644 --- a/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/PhotologViewModel.kt +++ b/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/PhotologViewModel.kt @@ -116,7 +116,8 @@ class PhotologViewModel(app: Application) : AndroidViewModel(app) { val app = getApplication() try { val rows = withContext(Dispatchers.IO) { - val embed = EmbedClient(ChatConfig.localBase(app), ChatConfig.modelEmbed(app)) + val (embedBase, embedKey, embedModel) = com.lemon.lifelog.chat.ModelConfig.slotEndpoint(app, com.lemon.lifelog.chat.ModelSlot.EMBED) + val embed = EmbedClient(embedBase, embedModel, embedKey) val q = embed.embed(query.trim()) val vecs = repo.photos.captionVecs() if (vecs.isEmpty()) return@withContext emptyList() @@ -138,6 +139,35 @@ class PhotologViewModel(app: Application) : AndroidViewModel(app) { } } + /** + * 补算 captionEmbedding:对已有 caption 但缺 embedding 的照片(embed 模型后配时回填), + * 只调 embed 端点,不重跑 VL。返回成功补算数量。 + */ + private val _embBackfill = MutableStateFlow?>(null) // (done, total) | null=idle + val embBackfill: StateFlow?> = _embBackfill + + fun backfillCaptionEmbeddings() { + if (_embBackfill.value != null) return + viewModelScope.launch(Dispatchers.IO) { + val app = getApplication() + val pending = repo.photos.withCaptionNoEmbedding() + if (pending.isEmpty()) { _embBackfill.value = null; return@launch } + val (embedBase, embedKey, embedModel) = com.lemon.lifelog.chat.ModelConfig.slotEndpoint(app, com.lemon.lifelog.chat.ModelSlot.EMBED) + val client = EmbedClient(embedBase, embedModel, embedKey) + var done = 0 + _embBackfill.value = 0 to pending.size + for (p in pending) { + runCatching { + val emb = client.embed(p.caption) + repo.photos.updateCaptionEmbedding(p.id, com.lemon.lifelog.vision.ImageUtil.packEmbedding(emb)) + done++ + } + _embBackfill.value = done to pending.size + } + _embBackfill.value = null + } + } + fun createPersonAndTag(faceId: Long, name: String) { viewModelScope.launch { // 手动建的人 = 我关心的 → 直接 managed。 diff --git a/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/SettingsScreen.kt b/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/SettingsScreen.kt index 95b64ba..619a71d 100644 --- a/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/SettingsScreen.kt +++ b/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/SettingsScreen.kt @@ -17,6 +17,7 @@ import androidx.compose.foundation.layout.height import androidx.compose.foundation.layout.padding import androidx.compose.foundation.rememberScrollState import androidx.compose.foundation.verticalScroll +import androidx.compose.material3.AlertDialog import androidx.compose.material3.Button import androidx.compose.material3.DropdownMenu import androidx.compose.material3.DropdownMenuItem @@ -26,6 +27,7 @@ import androidx.compose.material3.HorizontalDivider import androidx.compose.material3.MaterialTheme import androidx.compose.material3.OutlinedTextField import androidx.compose.material3.Text +import androidx.compose.material3.TextButton import androidx.compose.runtime.Composable import androidx.compose.runtime.LaunchedEffect import androidx.compose.runtime.getValue @@ -40,6 +42,9 @@ import com.lemon.lifelog.asr.SpeakerClusterer import com.lemon.lifelog.service.BlockBuffer import com.lemon.lifelog.service.QuietHours import com.lemon.lifelog.chat.ChatConfig +import com.lemon.lifelog.chat.LlmProvider +import com.lemon.lifelog.chat.ModelConfig +import com.lemon.lifelog.chat.ModelSlot import com.lemon.lifelog.storage.StorageManager import com.lemon.lifelog.summary.SummaryScheduler import com.lemon.lifelog.summary.SummaryWorker @@ -71,14 +76,29 @@ fun SettingsScreen(activity: Context) { var pruneMsg by remember { mutableStateOf("") } val scope = rememberCoroutineScope() LaunchedEffect(Unit) { audioUsage = StorageManager.audioUsageBytes(activity.filesDir) } - val backendTargets = remember { ChatConfig.targets(activity) } + var configRev by remember { mutableStateOf(0) } + val backendTargets = remember(configRev) { ChatConfig.targets(activity) } var localBase by remember { mutableStateOf(ChatConfig.localBase(activity)) } var m2b by remember { mutableStateOf(ChatConfig.model2b(activity)) } var m4b by remember { mutableStateOf(ChatConfig.model4b(activity)) } - var mEmbed by remember { mutableStateOf(ChatConfig.modelEmbed(activity)) } var endpointMsg by remember { mutableStateOf("") } + var hermesUrl by remember(configRev) { mutableStateOf(ModelConfig.hermesProvider(activity).baseUrl) } + var hermesKey by remember(configRev) { mutableStateOf(ModelConfig.hermesProvider(activity).apiKey) } + var hermesModel by remember(configRev) { mutableStateOf(ModelConfig.hermesProvider(activity).chatModel) } + var hermesSaveMsg by remember { mutableStateOf("") } + + var editingProvider by remember { mutableStateOf(null) } + var isNewProvider by remember { mutableStateOf(false) } + + var vlPid by remember(configRev) { mutableStateOf(ModelConfig.slotProviderId(activity, ModelSlot.VL)) } + var vlModel by remember(configRev) { mutableStateOf(ModelConfig.slotModel(activity, ModelSlot.VL)) } + var vlSaveMsg by remember { mutableStateOf("") } + var embedPid by remember(configRev) { mutableStateOf(ModelConfig.slotProviderId(activity, ModelSlot.EMBED)) } + var embedModel by remember(configRev) { mutableStateOf(ModelConfig.slotModel(activity, ModelSlot.EMBED)) } + var embedSaveMsg by remember { mutableStateOf("") } + var batteryExempt by remember { mutableStateOf(isBatteryExempt(activity)) } Column( @@ -125,7 +145,7 @@ fun SettingsScreen(activity: Context) { style = MaterialTheme.typography.labelSmall) HorizontalDivider() - Text("本地模型端点(对话 + 纪要共用 llama-swap)", style = MaterialTheme.typography.titleMedium) + Text("本地 llama-swap", style = MaterialTheme.typography.titleMedium) OutlinedTextField(value = localBase, onValueChange = { localBase = it }, label = { Text("baseUrl") }, singleLine = true, modifier = Modifier.fillMaxWidth()) Row(horizontalArrangement = Arrangement.spacedBy(8.dp)) { @@ -134,18 +154,101 @@ fun SettingsScreen(activity: Context) { OutlinedTextField(value = m4b, onValueChange = { m4b = it }, label = { Text("4B model 名") }, singleLine = true, modifier = Modifier.weight(1f)) } - OutlinedTextField(value = mEmbed, onValueChange = { mEmbed = it }, - label = { Text("embed model 名(语义搜图,llama-swap 里配 --embedding)") }, - singleLine = true, modifier = Modifier.fillMaxWidth()) Row(verticalAlignment = Alignment.CenterVertically, horizontalArrangement = Arrangement.spacedBy(8.dp)) { Button(onClick = { - ChatConfig.save(activity, localBase, m2b, m4b, mEmbed) + ChatConfig.save(activity, localBase, m2b, m4b) endpointMsg = "已保存" - }) { Text("保存端点") } + configRev++ + }) { Text("保存") } if (endpointMsg.isNotEmpty()) Text(endpointMsg, style = MaterialTheme.typography.labelSmall) } HorizontalDivider() + Text("Hermes 网关", style = MaterialTheme.typography.titleMedium) + OutlinedTextField(value = hermesUrl, onValueChange = { hermesUrl = it }, + label = { Text("接口地址") }, singleLine = true, modifier = Modifier.fillMaxWidth()) + OutlinedTextField(value = hermesKey, onValueChange = { hermesKey = it }, + label = { Text("API Key") }, singleLine = true, modifier = Modifier.fillMaxWidth()) + OutlinedTextField(value = hermesModel, onValueChange = { hermesModel = it }, + label = { Text("路由模型名(如 kimi-for-coding / qwen-plus)") }, + singleLine = true, modifier = Modifier.fillMaxWidth()) + Row(verticalAlignment = Alignment.CenterVertically, horizontalArrangement = Arrangement.spacedBy(8.dp)) { + Button(onClick = { + ModelConfig.saveHermes(activity, hermesUrl, hermesKey, hermesModel) + hermesSaveMsg = "已保存" + configRev++ + }) { Text("保存 Hermes") } + if (hermesSaveMsg.isNotEmpty()) Text(hermesSaveMsg, style = MaterialTheme.typography.labelSmall) + } + + HorizontalDivider() + Text("远程服务商", style = MaterialTheme.typography.titleMedium) + Text("添加任意 OpenAI 兼容 API(通义千问 / Kimi / MiniMax 等),对话模型名不为空则出现在聊天列表。", + style = MaterialTheme.typography.labelSmall) + val userProviders = remember(configRev) { ModelConfig.userProviders(activity) } + userProviders.forEach { p -> + Row( + modifier = Modifier.fillMaxWidth(), + verticalAlignment = Alignment.CenterVertically, + horizontalArrangement = Arrangement.spacedBy(8.dp), + ) { + Column(modifier = Modifier.weight(1f)) { + Text(p.label, style = MaterialTheme.typography.bodyMedium) + Text( + buildString { + append(p.baseUrl.take(40)) + if (p.chatModel.isNotBlank()) append(" ${p.chatModel}") + }, + style = MaterialTheme.typography.labelSmall, + ) + } + TextButton(onClick = { editingProvider = p; isNewProvider = false }) { Text("编辑") } + TextButton(onClick = { ModelConfig.deleteProvider(activity, p.id); configRev++ }) { Text("删除") } + } + } + Button(onClick = { + editingProvider = LlmProvider(ModelConfig.generateId(), "", "", "", "") + isNewProvider = true + }) { Text("+ 添加服务商") } + + HorizontalDivider() + Text("模型分配", style = MaterialTheme.typography.titleMedium) + Text("指定 VL(图像理解)和语义向量各用哪个服务商 + 模型。", + style = MaterialTheme.typography.labelSmall) + val allProviders = remember(configRev) { ModelConfig.allProviders(activity) } + // VL slot + Text("图像理解(VL)", style = MaterialTheme.typography.labelLarge) + DropdownSetting( + "服务商", allProviders.firstOrNull { it.id == vlPid }?.label ?: vlPid, + allProviders.map { it.label to it.id }, + ) { vlPid = it } + Row(verticalAlignment = Alignment.CenterVertically, horizontalArrangement = Arrangement.spacedBy(8.dp)) { + OutlinedTextField(value = vlModel, onValueChange = { vlModel = it }, + label = { Text("模型名") }, singleLine = true, modifier = Modifier.weight(1f)) + Button(onClick = { + ModelConfig.setSlot(activity, ModelSlot.VL, vlPid, vlModel) + vlSaveMsg = "已保存" + }) { Text("保存") } + } + if (vlSaveMsg.isNotEmpty()) Text(vlSaveMsg, style = MaterialTheme.typography.labelSmall) + // EMBED slot + Text("语义向量", style = MaterialTheme.typography.labelLarge) + DropdownSetting( + "服务商", allProviders.firstOrNull { it.id == embedPid }?.label ?: embedPid, + allProviders.map { it.label to it.id }, + ) { embedPid = it } + Row(verticalAlignment = Alignment.CenterVertically, horizontalArrangement = Arrangement.spacedBy(8.dp)) { + OutlinedTextField(value = embedModel, onValueChange = { embedModel = it }, + label = { Text("模型名(llama-swap 里配 --embedding 的 GGUF 名)") }, + singleLine = true, modifier = Modifier.weight(1f)) + Button(onClick = { + ModelConfig.setSlot(activity, ModelSlot.EMBED, embedPid, embedModel) + embedSaveMsg = "已保存" + }) { Text("保存") } + } + if (embedSaveMsg.isNotEmpty()) Text(embedSaveMsg, style = MaterialTheme.typography.labelSmall) + + HorizontalDivider() Text("录音 / 电池", style = MaterialTheme.typography.titleMedium) DropdownSetting("分段间隔(静音多久收尾一段)", "$blockSilenceMin 分钟", listOf(1, 2, 3, 5, 10, 15).map { "$it 分钟" to it }) { @@ -209,6 +312,65 @@ fun SettingsScreen(activity: Context) { Spacer(Modifier.height(24.dp)) } + + editingProvider?.let { p -> + ProviderEditDialog( + initial = p, + isNew = isNewProvider, + onConfirm = { updated -> + if (isNewProvider) ModelConfig.addProvider(activity, updated) + else ModelConfig.updateProvider(activity, updated) + editingProvider = null + configRev++ + }, + onDismiss = { editingProvider = null }, + ) + } +} + +@Composable +private fun ProviderEditDialog( + initial: LlmProvider, + isNew: Boolean, + onConfirm: (LlmProvider) -> Unit, + onDismiss: () -> Unit, +) { + var label by remember { mutableStateOf(initial.label) } + var baseUrl by remember { mutableStateOf(initial.baseUrl) } + var apiKey by remember { mutableStateOf(initial.apiKey) } + var chatModel by remember { mutableStateOf(initial.chatModel) } + AlertDialog( + onDismissRequest = onDismiss, + title = { Text(if (isNew) "添加服务商" else "编辑服务商") }, + text = { + Column(verticalArrangement = Arrangement.spacedBy(8.dp)) { + OutlinedTextField(value = label, onValueChange = { label = it }, + label = { Text("名称") }, singleLine = true, modifier = Modifier.fillMaxWidth()) + OutlinedTextField(value = baseUrl, onValueChange = { baseUrl = it }, + label = { Text("接口地址 (baseUrl)") }, singleLine = true, modifier = Modifier.fillMaxWidth()) + OutlinedTextField(value = apiKey, onValueChange = { apiKey = it }, + label = { Text("API Key") }, singleLine = true, modifier = Modifier.fillMaxWidth()) + OutlinedTextField(value = chatModel, onValueChange = { chatModel = it }, + label = { Text("对话模型名(留空则不出现在聊天列表)") }, + singleLine = true, modifier = Modifier.fillMaxWidth()) + Text("接口地址需兼容 OpenAI /v1/chat/completions 协议。", + style = MaterialTheme.typography.labelSmall) + } + }, + confirmButton = { + TextButton(onClick = { + if (label.isNotBlank() && baseUrl.isNotBlank()) { + onConfirm(initial.copy( + label = label.trim(), + baseUrl = baseUrl.trimEnd('/'), + apiKey = apiKey.trim(), + chatModel = chatModel.trim(), + )) + } + }) { Text("保存") } + }, + dismissButton = { TextButton(onClick = onDismiss) { Text("取消") } }, + ) } /** 字节数 → 人类可读(KB/MB/GB)。 */ diff --git a/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/chat/ChatClient.kt b/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/chat/ChatClient.kt index ec8840b..92c955e 100644 --- a/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/chat/ChatClient.kt +++ b/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/chat/ChatClient.kt @@ -12,6 +12,18 @@ import org.json.JSONObject import java.io.IOException import java.util.concurrent.TimeUnit +data class ToolCall( + val id: String, + val name: String, + val arguments: String, +) + +data class ToolChatResult( + val text: String, + val toolCalls: List?, + val rawToolCallsJson: JSONArray?, +) + /** * OpenAI 兼容的聊天客户端,支持 SSE 流式(stream:true)增量回复。 * @@ -124,4 +136,65 @@ class ChatClient { if (choices.length() == 0) return "" return choices.getJSONObject(0).optJSONObject("message")?.optString("content", "").orEmpty() } + + /** + * 非流式 function calling 请求。 + * 传入 [messages](JSONObject 列表,支持 tool/assistant with tool_calls 等 role)和 + * [tools](LifelogTools.SCHEMAS),返回 [ToolChatResult]: + * - 若模型给出 tool_calls → toolCalls 非 null,调用方执行工具后再次调用本方法 + * - 若模型给出最终文字 → toolCalls == null,text 是答案 + */ + @Throws(IOException::class) + suspend fun chatWithTools( + target: ChatTarget, + messages: List, + tools: JSONArray, + maxTokens: Int = 1024, + ): ToolChatResult = withContext(Dispatchers.IO) { + val payload = JSONObject().apply { + put("model", target.model) + put("stream", false) + put("max_tokens", maxTokens) + put("tools", tools) + put("tool_choice", "auto") + if (target.kind == ChatTargetKind.LOCAL) { + put("chat_template_kwargs", JSONObject().put("enable_thinking", false)) + } + put("messages", JSONArray().apply { messages.forEach { put(it) } }) + } + val builder = Request.Builder() + .url("${target.baseUrl}/v1/chat/completions") + .header("Content-Type", "application/json") + .post(payload.toString().toRequestBody(JSON)) + if (target.apiKey.isNotBlank()) builder.header("Authorization", "Bearer ${target.apiKey}") + + http.newCall(builder.build()).execute().use { resp -> + val body = resp.body?.string().orEmpty() + if (!resp.isSuccessful) throw IOException("${target.label} ${resp.code}: ${body.take(300)}") + parseToolChatResponse(body) + } + } + + private fun parseToolChatResponse(body: String): ToolChatResult { + val json = JSONObject(body) + val choices = json.optJSONArray("choices") ?: throw IOException("no choices") + if (choices.length() == 0) throw IOException("empty choices") + val message = choices.getJSONObject(0).optJSONObject("message") + ?: throw IOException("no message") + val text = if (message.isNull("content")) "" else message.optString("content", "") + val rawCalls = message.optJSONArray("tool_calls") + if (rawCalls != null && rawCalls.length() > 0) { + val calls = (0 until rawCalls.length()).map { i -> + val tc = rawCalls.getJSONObject(i) + val fn = tc.getJSONObject("function") + ToolCall( + id = tc.getString("id"), + name = fn.getString("name"), + arguments = fn.optString("arguments", "{}"), + ) + } + return ToolChatResult(text, calls, rawCalls) + } + return ToolChatResult(text, null, null) + } } diff --git a/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/chat/ChatConfig.kt b/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/chat/ChatConfig.kt index c0a17e3..a472a73 100644 --- a/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/chat/ChatConfig.kt +++ b/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/chat/ChatConfig.kt @@ -3,7 +3,7 @@ package com.lemon.lifelog.chat import android.content.Context import com.lemon.lifelog.BuildConfig -enum class ChatTargetKind { LOCAL, HERMES } +enum class ChatTargetKind { LOCAL, HERMES, REMOTE } /** * 一个可聊天的对象 = 一个 OpenAI 兼容端点 + 一个 model 名。 @@ -34,6 +34,7 @@ object ChatConfig { /** 语义搜索的文本 embedding model 名(llama-swap 里配 --embedding 的 GGUF,如 bge-small-zh)。 */ const val DEFAULT_MODEL_EMBED = "embed" + const val ID_LOCAL = "local-swap" const val ID_2B = "local-2b" const val ID_4B = "local-4b" const val ID_HERMES = "hermes" @@ -61,21 +62,20 @@ object ChatConfig { .apply() } - /** 当前可选的聊天对象列表。 */ + /** 当前可选的聊天对象列表(本地 2B/4B + Hermes + 用户添加的远程服务商)。 */ fun targets(ctx: Context): List { val local = localBase(ctx) - return listOf( - ChatTarget(ID_2B, "本地 2B", local, "", model2b(ctx), ChatTargetKind.LOCAL), - ChatTarget(ID_4B, "本地 4B", local, "", model4b(ctx), ChatTargetKind.LOCAL), - ChatTarget( - ID_HERMES, - "Hermes Agent", - BuildConfig.HERMES_API_URL.trimEnd('/'), - BuildConfig.HERMES_API_KEY, - BuildConfig.HERMES_MODEL, - ChatTargetKind.HERMES, - ), - ) + val hermes = ModelConfig.hermesProvider(ctx) + return buildList { + add(ChatTarget(ID_2B, "本地 2B", local, "", model2b(ctx), ChatTargetKind.LOCAL)) + add(ChatTarget(ID_4B, "本地 4B", local, "", model4b(ctx), ChatTargetKind.LOCAL)) + add(ChatTarget(ID_HERMES, hermes.label, hermes.baseUrl, hermes.apiKey, hermes.chatModel, ChatTargetKind.HERMES)) + ModelConfig.userProviders(ctx) + .filter { it.chatModel.isNotBlank() } + .forEach { p -> + add(ChatTarget(p.id, p.label, p.baseUrl, p.apiKey, p.chatModel, ChatTargetKind.REMOTE)) + } + } } fun targetById(ctx: Context, id: String?): ChatTarget { diff --git a/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/chat/ChatViewModel.kt b/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/chat/ChatViewModel.kt index d690a02..db2d363 100644 --- a/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/chat/ChatViewModel.kt +++ b/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/chat/ChatViewModel.kt @@ -10,6 +10,8 @@ import androidx.lifecycle.viewModelScope import com.lemon.lifelog.data.ChatConversation import com.lemon.lifelog.data.ChatMessage import com.lemon.lifelog.data.Repository +import org.json.JSONArray +import org.json.JSONObject import kotlinx.coroutines.Dispatchers import kotlinx.coroutines.launch import kotlinx.coroutines.withContext @@ -33,6 +35,10 @@ class ChatViewModel(app: Application) : AndroidViewModel(app) { private set var lifelogMode by mutableStateOf(prefs.getBoolean(K_LIFELOG, false)) private set + var toolMode by mutableStateOf(prefs.getBoolean(K_TOOL, false)) + private set + var toolStatus by mutableStateOf("") + private set var input by mutableStateOf("") var streaming by mutableStateOf(false) @@ -47,6 +53,13 @@ class ChatViewModel(app: Application) : AndroidViewModel(app) { prefs.edit().putBoolean(K_LIFELOG, on).apply() } + fun setTool(on: Boolean) { + toolMode = on + prefs.edit().putBoolean(K_TOOL, on).apply() + // 工具模式和 lifelog 模式互斥 + if (on && lifelogMode) setLifelog(false) + } + fun selectTarget(id: String) { selectedTargetId = id prefs.edit().putString(K_TARGET, id).apply() @@ -90,8 +103,8 @@ class ChatViewModel(app: Application) : AndroidViewModel(app) { input = "" val app = getApplication() val target = ChatConfig.targetById(app, selectedTargetId) - val useLifelog = lifelogMode viewModelScope.launch { + // ── 公共准备:建会话 + 存 user 消息 + 取历史 ────────────────── val now = System.currentTimeMillis() var convId = currentConvId if (convId == null) { @@ -108,48 +121,128 @@ class ChatViewModel(app: Application) : AndroidViewModel(app) { repo.chats.insertMessage(ChatMessage(conversationId = cid, role = "user", content = text, createdAt = now)) repo.chats.touch(cid, now) } - val history = withContext(Dispatchers.IO) { repo.chats.messagesOf(cid) }.map { it.role to it.content } + val history = withContext(Dispatchers.IO) { repo.chats.messagesOf(cid) } streaming = true streamingText = "" try { - val messagesToSend = if (useLifelog) { - val c = LifelogRetriever.buildContext(repo, text) - if (c != null) listOf("system" to c) + history else history + if (toolMode) { + agenticLoop(target, cid, history) } else { - history - } - val full = client.stream(target, messagesToSend) { delta -> streamingText += delta } - withContext(Dispatchers.IO) { - repo.chats.insertMessage( - ChatMessage( - conversationId = cid, role = "assistant", - content = full.ifBlank { "(空回复)" }, model = target.model, - createdAt = System.currentTimeMillis(), - ), - ) - repo.chats.touch(cid, System.currentTimeMillis()) + normalChat(text, target, cid, history.map { it.role to it.content }) } } catch (t: Throwable) { withContext(Dispatchers.IO) { - repo.chats.insertMessage( - ChatMessage( - conversationId = cid, role = "assistant", - content = "⚠ 出错:${t.message ?: t.toString()}", model = target.model, - createdAt = System.currentTimeMillis(), - ), - ) + repo.chats.insertMessage(ChatMessage( + conversationId = cid, role = "assistant", + content = "⚠ 出错:${t.message ?: t.toString()}", model = target.model, + createdAt = System.currentTimeMillis(), + )) } } finally { streaming = false streamingText = "" + toolStatus = "" + } + } + } + + private suspend fun normalChat( + text: String, + target: ChatTarget, + cid: Long, + history: List>, + ) { + val messagesToSend = if (lifelogMode) { + val c = LifelogRetriever.buildContext(repo, text) + if (c != null) listOf("system" to c) + history else history + } else { + history + } + val full = client.stream(target, messagesToSend) { delta -> streamingText += delta } + withContext(Dispatchers.IO) { + repo.chats.insertMessage(ChatMessage( + conversationId = cid, role = "assistant", + content = full.ifBlank { "(空回复)" }, model = target.model, + createdAt = System.currentTimeMillis(), + )) + repo.chats.touch(cid, System.currentTimeMillis()) + } + } + + /** + * 工具模式 agentic loop: + * 1. 发带 tools 的请求 → 若 LLM 返回 tool_calls,本地执行 → 追加结果 → 再请求 + * 2. 直到 LLM 不再调工具(给出最终文字)或达到最大轮数 + * 3. 只把 user 消息和最终 assistant 文字存 Room(中间工具消息不持久化) + */ + private suspend fun agenticLoop( + target: ChatTarget, + cid: Long, + history: List, + ) { + // 构建初始 messages(system prompt + 全部 Room 历史,已含刚存的 user 消息) + val messages: MutableList = mutableListOf() + messages.add(JSONObject().put("role", "system").put("content", TOOL_SYSTEM_PROMPT)) + history.forEach { m -> + messages.add(JSONObject().put("role", m.role).put("content", m.content)) + } + + var finalText = "" + var rounds = 0 + toolStatus = "思考中…" + + while (rounds < MAX_TOOL_ROUNDS) { + val result = withContext(Dispatchers.IO) { + client.chatWithTools(target, messages, LifelogTools.SCHEMAS) + } + if (result.toolCalls.isNullOrEmpty()) { + finalText = result.text + break + } + // assistant 消息(含 tool_calls,API 要求原样回传) + val assistantMsg = JSONObject().put("role", "assistant").put("content", result.text.ifBlank { "" }) + result.rawToolCallsJson?.let { assistantMsg.put("tool_calls", it) } + messages.add(assistantMsg) + + // 逐个执行工具 + for (call in result.toolCalls) { + toolStatus = "⚙ ${call.name}" + streamingText = "调用工具:${call.name}\n" + val args = runCatching { JSONObject(call.arguments) }.getOrElse { JSONObject() } + val toolResult = LifelogTools.execute(repo, call.name, args) + streamingText += toolResult.take(200) + messages.add(JSONObject() + .put("role", "tool") + .put("tool_call_id", call.id) + .put("content", toolResult)) } + rounds++ + toolStatus = "整理回复中(第 $rounds 轮)…" + streamingText = toolStatus + } + + if (finalText.isBlank()) finalText = "(未获得有效回复,请重试或检查服务商是否支持 function calling)" + withContext(Dispatchers.IO) { + repo.chats.insertMessage(ChatMessage( + conversationId = cid, role = "assistant", + content = finalText, model = target.model, + createdAt = System.currentTimeMillis(), + )) + repo.chats.touch(cid, System.currentTimeMillis()) } } companion object { private const val K_LIFELOG = "lifelog_mode" + private const val K_TOOL = "tool_mode" private const val K_TARGET = "selected_target" - private const val K_CONV = "current_conv" + private const val K_CONV = "current_conv" + private const val MAX_TOOL_ROUNDS = 8 + private val TOOL_SYSTEM_PROMPT = """ + 你是用户的 lifelog AI 助手,可以访问用户 24/7 录音转写、会话纪要、联系人档案和相册。 + 当用户询问记忆、联系人或照片时,请主动调用相应工具获取真实数据后再回答。 + 多步骤问题可多次调用工具。最终回复用中文,简洁清晰。 + """.trimIndent() } } diff --git a/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/chat/LifelogTools.kt b/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/chat/LifelogTools.kt new file mode 100644 index 0000000..9b052c4 --- /dev/null +++ b/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/chat/LifelogTools.kt @@ -0,0 +1,235 @@ +package com.lemon.lifelog.chat + +import com.lemon.lifelog.data.Repository +import kotlinx.coroutines.Dispatchers +import kotlinx.coroutines.flow.first +import kotlinx.coroutines.withContext +import org.json.JSONArray +import org.json.JSONObject +import java.text.SimpleDateFormat +import java.util.Date +import java.util.Locale + +/** + * lifelog 工具集:OpenAI function calling schema + 本地执行。 + * + * 所有工具直接查 Room DB,在 app 进程内执行,无需网络/额外服务。 + * agenticLoop(ChatViewModel)检测到 tool_calls → 调 execute() → 把结果以 tool role 追回给 LLM。 + * + * 当前 6 个工具: + * search_transcripts 搜录音转写片段 + * get_person 查联系人档案 + * list_persons 列 CRM 人物 + * get_summaries 取会话纪要 + * search_photos 关键词搜相册 + * get_followups 待跟进事项 + */ +object LifelogTools { + + private val TS = SimpleDateFormat("MM-dd HH:mm", Locale.US) + private val TSL = SimpleDateFormat("MM-dd HH:mm:ss", Locale.US) + + // ── Schema(OpenAI function calling 格式)──────────────────────────────── + + val SCHEMAS: JSONArray by lazy { + JSONArray().apply { + put(schema( + name = "search_transcripts", + desc = "在录音转写片段中搜索关键词,返回匹配段落(时间、说话人、内容)", + props = mapOf( + "query" to prop("string", "搜索关键词"), + "speaker" to prop("string", "限定说话人姓名,可选"), + "limit" to prop("integer", "返回条数,默认 12"), + ), + required = listOf("query"), + )) + put(schema( + name = "get_person", + desc = "按姓名或备注查找联系人档案(基本资料、AI 印象、简历、关联声纹/人脸)", + props = mapOf("name" to prop("string", "人物姓名或备注名")), + required = listOf("name"), + )) + put(schema( + name = "list_persons", + desc = "列出 CRM 中的联系人(默认只列我管理的人)", + props = mapOf( + "managed_only" to prop("boolean", "true=只看管理的人(默认),false=全部通讯录"), + ), + required = emptyList(), + )) + put(schema( + name = "get_summaries", + desc = "获取最近的会话纪要(LLM 自动生成的录音摘要,时间倒序)", + props = mapOf("limit" to prop("integer", "返回篇数,默认 5,最多 20")), + required = emptyList(), + )) + put(schema( + name = "search_photos", + desc = "在相册中按关键词或人名搜索照片(需要 VL 索引完成后才有描述内容)", + props = mapOf( + "query" to prop("string", "关键词或人名"), + "limit" to prop("integer", "返回张数,默认 8"), + ), + required = listOf("query"), + )) + put(schema( + name = "get_followups", + desc = "获取待跟进事项列表(从录音中 AI 抽取的 TODO/行动项,未完成在前)", + props = emptyMap(), + required = emptyList(), + )) + } + } + + private fun schema(name: String, desc: String, props: Map, required: List) = + JSONObject().apply { + put("type", "function") + put("function", JSONObject().apply { + put("name", name) + put("description", desc) + put("parameters", JSONObject().apply { + put("type", "object") + put("properties", JSONObject().apply { props.forEach { (k, v) -> put(k, v) } }) + put("required", JSONArray().apply { required.forEach { put(it) } }) + }) + }) + } + + private fun prop(type: String, description: String) = + JSONObject().put("type", type).put("description", description) + + // ── 执行(在 IO 线程调)────────────────────────────────────────────────── + + suspend fun execute(repo: Repository, name: String, args: JSONObject): String = + withContext(Dispatchers.IO) { + try { + when (name) { + "search_transcripts" -> searchTranscripts(repo, args) + "get_person" -> getPerson(repo, args) + "list_persons" -> listPersons(repo, args) + "get_summaries" -> getSummaries(repo, args) + "search_photos" -> searchPhotos(repo, args) + "get_followups" -> getFollowups(repo) + else -> "未知工具:$name" + } + } catch (t: Throwable) { + "工具执行出错($name):${t.message?.take(120)}" + } + } + + // ── 各工具实现 ──────────────────────────────────────────────────────────── + + private suspend fun searchTranscripts(repo: Repository, args: JSONObject): String { + val query = args.optString("query", "").ifBlank { return "请提供搜索关键词" } + val speaker = args.optString("speaker", "").ifBlank { null } + val limit = args.optInt("limit", 12).coerceIn(1, 40) + val rows = repo.segments.searchTextLike(query, limit * 3) + .filter { it.text != null } + .let { list -> if (speaker != null) list.filter { it.speakerName?.contains(speaker) == true } else list } + .take(limit) + if (rows.isEmpty()) return "未找到包含「$query」的录音片段。" + return buildString { + appendLine("找到 ${rows.size} 条片段(关键词:「$query」):") + rows.forEach { r -> + val who = r.speakerName ?: r.personName ?: "未知" + appendLine("[${TSL.format(Date(r.startTs))}] $who:${r.text?.trim()}") + } + } + } + + private suspend fun getPerson(repo: Repository, args: JSONObject): String { + val q = args.optString("name", "").ifBlank { return "请提供人物姓名" } + val persons = repo.persons.all().filter { p -> + p.name.contains(q) || p.realName?.contains(q) == true || + p.nickname?.contains(q) == true || p.remark?.contains(q) == true + } + if (persons.isEmpty()) return "未找到与「$q」匹配的联系人。" + return buildString { + persons.take(3).forEach { p -> + appendLine("=== ${p.name} ===") + p.realName?.takeIf { it.isNotBlank() }?.let { appendLine("真实姓名:$it") } + p.nickname?.takeIf { it.isNotBlank() }?.let { appendLine("微信昵称:$it") } + p.remark?.takeIf { it.isNotBlank() }?.let { appendLine("备注:$it") } + p.phone?.takeIf { it.isNotBlank() }?.let { appendLine("手机:$it") } + if (p.wechatFriend == true) appendLine("微信好友:是") + p.impression?.takeIf { it.isNotBlank() }?.let { appendLine("AI 印象:${it.take(400)}") } + p.resume?.takeIf { it.isNotBlank() }?.let { appendLine("简历:${it.take(300)}") } + val tags = buildList { + if (p.managed) add("CRM 管理中") + if (p.voiceSpeakerId != null) add("有声纹") + if (p.faceEmbedding != null) add("有人脸") + }.joinToString(" / ") + if (tags.isNotBlank()) appendLine("标签:$tags") + appendLine() + } + } + } + + private suspend fun listPersons(repo: Repository, args: JSONObject): String { + val managedOnly = args.optBoolean("managed_only", true) + val persons = repo.persons.all().let { if (managedOnly) it.filter { p -> p.managed } else it } + if (persons.isEmpty()) return if (managedOnly) "CRM 中暂无管理的人物。" else "通讯录为空。" + return buildString { + appendLine("共 ${persons.size} 位${if (managedOnly) "(管理中)" else ""}:") + persons.forEach { p -> + val tags = buildList { + if (p.wechatFriend == true) add("微信") + if (p.voiceSpeakerId != null) add("声纹") + if (p.faceEmbedding != null) add("人脸") + if (!p.phone.isNullOrBlank()) add("手机") + }.joinToString("/") + appendLine("- ${p.name}${if (tags.isNotEmpty()) "($tags)" else ""}") + } + } + } + + private suspend fun getSummaries(repo: Repository, args: JSONObject): String { + val limit = args.optInt("limit", 5).coerceIn(1, 20) + val list = repo.summaries.recentList(limit) + if (list.isEmpty()) return "暂无会话纪要(需先生成)。" + return buildString { + appendLine("最近 ${list.size} 篇纪要:") + list.forEach { s -> + val range = "${TS.format(Date(s.windowStartTs))}–${TS.format(Date(s.windowEndTs))}" + appendLine("\n【$range】") + appendLine(s.markdown.trim().take(700)) + } + } + } + + private suspend fun searchPhotos(repo: Repository, args: JSONObject): String { + val query = args.optString("query", "").ifBlank { return "请提供搜索关键词" } + val limit = args.optInt("limit", 8).coerceIn(1, 20) + val photos = repo.photos.searchByKeyword(query, limit) + if (photos.isEmpty()) return "未找到与「$query」相关的照片(VL 描述索引可能尚未完成)。" + return buildString { + appendLine("找到 ${photos.size} 张相关照片(关键词:「$query」):") + photos.forEach { p -> + val date = TS.format(Date(p.takenTs)) + val caption = p.caption?.take(100) ?: "(暂无场景描述)" + val faces = if (p.faceCount > 0) "${p.faceCount}人" else "无人脸" + appendLine("[$date $faces] $caption") + } + } + } + + private suspend fun getFollowups(repo: Repository): String { + val items = repo.followups.observeAll().first() + if (items.isEmpty()) return "暂无待跟进事项。" + val open = items.filter { !it.done } + val done = items.filter { it.done } + return buildString { + if (open.isNotEmpty()) { + appendLine("未完成(${open.size} 条):") + open.forEach { f -> + val who = f.personName?.let { "($it)" } ?: "" + appendLine("- ${f.text}$who [${TS.format(Date(f.sourceTs))}]") + } + } + if (done.isNotEmpty()) { + appendLine("\n已完成(${done.size} 条,最近 5 条):") + done.take(5).forEach { f -> appendLine("- ✓ ${f.text}") } + } + } + } +} diff --git a/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/chat/ModelConfig.kt b/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/chat/ModelConfig.kt new file mode 100644 index 0000000..42f27a4 --- /dev/null +++ b/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/chat/ModelConfig.kt @@ -0,0 +1,173 @@ +package com.lemon.lifelog.chat + +import android.content.Context +import com.lemon.lifelog.BuildConfig +import org.json.JSONArray +import org.json.JSONObject +import java.util.UUID + +data class LlmProvider( + val id: String, + val label: String, + val baseUrl: String, + val apiKey: String, + val chatModel: String, + val builtIn: Boolean = false, +) + +enum class ModelSlot(val key: String, val label: String, val defaultModel: String) { + VL("slot_vl", "图像理解(VL)", "vl"), + EMBED("slot_embed", "语义向量", "embed"), +} + +/** + * 统一服务商注册表 + 任务槽绑定。 + * + * 服务商(LlmProvider)= 一个 OpenAI 兼容端点(baseUrl + apiKey)+ 对话默认模型名。 + * 内置两个:本地 llama-swap (ID_LOCAL) 和 本地 Hermes (ID_HERMES),不可删除。 + * 用户可添加任意远程服务商(通义千问 / Kimi / MiniMax 等)。 + * + * 任务槽(ModelSlot)= 某项能力用哪个 (provider, model): + * - VL:图像理解,PhotoVlWorker 读取 + * - EMBED:语义向量,PhotologViewModel / PhotoVlWorker 读取 + * - CHAT:用哪个 target 聊天 → 由 ChatConfig.targets() + ChatViewModel.selectedTargetId 管理,不在此 + * - SUMMARY:纪要用哪个 target → 同上,存 SummaryWorker.KEY_BACKEND(ChatTarget.id) + * + * 持久化:SharedPreferences "model_config",JSON 数组 + 独立 key,无 Room 依赖。 + */ +object ModelConfig { + const val PREFS = "model_config" + private const val K_PROVIDERS = "user_providers" + private const val K_HERMES_URL = "hermes_url" + private const val K_HERMES_KEY = "hermes_key" + private const val K_HERMES_MODEL = "hermes_model" + + private fun prefs(ctx: Context) = ctx.getSharedPreferences(PREFS, Context.MODE_PRIVATE) + + // ── 内置:Hermes 网关 ────────────────────────────────────────────────────── + + fun hermesProvider(ctx: Context): LlmProvider { + val p = prefs(ctx) + return LlmProvider( + id = ChatConfig.ID_HERMES, + label = "本地 Hermes", + baseUrl = p.getString(K_HERMES_URL, null)?.ifBlank { null } ?: BuildConfig.HERMES_API_URL, + apiKey = p.getString(K_HERMES_KEY, null) ?: BuildConfig.HERMES_API_KEY, + chatModel = p.getString(K_HERMES_MODEL, null)?.ifBlank { null } ?: BuildConfig.HERMES_MODEL, + builtIn = true, + ) + } + + fun saveHermes(ctx: Context, url: String, key: String, model: String) { + prefs(ctx).edit() + .putString(K_HERMES_URL, url.trim()) + .putString(K_HERMES_KEY, key.trim()) + .putString(K_HERMES_MODEL, model.trim()) + .apply() + } + + // ── 内置:本地 llama-swap ───────────────────────────────────────────────── + + fun localSwapProvider(ctx: Context) = LlmProvider( + id = ChatConfig.ID_LOCAL, + label = "本地 llama-swap", + baseUrl = ChatConfig.localBase(ctx), + apiKey = "", + chatModel = "", + builtIn = true, + ) + + // ── 用户添加的远程服务商 ─────────────────────────────────────────────────── + + fun userProviders(ctx: Context): List { + val json = prefs(ctx).getString(K_PROVIDERS, null) ?: return emptyList() + return try { + val arr = JSONArray(json) + (0 until arr.length()).map { arr.getJSONObject(it).toProvider() } + } catch (t: Throwable) { + emptyList() + } + } + + fun allProviders(ctx: Context): List = buildList { + add(localSwapProvider(ctx)) + add(hermesProvider(ctx)) + addAll(userProviders(ctx)) + } + + fun addProvider(ctx: Context, p: LlmProvider) { + saveUserProviders(ctx, userProviders(ctx) + p) + } + + fun updateProvider(ctx: Context, p: LlmProvider) { + if (p.id == ChatConfig.ID_HERMES) { + saveHermes(ctx, p.baseUrl, p.apiKey, p.chatModel) + return + } + if (p.builtIn) return + val list = userProviders(ctx).toMutableList() + val idx = list.indexOfFirst { it.id == p.id } + if (idx >= 0) list[idx] = p else list.add(p) + saveUserProviders(ctx, list) + } + + fun deleteProvider(ctx: Context, id: String) { + if (id == ChatConfig.ID_LOCAL || id == ChatConfig.ID_HERMES) return + saveUserProviders(ctx, userProviders(ctx).filter { it.id != id }) + // 如果某个 slot 引用了被删的 provider,重置回默认 + for (slot in ModelSlot.entries) { + if (slotProviderId(ctx, slot) == id) { + setSlot(ctx, slot, ChatConfig.ID_LOCAL, slot.defaultModel) + } + } + } + + private fun saveUserProviders(ctx: Context, list: List) { + val arr = JSONArray() + list.forEach { arr.put(it.toJson()) } + prefs(ctx).edit().putString(K_PROVIDERS, arr.toString()).apply() + } + + // ── 任务槽 ──────────────────────────────────────────────────────────────── + + fun slotProviderId(ctx: Context, slot: ModelSlot): String = + prefs(ctx).getString("${slot.key}_pid", null)?.ifBlank { null } ?: ChatConfig.ID_LOCAL + + fun slotModel(ctx: Context, slot: ModelSlot): String = + prefs(ctx).getString("${slot.key}_model", null)?.ifBlank { null } ?: slot.defaultModel + + fun setSlot(ctx: Context, slot: ModelSlot, providerId: String, model: String) { + prefs(ctx).edit() + .putString("${slot.key}_pid", providerId) + .putString("${slot.key}_model", model.trim()) + .apply() + } + + /** (baseUrl, apiKey, modelName) 三元组,供 VisionClient / EmbedClient 直接使用。 */ + fun slotEndpoint(ctx: Context, slot: ModelSlot): Triple { + val pid = slotProviderId(ctx, slot) + val model = slotModel(ctx, slot) + val provider = allProviders(ctx).firstOrNull { it.id == pid } ?: localSwapProvider(ctx) + return Triple(provider.baseUrl, provider.apiKey, model) + } + + fun generateId(): String = "p_${UUID.randomUUID().toString().take(8)}" + + // ── JSON 序列化 ──────────────────────────────────────────────────────────── + + private fun JSONObject.toProvider() = LlmProvider( + id = getString("id"), + label = getString("label"), + baseUrl = getString("baseUrl"), + apiKey = optString("apiKey", ""), + chatModel = optString("chatModel", ""), + ) + + private fun LlmProvider.toJson() = JSONObject().apply { + put("id", id) + put("label", label) + put("baseUrl", baseUrl) + put("apiKey", apiKey) + put("chatModel", chatModel) + } +} diff --git a/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/data/Daos.kt b/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/data/Daos.kt index f8ce727..034fcf4 100644 --- a/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/data/Daos.kt +++ b/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/data/Daos.kt @@ -501,6 +501,9 @@ data class PhotoVec( val captionEmbedding: ByteArray, ) +/** 补算 embedding 用的轻投影:已有 caption 但缺 embedding 的照片。 */ +data class CaptionOnly(val id: Long, val caption: String) + /** 某人物名下照片数(相册「人物文件夹」封面 + 计数)。 */ data class PersonPhotoCount( val personId: Long, @@ -749,6 +752,14 @@ interface PhotoDao { /** 按 path 模糊查 id(清理旧版误导入的朋友圈图用,pat 形如 "%/moment__%")。 */ @Query("SELECT id FROM photos WHERE path LIKE :pat") suspend fun idsByPathLike(pat: String): List + + /** 补算向量:已有 caption 但还没算 embedding 的照片(embed 模型后配时回填用)。 */ + @Query("SELECT id, caption FROM photos WHERE caption IS NOT NULL AND caption != '' AND captionEmbedding IS NULL") + suspend fun withCaptionNoEmbedding(): List + + /** 只写 captionEmbedding,不改其他字段(backfill 用,不重跑 VL)。 */ + @Query("UPDATE photos SET captionEmbedding = :emb WHERE id = :id") + suspend fun updateCaptionEmbedding(id: Long, emb: ByteArray) } @Dao diff --git a/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/vision/EmbedClient.kt b/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/vision/EmbedClient.kt index 479499a..fddafa5 100644 --- a/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/vision/EmbedClient.kt +++ b/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/vision/EmbedClient.kt @@ -22,6 +22,7 @@ import kotlin.math.sqrt class EmbedClient( private val baseUrl: String, private val model: String = "embed", + private val apiKey: String = "", ) { companion object { private val JSON = "application/json; charset=utf-8".toMediaType() @@ -38,11 +39,12 @@ class EmbedClient( put("model", model) put("input", text) } - val req = Request.Builder() + val reqBuilder = Request.Builder() .url("$baseUrl/v1/embeddings") .header("Content-Type", "application/json") .post(payload.toString().toRequestBody(JSON)) - .build() + if (apiKey.isNotBlank()) reqBuilder.header("Authorization", "Bearer $apiKey") + val req = reqBuilder.build() http.newCall(req).execute().use { resp -> val body = resp.body?.string().orEmpty() if (!resp.isSuccessful) throw IOException("embed ${resp.code}: ${body.take(200)}") diff --git a/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/vision/PhotoVlWorker.kt b/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/vision/PhotoVlWorker.kt index addcace..0c03eae 100644 --- a/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/vision/PhotoVlWorker.kt +++ b/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/vision/PhotoVlWorker.kt @@ -4,7 +4,8 @@ import android.content.Context import android.util.Log import androidx.work.CoroutineWorker import androidx.work.WorkerParameters -import com.lemon.lifelog.chat.ChatConfig +import com.lemon.lifelog.chat.ModelConfig +import com.lemon.lifelog.chat.ModelSlot import com.lemon.lifelog.data.Repository import java.util.concurrent.atomic.AtomicBoolean @@ -28,9 +29,10 @@ class PhotoVlWorker(ctx: Context, params: WorkerParameters) : CoroutineWorker(ct val pending = repo.photos.pendingCaption() if (pending.isEmpty()) return Result.success() - val base = ChatConfig.localBase(applicationContext) - val vision = VisionClient(base) - val embed = EmbedClient(base, ChatConfig.modelEmbed(applicationContext)) + val (vlBase, vlKey, vlModel) = ModelConfig.slotEndpoint(applicationContext, ModelSlot.VL) + val (embedBase, embedKey, embedModel) = ModelConfig.slotEndpoint(applicationContext, ModelSlot.EMBED) + val vision = VisionClient(vlBase, vlModel, vlKey) + val embed = EmbedClient(embedBase, embedModel, embedKey) val proc = PhotoProcessor(applicationContext, repo) val deadline = System.currentTimeMillis() + RUN_BUDGET_MS var done = 0 diff --git a/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/vision/VisionClient.kt b/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/vision/VisionClient.kt index 05c86ff..3235a16 100644 --- a/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/vision/VisionClient.kt +++ b/lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/vision/VisionClient.kt @@ -17,6 +17,7 @@ import java.util.concurrent.TimeUnit class VisionClient( private val baseUrl: String, private val model: String = "vl", + private val apiKey: String = "", ) { companion object { private val JSON = "application/json; charset=utf-8".toMediaType() @@ -48,11 +49,12 @@ class VisionClient( }) })) } - val req = Request.Builder() + val reqBuilder = Request.Builder() .url("$baseUrl/v1/chat/completions") .header("Content-Type", "application/json") .post(payload.toString().toRequestBody(JSON)) - .build() + if (apiKey.isNotBlank()) reqBuilder.header("Authorization", "Bearer $apiKey") + val req = reqBuilder.build() http.newCall(req).execute().use { resp -> val body = resp.body?.string().orEmpty() if (!resp.isSuccessful) throw IOException("vl ${resp.code}: ${body.take(200)}")