Commit 263efd3399f89c5f7641a33e5d3779b022ad7497
1 parent
0f730253
feat(lifelog/photolog): 全相册自动索引 — 分层管线 + 搜图 + VL 暂停/停止
把整个手机相册自动跑成可搜索内容(v0.8)。本地 VL ~30-40s/张, 全量是按天计的长任务 → 分层:快的全量先铺,慢的 VL 排队慢慢补。 - GalleryScanner:MediaStore 游标扫全量,content:// uri 原地登记不拷文件, mediaId 去重增量;photos 加 source/mediaId/scanState/captionEmbedding(Room v14) - Tier-0 PhotoFaceWorker:扫描 + ML Kit 人脸 + 512d emb + GPS,秒级全量铺, battery-not-low;Tier-1 PhotoVlWorker:VL 场景描述 + caption 语义向量, 优先级队列(有人脸>近期),充电时跑,尊重暂停标志 - 搜图:关键词(caption+人名 LIKE) + 语义(caption 向量余弦,EmbedClient 打 llama-swap embed 模型;未配置优雅降级) - VL 控制(PhotoVlState):运行中「暂停/停止」、空闲「立即理解/继续」切换, 正在处理的格子转圈,格子角标 待索引/待理解/已描述 - 修 ImageUtil.decodeUri:bounds 预解码 decodeStream 在 inJustDecodeBounds 下按设计返回 null,原 `?:` 误判致每张提前 return(人脸全跑空) 迁移 13→14 纯增量:老的已处理照片直接标 scanState=2 不重跑。 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Showing
19 changed files
with
921 additions
and
99 deletions
lifelog-recorder/HANDOFF.md
| @@ -70,10 +70,26 @@ app/src/main/kotlin/com/lemon/lifelog/ | @@ -70,10 +70,26 @@ app/src/main/kotlin/com/lemon/lifelog/ | ||
| 70 | │ ├── HermesClient.kt — OpenAI 兼容(纪要 worker 用,指本地 llama-swap;readTimeout 600s) | 70 | │ ├── HermesClient.kt — OpenAI 兼容(纪要 worker 用,指本地 llama-swap;readTimeout 600s) |
| 71 | │ ├── ModelDownloader.kt — 首次启动下模型(paraformer 232 MB + speaker 28 MB) | 71 | │ ├── ModelDownloader.kt — 首次启动下模型(paraformer 232 MB + speaker 28 MB) |
| 72 | │ └── XfyunIatClient.kt — 讯飞大模型 WS 客户端,仅 fallback / 云复核用 | 72 | │ └── XfyunIatClient.kt — 讯飞大模型 WS 客户端,仅 fallback / 云复核用 |
| 73 | +├── PhotologScreen.kt — 「人物」+「相册」一级 tab(通讯录/档案/照片网格/全相册索引+搜索+VL 控制) | ||
| 74 | +├── PhotologViewModel.kt — photolog 总线(导入/标注/合并/绑声纹/AI 印象/搜图/VL 控制) | ||
| 75 | +├── PersonProfileScreen.kt — 联系人档案(照片/录音/微信聊天/朋友圈/AI 印象,tab 式 + 记录搜索) | ||
| 76 | +├── vision/ — ★ photolog 视觉层(人脸 + 全相册自动索引,见 §四下半) | ||
| 77 | +│ ├── GalleryScanner.kt — 全相册 MediaStore 游标扫描,content:// uri 原地登记(不拷文件),mediaId 去重增量 | ||
| 78 | +│ ├── PhotoFaceWorker.kt — Tier-0:扫描 + ML Kit 人脸 + 512d emb + GPS(快,battery-not-low) | ||
| 79 | +│ ├── PhotoVlWorker.kt — Tier-1:Qwen-VL 场景描述 + caption 语义向量(慢,充电时;尊重暂停标志) | ||
| 80 | +│ ├── PhotoVlState.kt — ★ VL 运行态 singleton(running/currentPhotoId/paused 持久化),UI 控制按钮 + 单张转圈 | ||
| 81 | +│ ├── PhotoScheduler.kt — 两条周期 worker(face 15min / vl 15min+charging)+ runNow/runVlNow/cancelVl | ||
| 82 | +│ ├── PhotoProcessor.kt — faceStage(Tier-0)/ captionStage(Tier-1),按来源解码 | ||
| 83 | +│ ├── FaceEngine.kt — ML Kit 检测+5点对齐 → TFLite MobileFaceNet 512d(非 onnx,避与 sherpa 冲突) | ||
| 84 | +│ ├── FaceMatcher.kt — 人脸归属(人物锚点+已标注脸 gallery,余弦 0.45)+ backfill | ||
| 85 | +│ ├── VisionClient.kt — llama-swap "vl" 模型出场景描述(不报人名) | ||
| 86 | +│ ├── EmbedClient.kt — llama-swap "embed" 模型出 caption 文本向量(语义搜;未配置则优雅降级) | ||
| 87 | +│ ├── ImageUtil.kt — EXIF/解码/缩放/embedding 打包;decode 按 source 走 file 或 content uri | ||
| 88 | +│ └── WeChatEnrollWorker.kt — 微信头像 → 人脸锚点入库 | ||
| 73 | └── data/ | 89 | └── data/ |
| 74 | - ├── Entities.kt — Segment / Transcript / Summary / SpeakerProfile / TranscriptFts / ChatConversation / ChatMessage | ||
| 75 | - ├── Daos.kt — observeTimeline + searchTimeline(FTS4) + Speaker/Summary/ChatDao + 范围查询 | ||
| 76 | - ├── AppDatabase.kt — Room v3,addMigrations(MIGRATION_2_3),★ 已删 fallbackToDestructiveMigration(改 schema 不写迁移会崩,绝不静默清库) | 90 | + ├── Entities.kt — Segment / Transcript / Summary / SpeakerProfile / TranscriptFts / Chat* / Person / Photo / Face / Moment / WechatMessage |
| 91 | + ├── Daos.kt — observeTimeline + searchTimeline(FTS4) + Photo/Face/Person/... DAO;★ photos 分层队列(pendingFace/pendingCaption) + observeProgress + searchByKeyword + captionVecs | ||
| 92 | + ├── AppDatabase.kt — Room **v14**,addMigrations(... MIGRATION_13_14),★ 已删 fallbackToDestructiveMigration(改 schema 不写迁移会崩,绝不静默清库) | ||
| 77 | └── Repository.kt | 93 | └── Repository.kt |
| 78 | ``` | 94 | ``` |
| 79 | 95 | ||
| @@ -188,6 +204,43 @@ VAD 参数(在 `RecorderService.runPipeline` 里调): | @@ -188,6 +204,43 @@ VAD 参数(在 `RecorderService.runPipeline` 里调): | ||
| 188 | 204 | ||
| 189 | --- | 205 | --- |
| 190 | 206 | ||
| 207 | +### photolog 全相册自动索引(v0.8,2026-06-09) | ||
| 208 | + | ||
| 209 | +把手机**整个相册**自动跑成可搜索内容。核心矛盾:本地 VL(Qwen2.5-VL-3B)真机 **~30-40s/张**,几千张全量 VL 是按天计的长任务 → **分层(tiered)**:快的全量先铺,慢的 VL 排队慢慢补。 | ||
| 210 | + | ||
| 211 | +``` | ||
| 212 | +GalleryScanner.scan() ── MediaStore 游标遍历全部图片 | ||
| 213 | + └─ 只存 content:// uri(不拷文件!全相册拷一份会撑爆私有目录)+ mediaId 去重 + DATE_TAKEN | ||
| 214 | + → photos 行 (source='gallery', scanState=0) | ||
| 215 | + │ | ||
| 216 | + ┌──────────────────────────┴───────────────────────────┐ | ||
| 217 | +Tier-0 PhotoFaceWorker (15min, battery-not-low) Tier-1 PhotoVlWorker (15min, +charging) | ||
| 218 | + 每轮先 scan() 增量登记,再处理 scanState=0: 处理 scanState=1,优先级队列(有人脸>近期): | ||
| 219 | + ├─ ImageUtil.decode(ctx,photo) ← 按 source 解码 ├─ VisionClient.caption(jpeg 448px) → 场景描述 | ||
| 220 | + ├─ FaceEngine 人脸+512d emb + FaceMatcher 认人 ├─ EmbedClient.embed(caption) → 语义向量(可选) | ||
| 221 | + ├─ ImageUtil.readGps (ACCESS_MEDIA_LOCATION) └─ markCaptionDone → scanState=2 + processedAt | ||
| 222 | + └─ markFaceDone → scanState=1 失败 markCaptionSkipped(=2,caption 空) 不卡队列 | ||
| 223 | + 秒级,几小时铺满全库 → 立刻可按人/时间搜 每张 7min 预算 + retry,避开 WM 10min 上限 | ||
| 224 | +``` | ||
| 225 | + | ||
| 226 | +**scanState 状态机**:`0 待索引 → 1 人脸/时间已铺 → 2 已出场景描述`。迁移 13→14 把老的已处理照片(`processedAt` 非空)直接标 2 不重跑;imported 源(手动导入,拷进私有目录)走同一管线,只是 `source='imported'` 且 `decode` 走 file path。 | ||
| 227 | + | ||
| 228 | +**搜索**:① 关键词 `searchByKeyword` = caption + 人名 LIKE 子串(中文子串比 FTS4 unicode61 靠谱,跟 transcript 的 searchTextLike 一致);② 语义 `searchSemantic` = embed query → 全库 `captionVecs()` 余弦 top-K(Kotlin 算,floor 0.20)。语义要 llama-swap 配 `embed` 模型(`--embedding` 的中文 GGUF,如 bge-small-zh),没配 `EmbedClient` 抛 IOException,UI 提示「未配置 embed 端点」,不影响关键词搜。 | ||
| 229 | + | ||
| 230 | +**VL 控制(PhotoVlState)**:worker 推 `running`/`currentPhotoId`,相册页据此: | ||
| 231 | +- 运行中 → 顶部「⏸ 暂停」「⏹ 停止」+ 正在 VL 的那格半透明遮罩转圈; | ||
| 232 | +- 空闲 → 「⚡ 立即理解」(暂停且未跑完 → 「▶ 继续理解」); | ||
| 233 | +- **暂停** = `paused` 持久化到 prefs,worker 每张前 `isPaused()` 检查 break,**连充电时的周期 worker 也尊重**,需「继续」才恢复;**停止** = cancelVl 取消手动这轮 + 清 paused(之后充电仍自动补完)。 | ||
| 234 | +- 格子角标:`👤N` 人脸数 / `📝` 已出描述(可搜) / `待索引·待理解·无描述` 未处理态。 | ||
| 235 | + | ||
| 236 | +> ⚠️ 关键坑(已修):`ImageUtil.decodeUri` 里 bounds 预解码用 `inJustDecodeBounds=true`,`BitmapFactory.decodeStream` **按设计返回 null**(只填 bounds)——不能拿它的返回值判失败,否则**每张都提前 return null**,人脸全跑空。`?:` 要判 `openInputStream` 的 null。 | ||
| 237 | + | ||
| 238 | +> ⚠️ WM worker 互斥:periodic + 手动 + retry 可能并发,多 VL worker 打同一 llama-server 会 502 → `PhotoVlWorker`/`PhotoFaceWorker` 各有 `AtomicBoolean isRunning` 守。`PhotoVlState.running` 只由拿到锁那个置位。 | ||
| 239 | + | ||
| 240 | +> 全相册扫描没有 sqlite3 在设备上:dev 改 photos 表用 `adb exec-out run-as com.lemon.lifelog cat databases/lifelog.db{,-wal,-shm}` 拉到 Mac,`sqlite3 'PRAGMA wal_checkpoint(TRUNCATE);'` 合并 WAL 后改,再 `run-as sh -c 'cat > databases/lifelog.db'` 推回 + 删 wal/shm。 | ||
| 241 | + | ||
| 242 | +--- | ||
| 243 | + | ||
| 191 | ## §五 §六 待办(按优先级) | 244 | ## §五 §六 待办(按优先级) |
| 192 | 245 | ||
| 193 | ✅ **已完成**: | 246 | ✅ **已完成**: |
| @@ -196,7 +249,8 @@ VAD 参数(在 `RecorderService.runPipeline` 里调): | @@ -196,7 +249,8 @@ VAD 参数(在 `RecorderService.runPipeline` 里调): | ||
| 196 | - v0.4 (2026-06):段播放 (MediaPlayer Opus) / bottom sheet 操作菜单 / enroll 后自动回填老段 | 249 | - v0.4 (2026-06):段播放 (MediaPlayer Opus) / bottom sheet 操作菜单 / enroll 后自动回填老段 |
| 197 | - v0.5 (2026-06):hermes api_server 端点 + SummaryWorker + 纪要 UI section | 250 | - v0.5 (2026-06):hermes api_server 端点 + SummaryWorker + 纪要 UI section |
| 198 | - v0.6 (2026-06-04):**声纹在线聚类**(SpeakerClusterer,自动「说话人N」+ reclusterAll)/ **5-tab UI**(录音/对话/时间轴/声纹/纪要,时间轴全 CRUD + 批量删导出 + 按说话人/时间/未匹配筛选 + 排序)/ **真暂停**(停采集留模型)/ **会话级纪要**(按时间间隔切会话、可调阈值、要点带 [HH:mm:ss] 点击跳回时间轴、折叠头显示说话人名单)/ **App 内对话**(本地 2B/4B + Hermes 多端点流式)/ **本地 LLM 栈**(llama.cpp HEAD + llama-swap 8080,纪要+对话全本地,见 §六)/ **Room v3 迁移**(去 destructive 改 fail-loud) | 251 | - v0.6 (2026-06-04):**声纹在线聚类**(SpeakerClusterer,自动「说话人N」+ reclusterAll)/ **5-tab UI**(录音/对话/时间轴/声纹/纪要,时间轴全 CRUD + 批量删导出 + 按说话人/时间/未匹配筛选 + 排序)/ **真暂停**(停采集留模型)/ **会话级纪要**(按时间间隔切会话、可调阈值、要点带 [HH:mm:ss] 点击跳回时间轴、折叠头显示说话人名单)/ **App 内对话**(本地 2B/4B + Hermes 多端点流式)/ **本地 LLM 栈**(llama.cpp HEAD + llama-swap 8080,纪要+对话全本地,见 §六)/ **Room v3 迁移**(去 destructive 改 fail-loud) |
| 199 | -- v0.7 (2026-06-07):**block-based diarization-first 录音管线**(攒块→整块 pyannote 分轮→逐 turn 转写+嵌入+认人,turn=时间轴一行;去实时字幕;BlockBuffer/BlockProcessor;崩溃 listPending 续跑;commit 07f3d22)/ **声纹纠偏 L1+L2**(L1=reclusterAll & assign 冻结命名 anchor 不漂移止串人;L2=重命名/合并时 rebuildAnchor 用该人全部段均值重建多 anchor)/ 阈值默认 0.45、设置页 0.35-0.65 可调 | 252 | +- v0.7 (2026-06-07):**block-based diarization-first 录音管线**(攒块→整块 pyannote 分轮→逐 turn 转写+嵌入+认人,turn=时间轴一行;去实时字幕;BlockBuffer/BlockProcessor;崩溃 listPending 续跑;commit 07f3d22)/ **声纹纠偏 L1+L2**(L1=reclusterAll & assign 冻结命名 anchor 不漂移止串人;L2=重命名/合并时 rebuildAnchor 用该人全部段均值重建多 anchor)/ 阈值默认 0.45、设置页 0.35-0.65 可调 / **photolog 人脉 CRM 层**(人物档案 + 相册人脸聚类 + 微信通讯录/朋友圈/AI 印象) |
| 253 | +- v0.8 (2026-06-09):**全相册自动索引 + 分层管线**(GalleryScanner 扫 MediaStore 全量、content uri 原地登记不拷文件;Tier-0 人脸/时间秒级全量铺、Tier-1 VL 充电时排队补;Room v14;详见 §四 photolog 段)/ **搜图**(关键词 caption+人名 LIKE / 语义 caption 向量余弦,embed 走 llama-swap)/ **VL 控制**(PhotoVlState:暂停/继续/停止切换 + 单张转圈 + 格子状态角标) | ||
| 200 | 254 | ||
| 201 | 剩下: | 255 | 剩下: |
| 202 | 256 | ||
| @@ -315,6 +369,9 @@ openrouter/nous metadata 超时),对开机服务可接受。 | @@ -315,6 +369,9 @@ openrouter/nous metadata 超时),对开机服务可接受。 | ||
| 315 | | hf-mirror.com 偶尔 truncate | HF mirror 限速 / 中断 | `ModelDownloader` 已带断点续传 | | 369 | | hf-mirror.com 偶尔 truncate | HF mirror 限速 / 中断 | `ModelDownloader` 已带断点续传 | |
| 316 | | `am start-foreground-service` 返回 "Requires permission not exported" | 服务 `android:exported="false"`,adb shell 没权 | 用 `su -c "am ..."` 提权 | | 370 | | `am start-foreground-service` 返回 "Requires permission not exported" | 服务 `android:exported="false"`,adb shell 没权 | 用 `su -c "am ..."` 提权 | |
| 317 | | `adb shell 'su 10288 -c "..."` 网络全断 | Magisk su 没传 supplementary groups | 加 `-G 3003 -G 9997 -G 20<n> -G 50<n>` | | 371 | | `adb shell 'su 10288 -c "..."` 网络全断 | Magisk su 没传 supplementary groups | 加 `-G 3003 -G 9997 -G 20<n> -G 50<n>` | |
| 372 | +| 全相册照片人脸全 0、`decode failed (face)` 刷屏 | `decodeUri` bounds 预解码处 `decodeStream(inJustDecodeBounds=true)` **按设计返回 null**,被 `?:` 当失败提前 return | `?:` 改判 `openInputStream` 的 null,不是 `decodeStream` 返回值(已修,见 `ImageUtil.decodeUri`) | | ||
| 373 | +| 语义搜 `embed 404: no router for requested model` | llama-swap 没配 `embed` 模型 | config.yaml 加一个 `embed` model(`llama-server --embedding` + 中文 GGUF);不配则只用关键词搜 | | ||
| 374 | +| VL 进度不动 / 点「立即理解」像没反应 | 单张 VL ~30-40s(含 llama-swap 热加载 vl 模型),且要 battery-not-low | 正常现象;看「场景描述 N/总」进度条;想自动补完插充电(周期 worker 要 charging) | | ||
| 318 | 375 | ||
| 319 | --- | 376 | --- |
| 320 | 377 |
lifelog-recorder/app/src/main/AndroidManifest.xml
| @@ -12,6 +12,12 @@ | @@ -12,6 +12,12 @@ | ||
| 12 | <uses-permission android:name="android.permission.INTERNET" /> | 12 | <uses-permission android:name="android.permission.INTERNET" /> |
| 13 | <uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" /> | 13 | <uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" /> |
| 14 | <uses-permission android:name="android.permission.READ_CONTACTS" /> | 14 | <uses-permission android:name="android.permission.READ_CONTACTS" /> |
| 15 | + <!-- 全相册自动索引:读图片(13+ 用 READ_MEDIA_IMAGES,12 及以下用 READ_EXTERNAL_STORAGE) --> | ||
| 16 | + <uses-permission android:name="android.permission.READ_MEDIA_IMAGES" /> | ||
| 17 | + <uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" | ||
| 18 | + android:maxSdkVersion="32" /> | ||
| 19 | + <!-- 读照片 EXIF GPS(脱敏原图需要) --> | ||
| 20 | + <uses-permission android:name="android.permission.ACCESS_MEDIA_LOCATION" /> | ||
| 15 | 21 | ||
| 16 | <application | 22 | <application |
| 17 | android:name=".LifelogApp" | 23 | android:name=".LifelogApp" |
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/LifelogApp.kt
| @@ -23,6 +23,7 @@ class LifelogApp : Application() { | @@ -23,6 +23,7 @@ class LifelogApp : Application() { | ||
| 23 | SummaryScheduler.schedulePeriodic(this) | 23 | SummaryScheduler.schedulePeriodic(this) |
| 24 | // 相册:未处理照片的周期后台处理(人脸+VL),空闲时跑,不阻塞导入。 | 24 | // 相册:未处理照片的周期后台处理(人脸+VL),空闲时跑,不阻塞导入。 |
| 25 | com.lemon.lifelog.vision.PhotoScheduler.schedulePeriodic(this) | 25 | com.lemon.lifelog.vision.PhotoScheduler.schedulePeriodic(this) |
| 26 | + com.lemon.lifelog.vision.PhotoVlState.init(this) // 载入持久化的「暂停」标志 | ||
| 26 | // 存储 TTL:每天清理超期录音 .opus(文字永留);默认关闭,设置页选保留天数才生效。 | 27 | // 存储 TTL:每天清理超期录音 .opus(文字永留);默认关闭,设置页选保留天数才生效。 |
| 27 | com.lemon.lifelog.storage.StorageScheduler.schedulePeriodic(this) | 28 | com.lemon.lifelog.storage.StorageScheduler.schedulePeriodic(this) |
| 28 | } | 29 | } |
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/PersonProfileScreen.kt
| @@ -148,7 +148,7 @@ internal fun PersonProfileScreen( | @@ -148,7 +148,7 @@ internal fun PersonProfileScreen( | ||
| 148 | ) { | 148 | ) { |
| 149 | gridItems(fPhotos, key = { it.id }) { ph -> | 149 | gridItems(fPhotos, key = { it.id }) { ph -> |
| 150 | AsyncImage( | 150 | AsyncImage( |
| 151 | - model = File(ph.path), contentDescription = ph.caption, contentScale = ContentScale.Crop, | 151 | + model = photoModel(ph.source, ph.path), contentDescription = ph.caption, contentScale = ContentScale.Crop, |
| 152 | modifier = Modifier.aspectRatio(1f).clip(RoundedCornerShape(6.dp)).clickable { onOpenPhoto(ph.id) }, | 152 | modifier = Modifier.aspectRatio(1f).clip(RoundedCornerShape(6.dp)).clickable { onOpenPhoto(ph.id) }, |
| 153 | ) | 153 | ) |
| 154 | } | 154 | } |
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/PhotologScreen.kt
| @@ -2,6 +2,8 @@ package com.lemon.lifelog | @@ -2,6 +2,8 @@ package com.lemon.lifelog | ||
| 2 | 2 | ||
| 3 | import android.Manifest | 3 | import android.Manifest |
| 4 | import android.content.pm.PackageManager | 4 | import android.content.pm.PackageManager |
| 5 | +import android.net.Uri | ||
| 6 | +import android.os.Build | ||
| 5 | import androidx.activity.compose.rememberLauncherForActivityResult | 7 | import androidx.activity.compose.rememberLauncherForActivityResult |
| 6 | import androidx.activity.result.PickVisualMediaRequest | 8 | import androidx.activity.result.PickVisualMediaRequest |
| 7 | import androidx.activity.result.contract.ActivityResultContracts | 9 | import androidx.activity.result.contract.ActivityResultContracts |
| @@ -22,6 +24,7 @@ import androidx.compose.foundation.layout.fillMaxWidth | @@ -22,6 +24,7 @@ import androidx.compose.foundation.layout.fillMaxWidth | ||
| 22 | import androidx.compose.foundation.layout.height | 24 | import androidx.compose.foundation.layout.height |
| 23 | import androidx.compose.foundation.layout.padding | 25 | import androidx.compose.foundation.layout.padding |
| 24 | import androidx.compose.foundation.layout.size | 26 | import androidx.compose.foundation.layout.size |
| 27 | +import androidx.compose.foundation.layout.width | ||
| 25 | import androidx.compose.foundation.shape.CircleShape | 28 | import androidx.compose.foundation.shape.CircleShape |
| 26 | import androidx.compose.ui.draw.clip | 29 | import androidx.compose.ui.draw.clip |
| 27 | import androidx.compose.ui.window.Dialog | 30 | import androidx.compose.ui.window.Dialog |
| @@ -41,6 +44,7 @@ import androidx.compose.material3.CircularProgressIndicator | @@ -41,6 +44,7 @@ import androidx.compose.material3.CircularProgressIndicator | ||
| 41 | import androidx.compose.material3.ExperimentalMaterial3Api | 44 | import androidx.compose.material3.ExperimentalMaterial3Api |
| 42 | import androidx.compose.material3.FilterChip | 45 | import androidx.compose.material3.FilterChip |
| 43 | import androidx.compose.material3.HorizontalDivider | 46 | import androidx.compose.material3.HorizontalDivider |
| 47 | +import androidx.compose.material3.LinearProgressIndicator | ||
| 44 | import androidx.compose.material3.MaterialTheme | 48 | import androidx.compose.material3.MaterialTheme |
| 45 | import androidx.compose.material3.ModalBottomSheet | 49 | import androidx.compose.material3.ModalBottomSheet |
| 46 | import androidx.compose.material3.OutlinedButton | 50 | import androidx.compose.material3.OutlinedButton |
| @@ -70,9 +74,12 @@ import androidx.lifecycle.viewmodel.compose.viewModel | @@ -70,9 +74,12 @@ import androidx.lifecycle.viewmodel.compose.viewModel | ||
| 70 | import coil.compose.AsyncImage | 74 | import coil.compose.AsyncImage |
| 71 | import com.lemon.lifelog.data.Face | 75 | import com.lemon.lifelog.data.Face |
| 72 | import com.lemon.lifelog.data.Person | 76 | import com.lemon.lifelog.data.Person |
| 77 | +import com.lemon.lifelog.data.PhotoProgress | ||
| 73 | import com.lemon.lifelog.data.PhotoRow | 78 | import com.lemon.lifelog.data.PhotoRow |
| 74 | import com.lemon.lifelog.data.Repository | 79 | import com.lemon.lifelog.data.Repository |
| 80 | +import com.lemon.lifelog.vision.PhotoVlState | ||
| 75 | import kotlinx.coroutines.Dispatchers | 81 | import kotlinx.coroutines.Dispatchers |
| 82 | +import kotlinx.coroutines.delay | ||
| 76 | import kotlinx.coroutines.flow.flowOf | 83 | import kotlinx.coroutines.flow.flowOf |
| 77 | import kotlinx.coroutines.withContext | 84 | import kotlinx.coroutines.withContext |
| 78 | import java.io.File | 85 | import java.io.File |
| @@ -122,38 +129,121 @@ internal fun ContactsTab(repo: Repository) { | @@ -122,38 +129,121 @@ internal fun ContactsTab(repo: Repository) { | ||
| 122 | } | 129 | } |
| 123 | } | 130 | } |
| 124 | 131 | ||
| 125 | -/** 「相册」一级 tab:照片网格(导入 + 详情 + 标注)。 */ | 132 | +/** PhotoRow → Coil model:gallery 源是 content uri,imported 源是私有文件。 */ |
| 133 | +internal fun photoModel(source: String, path: String): Any = | ||
| 134 | + if (source == "gallery") Uri.parse(path) else File(path) | ||
| 135 | + | ||
| 136 | +/** 「相册」一级 tab:全相册自动索引 + 进度 + 搜索 + 网格(详情 + 标注)。 */ | ||
| 126 | @OptIn(ExperimentalMaterial3Api::class) | 137 | @OptIn(ExperimentalMaterial3Api::class) |
| 127 | @Composable | 138 | @Composable |
| 128 | internal fun AlbumTab(repo: Repository) { | 139 | internal fun AlbumTab(repo: Repository) { |
| 129 | val vm: PhotologViewModel = viewModel() | 140 | val vm: PhotologViewModel = viewModel() |
| 141 | + val ctx = LocalContext.current | ||
| 130 | var detailPhotoId by remember { mutableStateOf<Long?>(null) } | 142 | var detailPhotoId by remember { mutableStateOf<Long?>(null) } |
| 131 | val persons by vm.repo.persons.observeAll().collectAsState(initial = emptyList()) | 143 | val persons by vm.repo.persons.observeAll().collectAsState(initial = emptyList()) |
| 132 | - val pendingCount by vm.repo.photos.observeUnprocessedCount().collectAsState(initial = 0) | 144 | + val progress by vm.repo.photos.observeProgress().collectAsState(initial = PhotoProgress(0, 0, 0)) |
| 133 | val photos by remember { vm.repo.photos.observeGrid(1000) }.collectAsState(initial = emptyList()) | 145 | val photos by remember { vm.repo.photos.observeGrid(1000) }.collectAsState(initial = emptyList()) |
| 146 | + // VL 运行态:控制按钮切换 + 单张处理指示。 | ||
| 147 | + val vlRunning by PhotoVlState.running.collectAsState() | ||
| 148 | + val vlPaused by PhotoVlState.paused.collectAsState() | ||
| 149 | + val vlCurrentId by PhotoVlState.currentPhotoId.collectAsState() | ||
| 150 | + | ||
| 151 | + // 读相册权限(13+ READ_MEDIA_IMAGES;12 及以下 READ_EXTERNAL_STORAGE)。授予即扫全相册。 | ||
| 152 | + val mediaPerm = if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.TIRAMISU) | ||
| 153 | + Manifest.permission.READ_MEDIA_IMAGES else Manifest.permission.READ_EXTERNAL_STORAGE | ||
| 154 | + var granted by remember { | ||
| 155 | + mutableStateOf(ContextCompat.checkSelfPermission(ctx, mediaPerm) == PackageManager.PERMISSION_GRANTED) | ||
| 156 | + } | ||
| 157 | + val permLauncher = rememberLauncherForActivityResult( | ||
| 158 | + ActivityResultContracts.RequestMultiplePermissions(), | ||
| 159 | + ) { res -> granted = res[mediaPerm] == true } | ||
| 160 | + LaunchedEffect(granted) { if (granted) vm.scanGallery() } | ||
| 161 | + // 读图片 + 读 EXIF GPS(ACCESS_MEDIA_LOCATION 是独立的危险权限,要一起请求)。 | ||
| 162 | + val requestPerms = remember(mediaPerm) { | ||
| 163 | + arrayOf(mediaPerm, Manifest.permission.ACCESS_MEDIA_LOCATION) | ||
| 164 | + } | ||
| 165 | + | ||
| 166 | + // 搜索:results=null 显示全部网格;非 null 显示搜索结果。 | ||
| 167 | + var query by remember { mutableStateOf("") } | ||
| 168 | + var results by remember { mutableStateOf<List<PhotoRow>?>(null) } | ||
| 169 | + var searchMsg by remember { mutableStateOf<String?>(null) } | ||
| 170 | + // 关键词即时搜(去抖 300ms)。语义搜走显式按钮。 | ||
| 171 | + LaunchedEffect(query) { | ||
| 172 | + if (query.isBlank()) { results = null; searchMsg = null; return@LaunchedEffect } | ||
| 173 | + delay(300) | ||
| 174 | + vm.searchKeyword(query) { results = it; searchMsg = "关键词命中 ${it.size}" } | ||
| 175 | + } | ||
| 134 | 176 | ||
| 135 | val picker = rememberLauncherForActivityResult( | 177 | val picker = rememberLauncherForActivityResult( |
| 136 | ActivityResultContracts.PickMultipleVisualMedia(30), | 178 | ActivityResultContracts.PickMultipleVisualMedia(30), |
| 137 | ) { uris -> vm.importUris(uris) } | 179 | ) { uris -> vm.importUris(uris) } |
| 138 | 180 | ||
| 181 | + val shown = results ?: photos | ||
| 182 | + | ||
| 139 | Column(modifier = Modifier.fillMaxSize()) { | 183 | Column(modifier = Modifier.fillMaxSize()) { |
| 184 | + // 权限 + 进度。 | ||
| 185 | + if (!granted) { | ||
| 186 | + Row( | ||
| 187 | + modifier = Modifier.fillMaxWidth().padding(horizontal = 12.dp, vertical = 6.dp), | ||
| 188 | + verticalAlignment = Alignment.CenterVertically, horizontalArrangement = Arrangement.spacedBy(8.dp), | ||
| 189 | + ) { | ||
| 190 | + Button(onClick = { permLauncher.launch(requestPerms) }) { Text("开启全相册索引") } | ||
| 191 | + Text("授予读相册权限后自动扫描全部照片", style = MaterialTheme.typography.labelSmall) | ||
| 192 | + } | ||
| 193 | + } else { | ||
| 194 | + ProgressHeader( | ||
| 195 | + p = progress, running = vlRunning, paused = vlPaused, | ||
| 196 | + onRun = { | ||
| 197 | + vm.scanGallery(); vm.runVlNow() | ||
| 198 | + android.widget.Toast.makeText( | ||
| 199 | + ctx, "已开始场景理解(每张约 30–40 秒,看「场景描述」进度)", android.widget.Toast.LENGTH_LONG, | ||
| 200 | + ).show() | ||
| 201 | + }, | ||
| 202 | + onPause = { vm.pauseVl() }, | ||
| 203 | + onStop = { vm.stopVl() }, | ||
| 204 | + ) | ||
| 205 | + } | ||
| 206 | + // 搜索框 + 语义按钮。 | ||
| 140 | Row( | 207 | Row( |
| 141 | - modifier = Modifier.fillMaxWidth().padding(horizontal = 12.dp, vertical = 6.dp), | ||
| 142 | - verticalAlignment = Alignment.CenterVertically, | ||
| 143 | - horizontalArrangement = Arrangement.spacedBy(8.dp), | 208 | + modifier = Modifier.fillMaxWidth().padding(horizontal = 12.dp, vertical = 4.dp), |
| 209 | + verticalAlignment = Alignment.CenterVertically, horizontalArrangement = Arrangement.spacedBy(8.dp), | ||
| 144 | ) { | 210 | ) { |
| 145 | - Button(onClick = { | 211 | + OutlinedTextField( |
| 212 | + value = query, onValueChange = { query = it }, | ||
| 213 | + placeholder = { Text("搜照片:场景 / 人名 / 物体…") }, singleLine = true, | ||
| 214 | + modifier = Modifier.weight(1f), | ||
| 215 | + ) | ||
| 216 | + Button( | ||
| 217 | + enabled = query.isNotBlank(), | ||
| 218 | + onClick = { | ||
| 219 | + searchMsg = "语义搜索中…" | ||
| 220 | + vm.searchSemantic( | ||
| 221 | + query, | ||
| 222 | + onResult = { results = it; searchMsg = "语义命中 ${it.size}" }, | ||
| 223 | + onError = { searchMsg = "⚠ $it" }, | ||
| 224 | + ) | ||
| 225 | + }, | ||
| 226 | + ) { Text("🧠语义") } | ||
| 227 | + } | ||
| 228 | + Row( | ||
| 229 | + modifier = Modifier.fillMaxWidth().padding(horizontal = 12.dp).padding(bottom = 4.dp), | ||
| 230 | + verticalAlignment = Alignment.CenterVertically, horizontalArrangement = Arrangement.spacedBy(8.dp), | ||
| 231 | + ) { | ||
| 232 | + OutlinedButton(onClick = { | ||
| 146 | picker.launch(PickVisualMediaRequest(ActivityResultContracts.PickVisualMedia.ImageOnly)) | 233 | picker.launch(PickVisualMediaRequest(ActivityResultContracts.PickVisualMedia.ImageOnly)) |
| 147 | - }) { Text("+ 导入照片") } | ||
| 148 | - if (pendingCount > 0) { | ||
| 149 | - CircularProgressIndicator(modifier = Modifier.height(18.dp), strokeWidth = 2.dp) | ||
| 150 | - Text("待处理 $pendingCount(后台识别中)", style = MaterialTheme.typography.labelSmall) | ||
| 151 | - } | 234 | + }) { Text("+ 手动导入") } |
| 235 | + searchMsg?.let { Text(it, style = MaterialTheme.typography.labelSmall) } | ||
| 152 | } | 236 | } |
| 153 | - if (photos.isEmpty()) { | 237 | + HorizontalDivider() |
| 238 | + if (shown.isEmpty()) { | ||
| 154 | Box(Modifier.fillMaxSize().padding(24.dp), contentAlignment = Alignment.Center) { | 239 | Box(Modifier.fillMaxSize().padding(24.dp), contentAlignment = Alignment.Center) { |
| 155 | Text( | 240 | Text( |
| 156 | - if (pendingCount > 0) "已导入 $pendingCount 张,后台识别中…稍后刷新" else "还没有照片。点「+ 导入照片」加入相册。\n导入瞬时完成,人脸+场景识别后台定时处理。", | 241 | + when { |
| 242 | + results != null -> "没搜到。换个词,或等场景描述(VL)跑得更多再搜。" | ||
| 243 | + !granted -> "点上方「开启全相册索引」,自动把手机相册所有照片纳入。" | ||
| 244 | + progress.total > 0 -> "正在索引 ${progress.total} 张…稍后刷新。" | ||
| 245 | + else -> "还没有照片。开启全相册索引,或「+ 手动导入」。" | ||
| 246 | + }, | ||
| 157 | style = MaterialTheme.typography.bodyMedium, | 247 | style = MaterialTheme.typography.bodyMedium, |
| 158 | ) | 248 | ) |
| 159 | } | 249 | } |
| @@ -164,7 +254,9 @@ internal fun AlbumTab(repo: Repository) { | @@ -164,7 +254,9 @@ internal fun AlbumTab(repo: Repository) { | ||
| 164 | verticalArrangement = Arrangement.spacedBy(4.dp), | 254 | verticalArrangement = Arrangement.spacedBy(4.dp), |
| 165 | horizontalArrangement = Arrangement.spacedBy(4.dp), | 255 | horizontalArrangement = Arrangement.spacedBy(4.dp), |
| 166 | ) { | 256 | ) { |
| 167 | - gridItems(photos, key = { it.id }) { p -> PhotoCell(p) { detailPhotoId = p.id } } | 257 | + gridItems(shown, key = { it.id }) { p -> |
| 258 | + PhotoCell(p, processing = p.id == vlCurrentId) { detailPhotoId = p.id } | ||
| 259 | + } | ||
| 168 | } | 260 | } |
| 169 | } | 261 | } |
| 170 | } | 262 | } |
| @@ -173,15 +265,118 @@ internal fun AlbumTab(repo: Repository) { | @@ -173,15 +265,118 @@ internal fun AlbumTab(repo: Repository) { | ||
| 173 | } | 265 | } |
| 174 | } | 266 | } |
| 175 | 267 | ||
| 268 | +/** | ||
| 269 | + * 分层进度:人脸(Tier-0)+ 场景描述(Tier-1 VL)两条进度条 + VL 控制。 | ||
| 270 | + * 运行中显示「暂停 / 停止」,空闲显示「立即理解」(暂停且未跑完时显示「继续」)。 | ||
| 271 | + */ | ||
| 272 | +@Composable | ||
| 273 | +private fun ProgressHeader( | ||
| 274 | + p: PhotoProgress, | ||
| 275 | + running: Boolean, | ||
| 276 | + paused: Boolean, | ||
| 277 | + onRun: () -> Unit, | ||
| 278 | + onPause: () -> Unit, | ||
| 279 | + onStop: () -> Unit, | ||
| 280 | +) { | ||
| 281 | + if (p.total == 0) { | ||
| 282 | + Row( | ||
| 283 | + modifier = Modifier.fillMaxWidth().padding(horizontal = 12.dp, vertical = 6.dp), | ||
| 284 | + verticalAlignment = Alignment.CenterVertically, horizontalArrangement = Arrangement.spacedBy(8.dp), | ||
| 285 | + ) { | ||
| 286 | + CircularProgressIndicator(modifier = Modifier.height(16.dp), strokeWidth = 2.dp) | ||
| 287 | + Text("等待扫描相册…", style = MaterialTheme.typography.labelSmall) | ||
| 288 | + } | ||
| 289 | + return | ||
| 290 | + } | ||
| 291 | + Column(modifier = Modifier.fillMaxWidth().padding(horizontal = 12.dp, vertical = 6.dp)) { | ||
| 292 | + Row(verticalAlignment = Alignment.CenterVertically, horizontalArrangement = Arrangement.spacedBy(8.dp)) { | ||
| 293 | + Text("人脸/时间 ${p.faceDone}/${p.total}", style = MaterialTheme.typography.labelSmall, modifier = Modifier.width(120.dp)) | ||
| 294 | + LinearProgressIndicator( | ||
| 295 | + progress = { if (p.total > 0) p.faceDone.toFloat() / p.total else 0f }, | ||
| 296 | + modifier = Modifier.weight(1f), | ||
| 297 | + ) | ||
| 298 | + } | ||
| 299 | + Spacer(Modifier.height(4.dp)) | ||
| 300 | + Row(verticalAlignment = Alignment.CenterVertically, horizontalArrangement = Arrangement.spacedBy(8.dp)) { | ||
| 301 | + Text("场景描述 ${p.captionDone}/${p.total}", style = MaterialTheme.typography.labelSmall, modifier = Modifier.width(120.dp)) | ||
| 302 | + LinearProgressIndicator( | ||
| 303 | + progress = { if (p.total > 0) p.captionDone.toFloat() / p.total else 0f }, | ||
| 304 | + modifier = Modifier.weight(1f), | ||
| 305 | + ) | ||
| 306 | + } | ||
| 307 | + Spacer(Modifier.height(6.dp)) | ||
| 308 | + if (running) { | ||
| 309 | + Row(verticalAlignment = Alignment.CenterVertically, horizontalArrangement = Arrangement.spacedBy(8.dp)) { | ||
| 310 | + CircularProgressIndicator(modifier = Modifier.size(16.dp), strokeWidth = 2.dp) | ||
| 311 | + Button(onClick = onPause) { Text("⏸ 暂停") } | ||
| 312 | + OutlinedButton(onClick = onStop) { Text("⏹ 停止") } | ||
| 313 | + Text("理解中 ${p.captionDone}/${p.total}", style = MaterialTheme.typography.labelSmall) | ||
| 314 | + } | ||
| 315 | + } else if (p.captionDone < p.total) { | ||
| 316 | + Row(verticalAlignment = Alignment.CenterVertically, horizontalArrangement = Arrangement.spacedBy(8.dp)) { | ||
| 317 | + Button(onClick = onRun) { Text(if (paused) "▶ 继续理解" else "⚡ 立即理解") } | ||
| 318 | + if (paused) OutlinedButton(onClick = onStop) { Text("⏹ 停止") } | ||
| 319 | + Text(if (paused) "已暂停" else "默认充电时跑", style = MaterialTheme.typography.labelSmall) | ||
| 320 | + } | ||
| 321 | + } else { | ||
| 322 | + Text("✓ 全部已理解", style = MaterialTheme.typography.labelSmall) | ||
| 323 | + } | ||
| 324 | + } | ||
| 325 | +} | ||
| 326 | + | ||
| 327 | +/** 角标:按 scanState/处理态显示「转圈 / 待理解 / 待索引」。 */ | ||
| 176 | @Composable | 328 | @Composable |
| 177 | -private fun PhotoCell(p: PhotoRow, onClick: () -> Unit) { | 329 | +private fun PhotoCell(p: PhotoRow, processing: Boolean, onClick: () -> Unit) { |
| 178 | Box(modifier = Modifier.aspectRatio(1f).clickable(onClick = onClick)) { | 330 | Box(modifier = Modifier.aspectRatio(1f).clickable(onClick = onClick)) { |
| 179 | AsyncImage( | 331 | AsyncImage( |
| 180 | - model = File(p.path), | 332 | + model = photoModel(p.source, p.path), |
| 181 | contentDescription = null, | 333 | contentDescription = null, |
| 182 | contentScale = ContentScale.Crop, | 334 | contentScale = ContentScale.Crop, |
| 183 | modifier = Modifier.fillMaxSize(), | 335 | modifier = Modifier.fillMaxSize(), |
| 184 | ) | 336 | ) |
| 337 | + // 正在 VL 这张:半透明遮罩 + 转圈。 | ||
| 338 | + if (processing) { | ||
| 339 | + Box( | ||
| 340 | + modifier = Modifier.fillMaxSize().background(MaterialTheme.colorScheme.scrim.copy(alpha = 0.45f)), | ||
| 341 | + contentAlignment = Alignment.Center, | ||
| 342 | + ) { | ||
| 343 | + CircularProgressIndicator(modifier = Modifier.size(22.dp), strokeWidth = 2.dp, color = MaterialTheme.colorScheme.onPrimary) | ||
| 344 | + } | ||
| 345 | + } | ||
| 346 | + // 左下角状态角标:0 待索引 / 1 待理解 / 2(无 caption) 无描述;已理解(有 caption)不标。 | ||
| 347 | + val statusLabel = when { | ||
| 348 | + processing -> null | ||
| 349 | + p.scanState == 0 -> "待索引" | ||
| 350 | + p.scanState == 1 -> "待理解" | ||
| 351 | + p.scanState >= 2 && p.caption.isNullOrBlank() -> "无描述" | ||
| 352 | + else -> null | ||
| 353 | + } | ||
| 354 | + statusLabel?.let { lbl -> | ||
| 355 | + Surface( | ||
| 356 | + color = MaterialTheme.colorScheme.surface.copy(alpha = 0.75f), | ||
| 357 | + shape = RoundedCornerShape(6.dp), | ||
| 358 | + modifier = Modifier.align(Alignment.BottomStart).padding(4.dp), | ||
| 359 | + ) { | ||
| 360 | + Text( | ||
| 361 | + lbl, style = MaterialTheme.typography.labelSmall, | ||
| 362 | + color = MaterialTheme.colorScheme.onSurface, | ||
| 363 | + modifier = Modifier.padding(horizontal = 5.dp, vertical = 1.dp), | ||
| 364 | + ) | ||
| 365 | + } | ||
| 366 | + } | ||
| 367 | + // 已出场景描述:左下角一个绿点提示「可搜」。 | ||
| 368 | + if (!processing && p.scanState >= 2 && !p.caption.isNullOrBlank()) { | ||
| 369 | + Surface( | ||
| 370 | + color = MaterialTheme.colorScheme.primary.copy(alpha = 0.85f), | ||
| 371 | + shape = RoundedCornerShape(8.dp), | ||
| 372 | + modifier = Modifier.align(Alignment.BottomStart).padding(4.dp), | ||
| 373 | + ) { | ||
| 374 | + Text( | ||
| 375 | + "📝", style = MaterialTheme.typography.labelSmall, | ||
| 376 | + modifier = Modifier.padding(horizontal = 4.dp, vertical = 1.dp), | ||
| 377 | + ) | ||
| 378 | + } | ||
| 379 | + } | ||
| 185 | if (p.faceCount > 0) { | 380 | if (p.faceCount > 0) { |
| 186 | Surface( | 381 | Surface( |
| 187 | color = MaterialTheme.colorScheme.primary.copy(alpha = 0.8f), | 382 | color = MaterialTheme.colorScheme.primary.copy(alpha = 0.8f), |
| @@ -214,7 +409,7 @@ private fun PhotoDetailSheet(vm: PhotologViewModel, photoId: Long, persons: List | @@ -214,7 +409,7 @@ private fun PhotoDetailSheet(vm: PhotologViewModel, photoId: Long, persons: List | ||
| 214 | Column(modifier = Modifier.fillMaxWidth().padding(16.dp)) { | 409 | Column(modifier = Modifier.fillMaxWidth().padding(16.dp)) { |
| 215 | photo?.let { p -> | 410 | photo?.let { p -> |
| 216 | AsyncImage( | 411 | AsyncImage( |
| 217 | - model = File(p.path), | 412 | + model = photoModel(p.source, p.path), |
| 218 | contentDescription = null, | 413 | contentDescription = null, |
| 219 | contentScale = ContentScale.Fit, | 414 | contentScale = ContentScale.Fit, |
| 220 | modifier = Modifier.fillMaxWidth().height(280.dp), | 415 | modifier = Modifier.fillMaxWidth().height(280.dp), |
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/PhotologViewModel.kt
| @@ -7,6 +7,7 @@ import androidx.lifecycle.AndroidViewModel | @@ -7,6 +7,7 @@ import androidx.lifecycle.AndroidViewModel | ||
| 7 | import androidx.lifecycle.viewModelScope | 7 | import androidx.lifecycle.viewModelScope |
| 8 | import com.lemon.lifelog.data.Person | 8 | import com.lemon.lifelog.data.Person |
| 9 | import com.lemon.lifelog.data.Photo | 9 | import com.lemon.lifelog.data.Photo |
| 10 | +import com.lemon.lifelog.data.PhotoRow | ||
| 10 | import com.lemon.lifelog.data.Repository | 11 | import com.lemon.lifelog.data.Repository |
| 11 | import com.lemon.lifelog.data.SpeakerProfile | 12 | import com.lemon.lifelog.data.SpeakerProfile |
| 12 | import androidx.work.ExistingWorkPolicy | 13 | import androidx.work.ExistingWorkPolicy |
| @@ -21,9 +22,12 @@ import com.lemon.lifelog.chat.ChatClient | @@ -21,9 +22,12 @@ import com.lemon.lifelog.chat.ChatClient | ||
| 21 | import com.lemon.lifelog.chat.ChatConfig | 22 | import com.lemon.lifelog.chat.ChatConfig |
| 22 | import com.lemon.lifelog.service.RecorderService | 23 | import com.lemon.lifelog.service.RecorderService |
| 23 | import com.lemon.lifelog.service.RecorderState | 24 | import com.lemon.lifelog.service.RecorderState |
| 25 | +import com.lemon.lifelog.vision.EmbedClient | ||
| 26 | +import com.lemon.lifelog.vision.FaceEngine | ||
| 24 | import com.lemon.lifelog.vision.FaceMatcher | 27 | import com.lemon.lifelog.vision.FaceMatcher |
| 25 | import com.lemon.lifelog.vision.ImageUtil | 28 | import com.lemon.lifelog.vision.ImageUtil |
| 26 | import com.lemon.lifelog.vision.PhotoScheduler | 29 | import com.lemon.lifelog.vision.PhotoScheduler |
| 30 | +import com.lemon.lifelog.vision.PhotoVlState | ||
| 27 | import com.lemon.lifelog.vision.WeChatEnrollWorker | 31 | import com.lemon.lifelog.vision.WeChatEnrollWorker |
| 28 | import kotlinx.coroutines.Dispatchers | 32 | import kotlinx.coroutines.Dispatchers |
| 29 | import kotlinx.coroutines.Job | 33 | import kotlinx.coroutines.Job |
| @@ -69,6 +73,66 @@ class PhotologViewModel(app: Application) : AndroidViewModel(app) { | @@ -69,6 +73,66 @@ class PhotologViewModel(app: Application) : AndroidViewModel(app) { | ||
| 69 | } | 73 | } |
| 70 | } | 74 | } |
| 71 | 75 | ||
| 76 | + /** 进相册页 / 拿到读相册权限后催一次全相册扫描 + Tier-0(幂等,已在跑则忽略)。 */ | ||
| 77 | + fun scanGallery() = PhotoScheduler.scanNow(getApplication()) | ||
| 78 | + | ||
| 79 | + /** 「立即理解」/「继续」:清暂停标志 + 催一次 VL(绕开充电约束)。 */ | ||
| 80 | + fun runVlNow() { | ||
| 81 | + PhotoVlState.setPaused(getApplication(), false) | ||
| 82 | + PhotoScheduler.runVlNow(getApplication()) | ||
| 83 | + } | ||
| 84 | + | ||
| 85 | + /** 暂停:worker 处理完当前张就停,标志持久化(周期/充电触发也不跑),可「继续」。 */ | ||
| 86 | + fun pauseVl() = PhotoVlState.setPaused(getApplication(), true) | ||
| 87 | + | ||
| 88 | + /** 停止:取消手动 VL + 清运行态;不留暂停标志(之后充电时周期照常推进)。 */ | ||
| 89 | + fun stopVl() { | ||
| 90 | + val app = getApplication<Application>() | ||
| 91 | + PhotoScheduler.cancelVl(app) | ||
| 92 | + PhotoVlState.setPaused(app, false) | ||
| 93 | + PhotoVlState.running.value = false | ||
| 94 | + PhotoVlState.currentPhotoId.value = null | ||
| 95 | + } | ||
| 96 | + | ||
| 97 | + /** 关键词搜图(caption + 人名 LIKE 子串),即时。 */ | ||
| 98 | + fun searchKeyword(kw: String, onResult: (List<PhotoRow>) -> Unit) { | ||
| 99 | + viewModelScope.launch { | ||
| 100 | + val r = withContext(Dispatchers.IO) { repo.photos.searchByKeyword(kw.trim()) } | ||
| 101 | + onResult(r) | ||
| 102 | + } | ||
| 103 | + } | ||
| 104 | + | ||
| 105 | + /** | ||
| 106 | + * 语义搜图:query → 本地 embed 端点出向量 → 跟所有 caption 向量算余弦 → top-K。 | ||
| 107 | + * embed 端点没配/挂了会走 onError(UI 提示),不影响关键词搜。 | ||
| 108 | + */ | ||
| 109 | + fun searchSemantic(query: String, onResult: (List<PhotoRow>) -> Unit, onError: (String) -> Unit) { | ||
| 110 | + viewModelScope.launch { | ||
| 111 | + val app = getApplication<Application>() | ||
| 112 | + try { | ||
| 113 | + val rows = withContext(Dispatchers.IO) { | ||
| 114 | + val embed = EmbedClient(ChatConfig.localBase(app), ChatConfig.modelEmbed(app)) | ||
| 115 | + val q = embed.embed(query.trim()) | ||
| 116 | + val vecs = repo.photos.captionVecs() | ||
| 117 | + if (vecs.isEmpty()) return@withContext emptyList<PhotoRow>() | ||
| 118 | + val scored = vecs.asSequence() | ||
| 119 | + .map { it.id to FaceEngine.cosine(q, ImageUtil.unpackEmbedding(it.captionEmbedding)) } | ||
| 120 | + .filter { it.second >= SEMANTIC_FLOOR } | ||
| 121 | + .sortedByDescending { it.second } | ||
| 122 | + .take(60) | ||
| 123 | + .map { it.first } | ||
| 124 | + .toList() | ||
| 125 | + if (scored.isEmpty()) return@withContext emptyList<PhotoRow>() | ||
| 126 | + val byId = repo.photos.rowsByIds(scored).associateBy { it.id } | ||
| 127 | + scored.mapNotNull { byId[it] } // 保持相似度降序 | ||
| 128 | + } | ||
| 129 | + onResult(rows) | ||
| 130 | + } catch (t: Throwable) { | ||
| 131 | + onError(t.message ?: "语义搜索失败(检查设置里的 embed 端点)") | ||
| 132 | + } | ||
| 133 | + } | ||
| 134 | + } | ||
| 135 | + | ||
| 72 | fun createPersonAndTag(faceId: Long, name: String) { | 136 | fun createPersonAndTag(faceId: Long, name: String) { |
| 73 | viewModelScope.launch { | 137 | viewModelScope.launch { |
| 74 | // 手动建的人 = 我关心的 → 直接 managed。 | 138 | // 手动建的人 = 我关心的 → 直接 managed。 |
| @@ -371,6 +435,8 @@ class PhotologViewModel(app: Application) : AndroidViewModel(app) { | @@ -371,6 +435,8 @@ class PhotologViewModel(app: Application) : AndroidViewModel(app) { | ||
| 371 | } | 435 | } |
| 372 | } | 436 | } |
| 373 | 437 | ||
| 438 | +private const val SEMANTIC_FLOOR = 0.20f // 语义余弦下限,低于此基本不相关;主要靠 top-K 排序 | ||
| 439 | + | ||
| 374 | /** 手机系统通讯录里的一条(内存态,未入库)。phoneDisplay 留原格式给 UI,phone 是纯数字去重键。 */ | 440 | /** 手机系统通讯录里的一条(内存态,未入库)。phoneDisplay 留原格式给 UI,phone 是纯数字去重键。 */ |
| 375 | data class PhoneContact( | 441 | data class PhoneContact( |
| 376 | val id: Long, | 442 | val id: Long, |
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/SettingsScreen.kt
| @@ -76,6 +76,7 @@ fun SettingsScreen(activity: Context) { | @@ -76,6 +76,7 @@ fun SettingsScreen(activity: Context) { | ||
| 76 | var localBase by remember { mutableStateOf(ChatConfig.localBase(activity)) } | 76 | var localBase by remember { mutableStateOf(ChatConfig.localBase(activity)) } |
| 77 | var m2b by remember { mutableStateOf(ChatConfig.model2b(activity)) } | 77 | var m2b by remember { mutableStateOf(ChatConfig.model2b(activity)) } |
| 78 | var m4b by remember { mutableStateOf(ChatConfig.model4b(activity)) } | 78 | var m4b by remember { mutableStateOf(ChatConfig.model4b(activity)) } |
| 79 | + var mEmbed by remember { mutableStateOf(ChatConfig.modelEmbed(activity)) } | ||
| 79 | var endpointMsg by remember { mutableStateOf("") } | 80 | var endpointMsg by remember { mutableStateOf("") } |
| 80 | 81 | ||
| 81 | var batteryExempt by remember { mutableStateOf(isBatteryExempt(activity)) } | 82 | var batteryExempt by remember { mutableStateOf(isBatteryExempt(activity)) } |
| @@ -133,9 +134,12 @@ fun SettingsScreen(activity: Context) { | @@ -133,9 +134,12 @@ fun SettingsScreen(activity: Context) { | ||
| 133 | OutlinedTextField(value = m4b, onValueChange = { m4b = it }, | 134 | OutlinedTextField(value = m4b, onValueChange = { m4b = it }, |
| 134 | label = { Text("4B model 名") }, singleLine = true, modifier = Modifier.weight(1f)) | 135 | label = { Text("4B model 名") }, singleLine = true, modifier = Modifier.weight(1f)) |
| 135 | } | 136 | } |
| 137 | + OutlinedTextField(value = mEmbed, onValueChange = { mEmbed = it }, | ||
| 138 | + label = { Text("embed model 名(语义搜图,llama-swap 里配 --embedding)") }, | ||
| 139 | + singleLine = true, modifier = Modifier.fillMaxWidth()) | ||
| 136 | Row(verticalAlignment = Alignment.CenterVertically, horizontalArrangement = Arrangement.spacedBy(8.dp)) { | 140 | Row(verticalAlignment = Alignment.CenterVertically, horizontalArrangement = Arrangement.spacedBy(8.dp)) { |
| 137 | Button(onClick = { | 141 | Button(onClick = { |
| 138 | - ChatConfig.save(activity, localBase, m2b, m4b) | 142 | + ChatConfig.save(activity, localBase, m2b, m4b, mEmbed) |
| 139 | endpointMsg = "已保存" | 143 | endpointMsg = "已保存" |
| 140 | }) { Text("保存端点") } | 144 | }) { Text("保存端点") } |
| 141 | if (endpointMsg.isNotEmpty()) Text(endpointMsg, style = MaterialTheme.typography.labelSmall) | 145 | if (endpointMsg.isNotEmpty()) Text(endpointMsg, style = MaterialTheme.typography.labelSmall) |
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/chat/ChatConfig.kt
| @@ -26,10 +26,13 @@ object ChatConfig { | @@ -26,10 +26,13 @@ object ChatConfig { | ||
| 26 | private const val K_LOCAL_BASE = "local_base" | 26 | private const val K_LOCAL_BASE = "local_base" |
| 27 | private const val K_MODEL_2B = "model_2b" | 27 | private const val K_MODEL_2B = "model_2b" |
| 28 | private const val K_MODEL_4B = "model_4b" | 28 | private const val K_MODEL_4B = "model_4b" |
| 29 | + private const val K_MODEL_EMBED = "model_embed" | ||
| 29 | 30 | ||
| 30 | const val DEFAULT_LOCAL_BASE = "http://127.0.0.1:8080" | 31 | const val DEFAULT_LOCAL_BASE = "http://127.0.0.1:8080" |
| 31 | const val DEFAULT_MODEL_2B = "2b" | 32 | const val DEFAULT_MODEL_2B = "2b" |
| 32 | const val DEFAULT_MODEL_4B = "4b" | 33 | const val DEFAULT_MODEL_4B = "4b" |
| 34 | + /** 语义搜索的文本 embedding model 名(llama-swap 里配 --embedding 的 GGUF,如 bge-small-zh)。 */ | ||
| 35 | + const val DEFAULT_MODEL_EMBED = "embed" | ||
| 33 | 36 | ||
| 34 | const val ID_2B = "local-2b" | 37 | const val ID_2B = "local-2b" |
| 35 | const val ID_4B = "local-4b" | 38 | const val ID_4B = "local-4b" |
| @@ -46,11 +49,15 @@ object ChatConfig { | @@ -46,11 +49,15 @@ object ChatConfig { | ||
| 46 | fun model4b(ctx: Context): String = | 49 | fun model4b(ctx: Context): String = |
| 47 | prefs(ctx).getString(K_MODEL_4B, DEFAULT_MODEL_4B)?.ifBlank { DEFAULT_MODEL_4B } ?: DEFAULT_MODEL_4B | 50 | prefs(ctx).getString(K_MODEL_4B, DEFAULT_MODEL_4B)?.ifBlank { DEFAULT_MODEL_4B } ?: DEFAULT_MODEL_4B |
| 48 | 51 | ||
| 49 | - fun save(ctx: Context, localBase: String, model2b: String, model4b: String) { | 52 | + fun modelEmbed(ctx: Context): String = |
| 53 | + prefs(ctx).getString(K_MODEL_EMBED, DEFAULT_MODEL_EMBED)?.ifBlank { DEFAULT_MODEL_EMBED } ?: DEFAULT_MODEL_EMBED | ||
| 54 | + | ||
| 55 | + fun save(ctx: Context, localBase: String, model2b: String, model4b: String, modelEmbed: String = DEFAULT_MODEL_EMBED) { | ||
| 50 | prefs(ctx).edit() | 56 | prefs(ctx).edit() |
| 51 | .putString(K_LOCAL_BASE, localBase.trim()) | 57 | .putString(K_LOCAL_BASE, localBase.trim()) |
| 52 | .putString(K_MODEL_2B, model2b.trim()) | 58 | .putString(K_MODEL_2B, model2b.trim()) |
| 53 | .putString(K_MODEL_4B, model4b.trim()) | 59 | .putString(K_MODEL_4B, model4b.trim()) |
| 60 | + .putString(K_MODEL_EMBED, modelEmbed.trim()) | ||
| 54 | .apply() | 61 | .apply() |
| 55 | } | 62 | } |
| 56 | 63 |
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/data/AppDatabase.kt
| @@ -22,7 +22,7 @@ import androidx.sqlite.db.SupportSQLiteDatabase | @@ -22,7 +22,7 @@ import androidx.sqlite.db.SupportSQLiteDatabase | ||
| 22 | WechatMessage::class, | 22 | WechatMessage::class, |
| 23 | Moment::class, | 23 | Moment::class, |
| 24 | ], | 24 | ], |
| 25 | - version = 13, | 25 | + version = 14, |
| 26 | exportSchema = false, | 26 | exportSchema = false, |
| 27 | ) | 27 | ) |
| 28 | abstract class AppDatabase : RoomDatabase() { | 28 | abstract class AppDatabase : RoomDatabase() { |
| @@ -132,6 +132,24 @@ abstract class AppDatabase : RoomDatabase() { | @@ -132,6 +132,24 @@ abstract class AppDatabase : RoomDatabase() { | ||
| 132 | } | 132 | } |
| 133 | } | 133 | } |
| 134 | 134 | ||
| 135 | + /** | ||
| 136 | + * v0.8 全相册自动索引:photos 加来源/MediaStore 去重键/分层进度/caption 语义向量。 | ||
| 137 | + * 老的「导入照片」(source 默认 imported);已处理过的(processedAt 非空)直接标 scanState=2,不重跑。 | ||
| 138 | + */ | ||
| 139 | + private val MIGRATION_13_14 = object : Migration(13, 14) { | ||
| 140 | + override fun migrate(db: SupportSQLiteDatabase) { | ||
| 141 | + db.execSQL("ALTER TABLE `photos` ADD COLUMN `source` TEXT NOT NULL DEFAULT 'imported'") | ||
| 142 | + db.execSQL("ALTER TABLE `photos` ADD COLUMN `mediaId` INTEGER") | ||
| 143 | + db.execSQL("ALTER TABLE `photos` ADD COLUMN `scanState` INTEGER NOT NULL DEFAULT 0") | ||
| 144 | + db.execSQL("ALTER TABLE `photos` ADD COLUMN `captionEmbedding` BLOB") | ||
| 145 | + // 已处理过的老照片不重跑(人脸+VL 都做完了)。 | ||
| 146 | + db.execSQL("UPDATE `photos` SET `scanState` = 2 WHERE `processedAt` IS NOT NULL") | ||
| 147 | + // 加速 Tier-0/Tier-1 取队列 + MediaStore 去重。 | ||
| 148 | + db.execSQL("CREATE INDEX IF NOT EXISTS `index_photos_scanState` ON `photos` (`scanState`)") | ||
| 149 | + db.execSQL("CREATE INDEX IF NOT EXISTS `index_photos_mediaId` ON `photos` (`mediaId`)") | ||
| 150 | + } | ||
| 151 | + } | ||
| 152 | + | ||
| 135 | fun get(ctx: Context): AppDatabase = | 153 | fun get(ctx: Context): AppDatabase = |
| 136 | instance ?: synchronized(this) { | 154 | instance ?: synchronized(this) { |
| 137 | instance ?: Room.databaseBuilder( | 155 | instance ?: Room.databaseBuilder( |
| @@ -139,7 +157,7 @@ abstract class AppDatabase : RoomDatabase() { | @@ -139,7 +157,7 @@ abstract class AppDatabase : RoomDatabase() { | ||
| 139 | AppDatabase::class.java, | 157 | AppDatabase::class.java, |
| 140 | "lifelog.db", | 158 | "lifelog.db", |
| 141 | ) | 159 | ) |
| 142 | - .addMigrations(MIGRATION_2_3, MIGRATION_3_4, MIGRATION_4_5, MIGRATION_5_6, MIGRATION_6_7, MIGRATION_7_9, MIGRATION_9_10, MIGRATION_10_11, MIGRATION_11_12, MIGRATION_12_13) | 160 | + .addMigrations(MIGRATION_2_3, MIGRATION_3_4, MIGRATION_4_5, MIGRATION_5_6, MIGRATION_6_7, MIGRATION_7_9, MIGRATION_9_10, MIGRATION_10_11, MIGRATION_11_12, MIGRATION_12_13, MIGRATION_13_14) |
| 143 | // 不再 fallbackToDestructiveMigration:以后改 schema 必须写 Migration, | 161 | // 不再 fallbackToDestructiveMigration:以后改 schema 必须写 Migration, |
| 144 | // 否则启动直接抛 IllegalStateException(fail-loud),绝不静默清掉真实录音。 | 162 | // 否则启动直接抛 IllegalStateException(fail-loud),绝不静默清掉真实录音。 |
| 145 | .build() | 163 | .build() |
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/data/Daos.kt
| @@ -287,13 +287,32 @@ interface SummaryDao { | @@ -287,13 +287,32 @@ interface SummaryDao { | ||
| 287 | data class PhotoRow( | 287 | data class PhotoRow( |
| 288 | val id: Long, | 288 | val id: Long, |
| 289 | val path: String, | 289 | val path: String, |
| 290 | + /** 'imported' = File(path);'gallery' = Uri.parse(path)。UI 据此选解码方式。 */ | ||
| 291 | + val source: String, | ||
| 290 | val takenTs: Long, | 292 | val takenTs: Long, |
| 291 | val caption: String?, | 293 | val caption: String?, |
| 292 | val faceCount: Int, | 294 | val faceCount: Int, |
| 295 | + /** 0=待索引 1=人脸已铺待理解 2=已出场景描述。格子角标据此显示状态。 */ | ||
| 296 | + val scanState: Int, | ||
| 293 | val lat: Double?, | 297 | val lat: Double?, |
| 294 | val lon: Double?, | 298 | val lon: Double?, |
| 295 | ) | 299 | ) |
| 296 | 300 | ||
| 301 | +/** 语义搜索用的轻投影:只取 id + caption 向量,cosine 在 Kotlin 里算(Room 算不了)。 */ | ||
| 302 | +data class PhotoVec( | ||
| 303 | + val id: Long, | ||
| 304 | + val captionEmbedding: ByteArray, | ||
| 305 | +) | ||
| 306 | + | ||
| 307 | +/** 相册分层进度(进度条用)。 */ | ||
| 308 | +data class PhotoProgress( | ||
| 309 | + val total: Int, | ||
| 310 | + /** Tier-0 已铺(人脸+时间,scanState≥1) */ | ||
| 311 | + val faceDone: Int, | ||
| 312 | + /** Tier-1 已出场景描述(scanState≥2) */ | ||
| 313 | + val captionDone: Int, | ||
| 314 | +) | ||
| 315 | + | ||
| 297 | @Dao | 316 | @Dao |
| 298 | interface PersonDao { | 317 | interface PersonDao { |
| 299 | @Insert | 318 | @Insert |
| @@ -400,33 +419,75 @@ interface PhotoDao { | @@ -400,33 +419,75 @@ interface PhotoDao { | ||
| 400 | @Query("SELECT * FROM photos WHERE path = :path LIMIT 1") | 419 | @Query("SELECT * FROM photos WHERE path = :path LIMIT 1") |
| 401 | suspend fun getByPath(path: String): Photo? | 420 | suspend fun getByPath(path: String): Photo? |
| 402 | 421 | ||
| 403 | - /** 还没跑过人脸+VL 的照片(处理 worker 取这个)。 */ | ||
| 404 | - @Query("SELECT * FROM photos WHERE processedAt IS NULL ORDER BY takenTs DESC") | ||
| 405 | - suspend fun unprocessed(): List<Photo> | 422 | + // ----- 全相册扫描:MediaStore 去重 + 增量 ----- |
| 423 | + | ||
| 424 | + /** 已登记的所有 MediaStore _ID(gallery 源去重)。 */ | ||
| 425 | + @Query("SELECT mediaId FROM photos WHERE mediaId IS NOT NULL") | ||
| 426 | + suspend fun knownMediaIds(): List<Long> | ||
| 427 | + | ||
| 428 | + /** 已登记的最大 MediaStore _ID(增量扫描可只看更大的,但默认仍全量去重)。 */ | ||
| 429 | + @Query("SELECT MAX(mediaId) FROM photos WHERE source = 'gallery'") | ||
| 430 | + suspend fun maxMediaId(): Long? | ||
| 431 | + | ||
| 432 | + // ----- 分层管线队列 ----- | ||
| 433 | + | ||
| 434 | + /** Tier-0 队列:还没铺人脸(scanState=0),近期优先。 */ | ||
| 435 | + @Query("SELECT * FROM photos WHERE scanState = 0 ORDER BY takenTs DESC LIMIT :limit") | ||
| 436 | + suspend fun pendingFace(limit: Int = 5000): List<Photo> | ||
| 437 | + | ||
| 438 | + /** | ||
| 439 | + * Tier-1 VL 队列:人脸已铺待出场景描述(scanState=1)。 | ||
| 440 | + * 优先级:有人脸的优先、再近期优先(合照/有人的更值得先理解)。 | ||
| 441 | + */ | ||
| 442 | + @Query( | ||
| 443 | + "SELECT * FROM photos WHERE scanState = 1 ORDER BY (faceCount > 0) DESC, takenTs DESC LIMIT :limit", | ||
| 444 | + ) | ||
| 445 | + suspend fun pendingCaption(limit: Int = 5000): List<Photo> | ||
| 446 | + | ||
| 447 | + // ----- 分层进度 ----- | ||
| 448 | + | ||
| 449 | + @Query( | ||
| 450 | + """ | ||
| 451 | + SELECT COUNT(*) AS total, | ||
| 452 | + COALESCE(SUM(CASE WHEN scanState >= 1 THEN 1 ELSE 0 END), 0) AS faceDone, | ||
| 453 | + COALESCE(SUM(CASE WHEN scanState >= 2 THEN 1 ELSE 0 END), 0) AS captionDone | ||
| 454 | + FROM photos | ||
| 455 | + """, | ||
| 456 | + ) | ||
| 457 | + fun observeProgress(): Flow<PhotoProgress> | ||
| 458 | + | ||
| 459 | + // ----- 分层写回 ----- | ||
| 406 | 460 | ||
| 407 | - /** 未处理照片数(UI 显示"待处理 N")。 */ | ||
| 408 | - @Query("SELECT COUNT(*) FROM photos WHERE processedAt IS NULL") | ||
| 409 | - fun observeUnprocessedCount(): Flow<Int> | 461 | + /** Tier-0 完成:写人脸数 + GPS,进度推到 1。 */ |
| 462 | + @Query("UPDATE photos SET faceCount = :n, lat = :lat, lon = :lon, scanState = 1 WHERE id = :id") | ||
| 463 | + suspend fun markFaceDone(id: Long, n: Int, lat: Double?, lon: Double?) | ||
| 410 | 464 | ||
| 411 | - @Query("UPDATE photos SET caption = :caption, captionModel = :model WHERE id = :id") | ||
| 412 | - suspend fun updateCaption(id: Long, caption: String, model: String) | 465 | + /** Tier-1 完成:写 caption + 语义向量,进度推到 2 + processedAt。 */ |
| 466 | + @Query( | ||
| 467 | + """ | ||
| 468 | + UPDATE photos SET caption = :caption, captionModel = :model, | ||
| 469 | + captionEmbedding = :emb, scanState = 2, processedAt = :ts WHERE id = :id | ||
| 470 | + """, | ||
| 471 | + ) | ||
| 472 | + suspend fun markCaptionDone(id: Long, caption: String?, model: String?, emb: ByteArray?, ts: Long) | ||
| 413 | 473 | ||
| 414 | - @Query("UPDATE photos SET faceCount = :n, processedAt = :ts WHERE id = :id") | ||
| 415 | - suspend fun markProcessed(id: Long, n: Int, ts: Long) | 474 | + /** Tier-1 失败也别卡队列:标 scanState=2(caption 留空),下轮不再取。 */ |
| 475 | + @Query("UPDATE photos SET scanState = 2, processedAt = :ts WHERE id = :id") | ||
| 476 | + suspend fun markCaptionSkipped(id: Long, ts: Long) | ||
| 416 | 477 | ||
| 417 | @Query("UPDATE photos SET lat = :lat, lon = :lon WHERE id = :id") | 478 | @Query("UPDATE photos SET lat = :lat, lon = :lon WHERE id = :id") |
| 418 | suspend fun updateGeo(id: Long, lat: Double?, lon: Double?) | 479 | suspend fun updateGeo(id: Long, lat: Double?, lon: Double?) |
| 419 | 480 | ||
| 420 | @Query( | 481 | @Query( |
| 421 | - "SELECT id, path, takenTs, caption, faceCount, lat, lon FROM photos ORDER BY takenTs DESC LIMIT :limit", | 482 | + "SELECT id, path, source, takenTs, caption, faceCount, scanState, lat, lon FROM photos ORDER BY takenTs DESC LIMIT :limit", |
| 422 | ) | 483 | ) |
| 423 | fun observeGrid(limit: Int = 500): Flow<List<PhotoRow>> | 484 | fun observeGrid(limit: Int = 500): Flow<List<PhotoRow>> |
| 424 | 485 | ||
| 425 | /** 某个人出现的所有照片(faces.personId join)。 */ | 486 | /** 某个人出现的所有照片(faces.personId join)。 */ |
| 426 | @Query( | 487 | @Query( |
| 427 | """ | 488 | """ |
| 428 | - SELECT DISTINCT p.id AS id, p.path AS path, p.takenTs AS takenTs, | ||
| 429 | - p.caption AS caption, p.faceCount AS faceCount, p.lat AS lat, p.lon AS lon | 489 | + SELECT DISTINCT p.id AS id, p.path AS path, p.source AS source, p.takenTs AS takenTs, |
| 490 | + p.caption AS caption, p.faceCount AS faceCount, p.scanState AS scanState, p.lat AS lat, p.lon AS lon | ||
| 430 | FROM photos p INNER JOIN faces f ON f.photoId = p.id | 491 | FROM photos p INNER JOIN faces f ON f.photoId = p.id |
| 431 | WHERE f.personId = :personId | 492 | WHERE f.personId = :personId |
| 432 | ORDER BY p.takenTs DESC | 493 | ORDER BY p.takenTs DESC |
| @@ -434,6 +495,35 @@ interface PhotoDao { | @@ -434,6 +495,35 @@ interface PhotoDao { | ||
| 434 | ) | 495 | ) |
| 435 | fun observePhotosOfPerson(personId: Long): Flow<List<PhotoRow>> | 496 | fun observePhotosOfPerson(personId: Long): Flow<List<PhotoRow>> |
| 436 | 497 | ||
| 498 | + // ----- 搜索(P3 关键词 LIKE:caption + 人名;中文子串比 FTS4 unicode61 更靠谱) ----- | ||
| 499 | + | ||
| 500 | + @Query( | ||
| 501 | + """ | ||
| 502 | + SELECT DISTINCT p.id AS id, p.path AS path, p.source AS source, p.takenTs AS takenTs, | ||
| 503 | + p.caption AS caption, p.faceCount AS faceCount, p.scanState AS scanState, p.lat AS lat, p.lon AS lon | ||
| 504 | + FROM photos p | ||
| 505 | + LEFT JOIN faces f ON f.photoId = p.id | ||
| 506 | + LEFT JOIN persons pe ON pe.id = f.personId | ||
| 507 | + WHERE p.caption LIKE '%' || :kw || '%' OR pe.name LIKE '%' || :kw || '%' | ||
| 508 | + ORDER BY p.takenTs DESC | ||
| 509 | + LIMIT :limit | ||
| 510 | + """, | ||
| 511 | + ) | ||
| 512 | + suspend fun searchByKeyword(kw: String, limit: Int = 500): List<PhotoRow> | ||
| 513 | + | ||
| 514 | + // ----- 搜索(P4 语义向量:取全部 caption 向量,cosine 在 Kotlin 里算) ----- | ||
| 515 | + | ||
| 516 | + @Query("SELECT id, captionEmbedding FROM photos WHERE captionEmbedding IS NOT NULL") | ||
| 517 | + suspend fun captionVecs(): List<PhotoVec> | ||
| 518 | + | ||
| 519 | + @Query( | ||
| 520 | + """ | ||
| 521 | + SELECT id, path, source, takenTs, caption, faceCount, scanState, lat, lon | ||
| 522 | + FROM photos WHERE id IN (:ids) | ||
| 523 | + """, | ||
| 524 | + ) | ||
| 525 | + suspend fun rowsByIds(ids: List<Long>): List<PhotoRow> | ||
| 526 | + | ||
| 437 | @Query("DELETE FROM photos WHERE id = :id") | 527 | @Query("DELETE FROM photos WHERE id = :id") |
| 438 | suspend fun delete(id: Long) | 528 | suspend fun delete(id: Long) |
| 439 | } | 529 | } |
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/data/Entities.kt
| @@ -2,6 +2,7 @@ package com.lemon.lifelog.data | @@ -2,6 +2,7 @@ package com.lemon.lifelog.data | ||
| 2 | 2 | ||
| 3 | import androidx.room.Entity | 3 | import androidx.room.Entity |
| 4 | import androidx.room.Fts4 | 4 | import androidx.room.Fts4 |
| 5 | +import androidx.room.Index | ||
| 5 | import androidx.room.PrimaryKey | 6 | import androidx.room.PrimaryKey |
| 6 | 7 | ||
| 7 | @Entity(tableName = "segments") | 8 | @Entity(tableName = "segments") |
| @@ -121,11 +122,19 @@ data class Person( | @@ -121,11 +122,19 @@ data class Person( | ||
| 121 | ) | 122 | ) |
| 122 | 123 | ||
| 123 | /** 一张照片 + Qwen-VL 场景描述 + EXIF 时间/GPS。 */ | 124 | /** 一张照片 + Qwen-VL 场景描述 + EXIF 时间/GPS。 */ |
| 124 | -@Entity(tableName = "photos") | 125 | +@Entity( |
| 126 | + tableName = "photos", | ||
| 127 | + indices = [Index("scanState"), Index("mediaId")], | ||
| 128 | +) | ||
| 125 | data class Photo( | 129 | data class Photo( |
| 126 | @PrimaryKey(autoGenerate = true) val id: Long = 0, | 130 | @PrimaryKey(autoGenerate = true) val id: Long = 0, |
| 131 | + /** imported 源 = app 私有目录文件绝对路径;gallery 源 = MediaStore content:// uri 字符串(原地读,不拷贝)。 */ | ||
| 127 | val path: String, | 132 | val path: String, |
| 128 | - /** 拍摄时间(EXIF DateTimeOriginal,回退文件时间) */ | 133 | + /** 来源:'imported'(手动选进来,拷进私有目录)| 'gallery'(全相册自动扫描,原地引用)。 */ |
| 134 | + val source: String = "imported", | ||
| 135 | + /** MediaStore _ID(gallery 源用于去重 + 增量扫描);imported 源为 null。 */ | ||
| 136 | + val mediaId: Long? = null, | ||
| 137 | + /** 拍摄时间(EXIF DateTimeOriginal / MediaStore DATE_TAKEN,回退文件时间) */ | ||
| 129 | val takenTs: Long, | 138 | val takenTs: Long, |
| 130 | val lat: Double? = null, | 139 | val lat: Double? = null, |
| 131 | val lon: Double? = null, | 140 | val lon: Double? = null, |
| @@ -133,7 +142,14 @@ data class Photo( | @@ -133,7 +142,14 @@ data class Photo( | ||
| 133 | val caption: String? = null, | 142 | val caption: String? = null, |
| 134 | val captionModel: String? = null, | 143 | val captionModel: String? = null, |
| 135 | val faceCount: Int = 0, | 144 | val faceCount: Int = 0, |
| 136 | - /** 人脸+VL 处理完成时间,null = 还没处理 */ | 145 | + /** |
| 146 | + * 分层管线进度:0 = 已登记未处理;1 = Tier-0(人脸 + EXIF)已铺;2 = Tier-1(VL 场景描述)已出。 | ||
| 147 | + * 全相册几千张,Tier-0 秒级全量铺开先可按人/时间搜,Tier-1 VL 慢、按优先级队列慢慢补。 | ||
| 148 | + */ | ||
| 149 | + val scanState: Int = 0, | ||
| 150 | + /** caption 的本地文本 embedding(LE float bytes),用于语义搜索;null = 还没算/无 caption。 */ | ||
| 151 | + val captionEmbedding: ByteArray? = null, | ||
| 152 | + /** Tier-1 VL 处理完成时间,null = 还没出 caption。 */ | ||
| 137 | val processedAt: Long? = null, | 153 | val processedAt: Long? = null, |
| 138 | ) | 154 | ) |
| 139 | 155 |
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/vision/EmbedClient.kt
0 → 100644
| 1 | +package com.lemon.lifelog.vision | ||
| 2 | + | ||
| 3 | +import okhttp3.MediaType.Companion.toMediaType | ||
| 4 | +import okhttp3.OkHttpClient | ||
| 5 | +import okhttp3.Request | ||
| 6 | +import okhttp3.RequestBody.Companion.toRequestBody | ||
| 7 | +import org.json.JSONArray | ||
| 8 | +import org.json.JSONObject | ||
| 9 | +import java.io.IOException | ||
| 10 | +import java.util.concurrent.TimeUnit | ||
| 11 | +import kotlin.math.sqrt | ||
| 12 | + | ||
| 13 | +/** | ||
| 14 | + * 文本 embedding(P4 语义搜索):打本地 llama-swap 的 OpenAI 兼容 /v1/embeddings。 | ||
| 15 | + * | ||
| 16 | + * 部署:llama-swap config.yaml 加一个 model(默认名 "embed"),用 llama-server --embedding 起一个 | ||
| 17 | + * 中文 embedding GGUF(如 bge-small-zh / gte-small-zh)。没配这个 model 时调用会抛 IOException, | ||
| 18 | + * 语义搜索优雅降级(UI 提示「未配置 embed 端点」),不影响关键词搜。 | ||
| 19 | + * | ||
| 20 | + * 返回向量统一 L2 归一化 → 余弦相似度 = 点积(与人脸/声纹 embedding 同约定)。 | ||
| 21 | + */ | ||
| 22 | +class EmbedClient( | ||
| 23 | + private val baseUrl: String, | ||
| 24 | + private val model: String = "embed", | ||
| 25 | +) { | ||
| 26 | + companion object { | ||
| 27 | + private val JSON = "application/json; charset=utf-8".toMediaType() | ||
| 28 | + } | ||
| 29 | + | ||
| 30 | + private val http = OkHttpClient.Builder() | ||
| 31 | + .connectTimeout(10, TimeUnit.SECONDS) | ||
| 32 | + .readTimeout(120, TimeUnit.SECONDS) // 含 llama-swap 切模型 | ||
| 33 | + .build() | ||
| 34 | + | ||
| 35 | + @Throws(IOException::class) | ||
| 36 | + fun embed(text: String): FloatArray { | ||
| 37 | + val payload = JSONObject().apply { | ||
| 38 | + put("model", model) | ||
| 39 | + put("input", text) | ||
| 40 | + } | ||
| 41 | + val req = Request.Builder() | ||
| 42 | + .url("$baseUrl/v1/embeddings") | ||
| 43 | + .header("Content-Type", "application/json") | ||
| 44 | + .post(payload.toString().toRequestBody(JSON)) | ||
| 45 | + .build() | ||
| 46 | + http.newCall(req).execute().use { resp -> | ||
| 47 | + val body = resp.body?.string().orEmpty() | ||
| 48 | + if (!resp.isSuccessful) throw IOException("embed ${resp.code}: ${body.take(200)}") | ||
| 49 | + val data = JSONObject(body).optJSONArray("data") ?: throw IOException("no data") | ||
| 50 | + if (data.length() == 0) throw IOException("empty data") | ||
| 51 | + val arr: JSONArray = data.getJSONObject(0).optJSONArray("embedding") | ||
| 52 | + ?: throw IOException("no embedding") | ||
| 53 | + val e = FloatArray(arr.length()) { arr.getDouble(it).toFloat() } | ||
| 54 | + return l2norm(e) | ||
| 55 | + } | ||
| 56 | + } | ||
| 57 | + | ||
| 58 | + private fun l2norm(e: FloatArray): FloatArray { | ||
| 59 | + var n = 0f | ||
| 60 | + for (v in e) n += v * v | ||
| 61 | + n = sqrt(n) | ||
| 62 | + if (n > 0f) for (i in e.indices) e[i] /= n | ||
| 63 | + return e | ||
| 64 | + } | ||
| 65 | +} |
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/vision/GalleryScanner.kt
0 → 100644
| 1 | +package com.lemon.lifelog.vision | ||
| 2 | + | ||
| 3 | +import android.content.ContentUris | ||
| 4 | +import android.content.Context | ||
| 5 | +import android.provider.MediaStore | ||
| 6 | +import android.util.Log | ||
| 7 | +import com.lemon.lifelog.data.Photo | ||
| 8 | +import com.lemon.lifelog.data.Repository | ||
| 9 | + | ||
| 10 | +/** | ||
| 11 | + * 全相册扫描:游标遍历 MediaStore 所有图片,把还没登记的注册成 photos 行(scanState=0)。 | ||
| 12 | + * | ||
| 13 | + * 关键取舍:**不拷贝文件**——gallery 源只存 content:// uri,原地引用(全相册几千上万张, | ||
| 14 | + * 拷一份会撑爆私有目录)。拍摄时间直接取 MediaStore DATE_TAKEN(秒级全量铺好时间轴), | ||
| 15 | + * GPS 留到 Tier-0 解码时按需补(要 ACCESS_MEDIA_LOCATION)。 | ||
| 16 | + * | ||
| 17 | + * 去重按 MediaStore _ID(mediaId):已登记的跳过,所以可反复跑、增量补新图。 | ||
| 18 | + */ | ||
| 19 | +object GalleryScanner { | ||
| 20 | + private const val TAG = "GalleryScanner" | ||
| 21 | + | ||
| 22 | + /** 扫一遍相册,登记新图片。返回本轮新增数。 */ | ||
| 23 | + suspend fun scan(ctx: Context, repo: Repository): Int { | ||
| 24 | + val known = repo.photos.knownMediaIds().toHashSet() | ||
| 25 | + val collection = MediaStore.Images.Media.getContentUri(MediaStore.VOLUME_EXTERNAL) | ||
| 26 | + val proj = arrayOf( | ||
| 27 | + MediaStore.Images.Media._ID, | ||
| 28 | + MediaStore.Images.Media.DATE_TAKEN, // ms(可能为 0/null) | ||
| 29 | + MediaStore.Images.Media.DATE_ADDED, // 秒 | ||
| 30 | + ) | ||
| 31 | + var added = 0 | ||
| 32 | + try { | ||
| 33 | + ctx.contentResolver.query( | ||
| 34 | + collection, proj, null, null, | ||
| 35 | + "${MediaStore.Images.Media._ID} DESC", | ||
| 36 | + )?.use { c -> | ||
| 37 | + val idIdx = c.getColumnIndexOrThrow(MediaStore.Images.Media._ID) | ||
| 38 | + val takenIdx = c.getColumnIndexOrThrow(MediaStore.Images.Media.DATE_TAKEN) | ||
| 39 | + val addedIdx = c.getColumnIndexOrThrow(MediaStore.Images.Media.DATE_ADDED) | ||
| 40 | + while (c.moveToNext()) { | ||
| 41 | + val id = c.getLong(idIdx) | ||
| 42 | + if (!known.add(id)) continue // 已登记 | ||
| 43 | + val takenMs = if (!c.isNull(takenIdx)) c.getLong(takenIdx) else 0L | ||
| 44 | + val ts = if (takenMs > 0) takenMs else c.getLong(addedIdx) * 1000L | ||
| 45 | + val uri = ContentUris.withAppendedId(collection, id) | ||
| 46 | + repo.photos.insert( | ||
| 47 | + Photo( | ||
| 48 | + path = uri.toString(), | ||
| 49 | + source = "gallery", | ||
| 50 | + mediaId = id, | ||
| 51 | + takenTs = ts, | ||
| 52 | + scanState = 0, | ||
| 53 | + ), | ||
| 54 | + ) | ||
| 55 | + added++ | ||
| 56 | + } | ||
| 57 | + } | ||
| 58 | + } catch (t: Throwable) { | ||
| 59 | + Log.w(TAG, "scan failed: ${t.message}") | ||
| 60 | + } | ||
| 61 | + if (added > 0) Log.i(TAG, "registered $added new gallery photos") | ||
| 62 | + return added | ||
| 63 | + } | ||
| 64 | +} |
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/vision/ImageUtil.kt
| 1 | package com.lemon.lifelog.vision | 1 | package com.lemon.lifelog.vision |
| 2 | 2 | ||
| 3 | +import android.content.Context | ||
| 3 | import android.graphics.Bitmap | 4 | import android.graphics.Bitmap |
| 4 | import android.graphics.BitmapFactory | 5 | import android.graphics.BitmapFactory |
| 5 | import android.graphics.Matrix | 6 | import android.graphics.Matrix |
| 7 | +import android.net.Uri | ||
| 8 | +import android.provider.MediaStore | ||
| 6 | import androidx.exifinterface.media.ExifInterface | 9 | import androidx.exifinterface.media.ExifInterface |
| 10 | +import com.lemon.lifelog.data.Photo | ||
| 7 | import java.io.ByteArrayOutputStream | 11 | import java.io.ByteArrayOutputStream |
| 8 | import java.io.File | 12 | import java.io.File |
| 13 | +import java.io.InputStream | ||
| 9 | import java.nio.ByteBuffer | 14 | import java.nio.ByteBuffer |
| 10 | import java.nio.ByteOrder | 15 | import java.nio.ByteOrder |
| 11 | import java.text.SimpleDateFormat | 16 | import java.text.SimpleDateFormat |
| 12 | import java.util.Locale | 17 | import java.util.Locale |
| 13 | import kotlin.math.min | 18 | import kotlin.math.min |
| 14 | 19 | ||
| 15 | -/** 照片 EXIF + 解码 + 缩放工具。 */ | 20 | +/** 照片 EXIF + 解码 + 缩放工具。imported 源读私有文件,gallery 源原地读 MediaStore content uri。 */ |
| 16 | object ImageUtil { | 21 | object ImageUtil { |
| 17 | data class Meta(val takenTs: Long, val lat: Double?, val lon: Double?) | 22 | data class Meta(val takenTs: Long, val lat: Double?, val lon: Double?) |
| 18 | 23 | ||
| @@ -31,6 +36,67 @@ object ImageUtil { | @@ -31,6 +36,67 @@ object ImageUtil { | ||
| 31 | } | 36 | } |
| 32 | } | 37 | } |
| 33 | 38 | ||
| 39 | + /** 按来源解码:gallery 源走 content uri(原地),imported 源走文件路径。 */ | ||
| 40 | + fun decode(ctx: Context, photo: Photo, reqMax: Int = 2048): Bitmap? = | ||
| 41 | + if (photo.source == "gallery") decodeUri(ctx, Uri.parse(photo.path), reqMax) | ||
| 42 | + else decodeOriented(photo.path, reqMax) | ||
| 43 | + | ||
| 44 | + /** | ||
| 45 | + * 读 GPS(Tier-0 用)。gallery 源需 ACCESS_MEDIA_LOCATION + setRequireOriginal 拿未脱敏 EXIF; | ||
| 46 | + * 没权限/失败就返回 (null,null),不阻断流程。takenTs 在扫描时已从 MediaStore 取,这里只补经纬度。 | ||
| 47 | + */ | ||
| 48 | + fun readGps(ctx: Context, photo: Photo): Pair<Double?, Double?> = try { | ||
| 49 | + val stream: InputStream? = if (photo.source == "gallery") { | ||
| 50 | + val orig = runCatching { MediaStore.setRequireOriginal(Uri.parse(photo.path)) } | ||
| 51 | + .getOrDefault(Uri.parse(photo.path)) | ||
| 52 | + ctx.contentResolver.openInputStream(orig) | ||
| 53 | + } else { | ||
| 54 | + File(photo.path).inputStream() | ||
| 55 | + } | ||
| 56 | + stream?.use { | ||
| 57 | + val ll = ExifInterface(it).latLong | ||
| 58 | + if (ll != null) ll[0] to ll[1] else null to null | ||
| 59 | + } ?: (null to null) | ||
| 60 | + } catch (t: Throwable) { | ||
| 61 | + null to null | ||
| 62 | + } | ||
| 63 | + | ||
| 64 | + /** 从 content uri 解码 + 按 EXIF 方向摆正(gallery 源)。 */ | ||
| 65 | + private fun decodeUri(ctx: Context, uri: Uri, reqMax: Int): Bitmap? { | ||
| 66 | + val cr = ctx.contentResolver | ||
| 67 | + val bounds = BitmapFactory.Options().apply { inJustDecodeBounds = true } | ||
| 68 | + // 注意:inJustDecodeBounds 模式下 decodeStream 本就返回 null(只填 bounds), | ||
| 69 | + // 不能拿它的返回值判失败——要判的是 openInputStream。 | ||
| 70 | + (cr.openInputStream(uri) ?: return null).use { BitmapFactory.decodeStream(it, null, bounds) } | ||
| 71 | + var sample = 1 | ||
| 72 | + val longest = maxOf(bounds.outWidth, bounds.outHeight) | ||
| 73 | + while (longest / sample > reqMax) sample *= 2 | ||
| 74 | + val decoded = cr.openInputStream(uri)?.use { | ||
| 75 | + BitmapFactory.decodeStream(it, null, BitmapFactory.Options().apply { inSampleSize = sample }) | ||
| 76 | + } ?: return null | ||
| 77 | + val orientation = try { | ||
| 78 | + cr.openInputStream(uri)?.use { | ||
| 79 | + ExifInterface(it).getAttributeInt(ExifInterface.TAG_ORIENTATION, ExifInterface.ORIENTATION_NORMAL) | ||
| 80 | + } ?: ExifInterface.ORIENTATION_NORMAL | ||
| 81 | + } catch (t: Throwable) { | ||
| 82 | + ExifInterface.ORIENTATION_NORMAL | ||
| 83 | + } | ||
| 84 | + return applyOrientation(decoded, orientation) | ||
| 85 | + } | ||
| 86 | + | ||
| 87 | + private fun applyOrientation(decoded: Bitmap, orientation: Int): Bitmap { | ||
| 88 | + val m = Matrix() | ||
| 89 | + when (orientation) { | ||
| 90 | + ExifInterface.ORIENTATION_ROTATE_90 -> m.postRotate(90f) | ||
| 91 | + ExifInterface.ORIENTATION_ROTATE_180 -> m.postRotate(180f) | ||
| 92 | + ExifInterface.ORIENTATION_ROTATE_270 -> m.postRotate(270f) | ||
| 93 | + else -> return decoded | ||
| 94 | + } | ||
| 95 | + val rotated = Bitmap.createBitmap(decoded, 0, 0, decoded.width, decoded.height, m, true) | ||
| 96 | + if (rotated != decoded) decoded.recycle() | ||
| 97 | + return rotated | ||
| 98 | + } | ||
| 99 | + | ||
| 34 | /** 解码并按 EXIF 方向摆正。可选下采样 reqMax(最长边)以省内存。 */ | 100 | /** 解码并按 EXIF 方向摆正。可选下采样 reqMax(最长边)以省内存。 */ |
| 35 | fun decodeOriented(path: String, reqMax: Int = 2048): Bitmap? { | 101 | fun decodeOriented(path: String, reqMax: Int = 2048): Bitmap? { |
| 36 | val opts = BitmapFactory.Options() | 102 | val opts = BitmapFactory.Options() |
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/vision/PhotoWorker.kt renamed to lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/vision/PhotoFaceWorker.kt
| @@ -4,44 +4,48 @@ import android.content.Context | @@ -4,44 +4,48 @@ import android.content.Context | ||
| 4 | import android.util.Log | 4 | import android.util.Log |
| 5 | import androidx.work.CoroutineWorker | 5 | import androidx.work.CoroutineWorker |
| 6 | import androidx.work.WorkerParameters | 6 | import androidx.work.WorkerParameters |
| 7 | -import com.lemon.lifelog.chat.ChatConfig | ||
| 8 | import com.lemon.lifelog.data.Repository | 7 | import com.lemon.lifelog.data.Repository |
| 9 | import java.util.concurrent.atomic.AtomicBoolean | 8 | import java.util.concurrent.atomic.AtomicBoolean |
| 10 | 9 | ||
| 11 | /** | 10 | /** |
| 12 | - * 后台批处理未处理照片(processedAt=null):人脸 + Qwen-VL 场景描述。 | ||
| 13 | - * 导入只登记照片行,处理交给这里——闲时/定时跑,不让用户在导入时干等。 | ||
| 14 | - * 单轮有时间预算,超了下轮(retry/periodic)继续,避开 WorkManager 执行上限。 | 11 | + * Tier-0 worker:① 扫全相册登记新图 ② 给未铺人脸的照片(scanState=0)跑 ML Kit 人脸 + GPS。 |
| 12 | + * | ||
| 13 | + * 快(<1s/张),不依赖 llama-server,battery-not-low 即可跑——目标是几小时内把整个相册 | ||
| 14 | + * 的人脸/时间线铺满,先能「按人/按时间」搜。VL 场景描述是另一条慢队列(PhotoVlWorker)。 | ||
| 15 | + * | ||
| 16 | + * 单轮 7min 预算 + retry,避开 WorkManager ~10min 执行上限。 | ||
| 15 | */ | 17 | */ |
| 16 | -class PhotoWorker(ctx: Context, params: WorkerParameters) : CoroutineWorker(ctx, params) { | 18 | +class PhotoFaceWorker(ctx: Context, params: WorkerParameters) : CoroutineWorker(ctx, params) { |
| 17 | 19 | ||
| 18 | override suspend fun doWork(): Result { | 20 | override suspend fun doWork(): Result { |
| 19 | if (!isRunning.compareAndSet(false, true)) return Result.success() | 21 | if (!isRunning.compareAndSet(false, true)) return Result.success() |
| 20 | try { | 22 | try { |
| 21 | val repo = Repository(applicationContext) | 23 | val repo = Repository(applicationContext) |
| 22 | - val pending = repo.photos.unprocessed() | 24 | + // 先增量登记全相册新图(需要已授予读相册权限;没权限就只处理已登记的)。 |
| 25 | + runCatching { GalleryScanner.scan(applicationContext, repo) } | ||
| 26 | + | ||
| 27 | + val pending = repo.photos.pendingFace() | ||
| 23 | if (pending.isEmpty()) return Result.success() | 28 | if (pending.isEmpty()) return Result.success() |
| 24 | 29 | ||
| 25 | val engine = FaceEngine.load(applicationContext) | 30 | val engine = FaceEngine.load(applicationContext) |
| 26 | - val vision = VisionClient(ChatConfig.localBase(applicationContext)) | ||
| 27 | - val proc = PhotoProcessor(repo, engine, vision) | 31 | + val proc = PhotoProcessor(applicationContext, repo) |
| 28 | val deadline = System.currentTimeMillis() + RUN_BUDGET_MS | 32 | val deadline = System.currentTimeMillis() + RUN_BUDGET_MS |
| 29 | var done = 0 | 33 | var done = 0 |
| 30 | try { | 34 | try { |
| 31 | for (p in pending) { | 35 | for (p in pending) { |
| 32 | if (System.currentTimeMillis() > deadline) break | 36 | if (System.currentTimeMillis() > deadline) break |
| 33 | try { | 37 | try { |
| 34 | - proc.process(p) | 38 | + proc.faceStage(p, engine) |
| 35 | } catch (t: Throwable) { | 39 | } catch (t: Throwable) { |
| 36 | - Log.w(TAG, "process failed ${p.path}: ${t.message}") | 40 | + Log.w(TAG, "faceStage failed ${p.path}: ${t.message}") |
| 41 | + repo.photos.markFaceDone(p.id, 0, null, null) // 别卡队列 | ||
| 37 | } | 42 | } |
| 38 | done++ | 43 | done++ |
| 39 | } | 44 | } |
| 40 | } finally { | 45 | } finally { |
| 41 | engine.close() | 46 | engine.close() |
| 42 | } | 47 | } |
| 43 | - Log.i(TAG, "processed $done / ${pending.size} pending") | ||
| 44 | - // 还有剩的就 retry 继续(带退避),否则结束等下个周期。 | 48 | + Log.i(TAG, "tier0 face: $done / ${pending.size}") |
| 45 | return if (done < pending.size) Result.retry() else Result.success() | 49 | return if (done < pending.size) Result.retry() else Result.success() |
| 46 | } catch (t: Throwable) { | 50 | } catch (t: Throwable) { |
| 47 | Log.e(TAG, "worker failed", t) | 51 | Log.e(TAG, "worker failed", t) |
| @@ -52,10 +56,9 @@ class PhotoWorker(ctx: Context, params: WorkerParameters) : CoroutineWorker(ctx, | @@ -52,10 +56,9 @@ class PhotoWorker(ctx: Context, params: WorkerParameters) : CoroutineWorker(ctx, | ||
| 52 | } | 56 | } |
| 53 | 57 | ||
| 54 | companion object { | 58 | companion object { |
| 55 | - const val TAG = "PhotoWorker" | ||
| 56 | - /** 单轮预算 7min(VL ~45s/张,约 9 张/轮),避开 WorkManager ~10min 上限。 */ | 59 | + const val TAG = "PhotoFaceWorker" |
| 60 | + /** 单轮预算 7min,避开 WorkManager ~10min 上限。 */ | ||
| 57 | const val RUN_BUDGET_MS = 7 * 60 * 1000L | 61 | const val RUN_BUDGET_MS = 7 * 60 * 1000L |
| 58 | - /** 防并发:periodic + one-shot + retry 可能同时触发,多 worker 打同一 llama-server 会 502。 */ | ||
| 59 | private val isRunning = AtomicBoolean(false) | 62 | private val isRunning = AtomicBoolean(false) |
| 60 | } | 63 | } |
| 61 | } | 64 | } |
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/vision/PhotoProcessor.kt
| 1 | package com.lemon.lifelog.vision | 1 | package com.lemon.lifelog.vision |
| 2 | 2 | ||
| 3 | +import android.content.Context | ||
| 3 | import android.util.Log | 4 | import android.util.Log |
| 4 | import com.lemon.lifelog.data.Face | 5 | import com.lemon.lifelog.data.Face |
| 5 | import com.lemon.lifelog.data.Photo | 6 | import com.lemon.lifelog.data.Photo |
| 6 | import com.lemon.lifelog.data.Repository | 7 | import com.lemon.lifelog.data.Repository |
| 7 | import kotlinx.coroutines.Dispatchers | 8 | import kotlinx.coroutines.Dispatchers |
| 8 | import kotlinx.coroutines.withContext | 9 | import kotlinx.coroutines.withContext |
| 9 | -import java.io.File | ||
| 10 | 10 | ||
| 11 | /** | 11 | /** |
| 12 | - * 单张照片处理:人脸(FaceEngine) + 场景描述(VisionClient) → 更新已登记的 photos 行 + 写 faces。 | ||
| 13 | - * 导入时只登记 photos 行(processedAt=null);这里由后台 PhotoWorker 闲时调用,不阻塞 UI。 | 12 | + * 单张照片的分层处理: |
| 13 | + * - Tier-0 [faceStage]:人脸检测 + 512d embedding + GPS。秒级,全相册先铺满(按人/时间可搜)。 | ||
| 14 | + * - Tier-1 [captionStage]:Qwen-VL 场景描述 + caption 语义向量。慢,按优先级队列充电时跑。 | ||
| 15 | + * | ||
| 16 | + * 两段都从来源(imported 私有文件 / gallery content uri)解码,互不依赖,可在不同时机/约束下跑。 | ||
| 14 | */ | 17 | */ |
| 15 | class PhotoProcessor( | 18 | class PhotoProcessor( |
| 19 | + private val ctx: Context, | ||
| 16 | private val repo: Repository, | 20 | private val repo: Repository, |
| 17 | - private val face: FaceEngine, | ||
| 18 | - private val vision: VisionClient, | ||
| 19 | ) { | 21 | ) { |
| 20 | - /** 处理一张已登记但未处理的照片(processedAt=null)。 */ | ||
| 21 | - suspend fun process(photo: Photo) { | 22 | + /** Tier-0:人脸 + GPS → faces 行 + 自动认人,scanState 推到 1。 */ |
| 23 | + suspend fun faceStage(photo: Photo, face: FaceEngine) { | ||
| 22 | withContext(Dispatchers.IO) { | 24 | withContext(Dispatchers.IO) { |
| 23 | - val bmp = ImageUtil.decodeOriented(photo.path) ?: run { | ||
| 24 | - Log.w(TAG, "decode failed: ${photo.path}") | ||
| 25 | - repo.photos.markProcessed(photo.id, 0, System.currentTimeMillis()) | 25 | + val bmp = ImageUtil.decode(ctx, photo) ?: run { |
| 26 | + Log.w(TAG, "decode failed (face): ${photo.path}") | ||
| 27 | + repo.photos.markFaceDone(photo.id, 0, null, null) | ||
| 26 | return@withContext | 28 | return@withContext |
| 27 | } | 29 | } |
| 28 | - | ||
| 29 | val faces = try { | 30 | val faces = try { |
| 30 | face.analyze(bmp) | 31 | face.analyze(bmp) |
| 31 | } catch (t: Throwable) { | 32 | } catch (t: Throwable) { |
| 32 | Log.w(TAG, "face analyze failed: ${t.message}"); emptyList() | 33 | Log.w(TAG, "face analyze failed: ${t.message}"); emptyList() |
| 33 | } | 34 | } |
| 34 | - val caption = try { | ||
| 35 | - vision.caption(ImageUtil.toJpeg(bmp)) | ||
| 36 | - } catch (t: Throwable) { | ||
| 37 | - Log.w(TAG, "caption failed: ${t.message}"); null | ||
| 38 | - } | ||
| 39 | bmp.recycle() | 35 | bmp.recycle() |
| 40 | 36 | ||
| 41 | - if (caption != null) repo.photos.updateCaption(photo.id, caption, "vl") | 37 | + val (lat, lon) = ImageUtil.readGps(ctx, photo) |
| 42 | 38 | ||
| 43 | // 匹配 gallery = 微信头像锚点 + 已标注脸样本。 | 39 | // 匹配 gallery = 微信头像锚点 + 已标注脸样本。 |
| 44 | val gallery = FaceMatcher.gallery(repo) | 40 | val gallery = FaceMatcher.gallery(repo) |
| @@ -55,8 +51,38 @@ class PhotoProcessor( | @@ -55,8 +51,38 @@ class PhotoProcessor( | ||
| 55 | ), | 51 | ), |
| 56 | ) | 52 | ) |
| 57 | } | 53 | } |
| 58 | - repo.photos.markProcessed(photo.id, faces.size, System.currentTimeMillis()) | ||
| 59 | - Log.i(TAG, "✓ ${File(photo.path).name}: ${faces.size} 脸, caption=${caption?.take(30)}") | 54 | + repo.photos.markFaceDone(photo.id, faces.size, lat, lon) |
| 55 | + } | ||
| 56 | + } | ||
| 57 | + | ||
| 58 | + /** Tier-1:VL 场景描述 + caption 语义向量 → caption/captionEmbedding,scanState 推到 2。 */ | ||
| 59 | + suspend fun captionStage(photo: Photo, vision: VisionClient, embed: EmbedClient?) { | ||
| 60 | + withContext(Dispatchers.IO) { | ||
| 61 | + val bmp = ImageUtil.decode(ctx, photo) ?: run { | ||
| 62 | + Log.w(TAG, "decode failed (caption): ${photo.path}") | ||
| 63 | + repo.photos.markCaptionSkipped(photo.id, System.currentTimeMillis()) | ||
| 64 | + return@withContext | ||
| 65 | + } | ||
| 66 | + val caption = try { | ||
| 67 | + vision.caption(ImageUtil.toJpeg(bmp, maxDim = VL_MAX_DIM)).takeIf { it.isNotBlank() } | ||
| 68 | + } catch (t: Throwable) { | ||
| 69 | + Log.w(TAG, "caption failed: ${t.message}"); null | ||
| 70 | + } | ||
| 71 | + bmp.recycle() | ||
| 72 | + | ||
| 73 | + // caption 出来了就顺手算语义向量(embed 端点没配/失败就只存文本,不卡进度)。 | ||
| 74 | + val emb = if (caption != null && embed != null) { | ||
| 75 | + try { | ||
| 76 | + ImageUtil.packEmbedding(embed.embed(caption)) | ||
| 77 | + } catch (t: Throwable) { | ||
| 78 | + Log.w(TAG, "embed failed: ${t.message}"); null | ||
| 79 | + } | ||
| 80 | + } else { | ||
| 81 | + null | ||
| 82 | + } | ||
| 83 | + | ||
| 84 | + repo.photos.markCaptionDone(photo.id, caption, caption?.let { "vl" }, emb, System.currentTimeMillis()) | ||
| 85 | + Log.i(TAG, "✓ photo#${photo.id}: caption=${caption?.take(30)} emb=${emb != null}") | ||
| 60 | } | 86 | } |
| 61 | } | 87 | } |
| 62 | 88 | ||
| @@ -64,5 +90,7 @@ class PhotoProcessor( | @@ -64,5 +90,7 @@ class PhotoProcessor( | ||
| 64 | private const val TAG = "PhotoProcessor" | 90 | private const val TAG = "PhotoProcessor" |
| 65 | /** 同人余弦阈值(实测同人 0.96 / 异人 0.13,取 0.45 稳)。 */ | 91 | /** 同人余弦阈值(实测同人 0.96 / 异人 0.13,取 0.45 稳)。 */ |
| 66 | const val THRESH = 0.45f | 92 | const val THRESH = 0.45f |
| 93 | + /** 喂 VL 的最长边:448 比 512 再省些 vision token,粗粒度场景描述精度几乎不掉。 */ | ||
| 94 | + const val VL_MAX_DIM = 448 | ||
| 67 | } | 95 | } |
| 68 | } | 96 | } |
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/vision/PhotoScheduler.kt
| @@ -12,40 +12,67 @@ import androidx.work.WorkManager | @@ -12,40 +12,67 @@ import androidx.work.WorkManager | ||
| 12 | import java.util.concurrent.TimeUnit | 12 | import java.util.concurrent.TimeUnit |
| 13 | 13 | ||
| 14 | /** | 14 | /** |
| 15 | - * 相册照片处理调度(与纪要 SummaryScheduler 同构): | ||
| 16 | - * - schedulePeriodic:每 15min(WorkManager 硬下限)扫一遍未处理照片,闲时跑。 | ||
| 17 | - * - runNow:导入后触发一次,让新照片尽快在后台开始处理(不阻塞导入 UI)。 | 15 | + * 相册分层处理调度: |
| 16 | + * - Tier-0(人脸+扫描,PhotoFaceWorker):每 15min,只要不低电就跑——快,尽快铺满全相册人脸/时间线。 | ||
| 17 | + * - Tier-1(VL 场景描述,PhotoVlWorker):每 15min,**充电 + 不低电**才跑——慢、耗电,长期后台补。 | ||
| 18 | + * - runNow / scanNow:导入或进相册页时手动催一次(不阻塞 UI)。 | ||
| 18 | */ | 19 | */ |
| 19 | object PhotoScheduler { | 20 | object PhotoScheduler { |
| 20 | - private const val PERIODIC_TAG = "lifelog-photo-periodic" | ||
| 21 | - private const val ONE_SHOT_TAG = "lifelog-photo-once" | 21 | + private const val FACE_PERIODIC = "lifelog-photo-face-periodic" |
| 22 | + private const val VL_PERIODIC = "lifelog-photo-vl-periodic" | ||
| 23 | + private const val FACE_ONCE = "lifelog-photo-face-once" | ||
| 24 | + private const val VL_ONCE = "lifelog-photo-vl-once" | ||
| 22 | 25 | ||
| 23 | fun schedulePeriodic(ctx: Context) { | 26 | fun schedulePeriodic(ctx: Context) { |
| 24 | - val constraints = Constraints.Builder() | ||
| 25 | - .setRequiredNetworkType(NetworkType.NOT_REQUIRED) // VL 在本机 | ||
| 26 | - .setRequiresBatteryNotLow(true) | 27 | + val wm = WorkManager.getInstance(ctx) |
| 28 | + | ||
| 29 | + val faceReq = PeriodicWorkRequestBuilder<PhotoFaceWorker>(15, TimeUnit.MINUTES) | ||
| 30 | + .setConstraints( | ||
| 31 | + Constraints.Builder() | ||
| 32 | + .setRequiredNetworkType(NetworkType.NOT_REQUIRED) | ||
| 33 | + .setRequiresBatteryNotLow(true) | ||
| 34 | + .build(), | ||
| 35 | + ) | ||
| 36 | + .setBackoffCriteria(BackoffPolicy.LINEAR, 1, TimeUnit.MINUTES) | ||
| 27 | .build() | 37 | .build() |
| 28 | - val request = PeriodicWorkRequestBuilder<PhotoWorker>(15, TimeUnit.MINUTES) | ||
| 29 | - .setConstraints(constraints) | 38 | + wm.enqueueUniquePeriodicWork(FACE_PERIODIC, ExistingPeriodicWorkPolicy.UPDATE, faceReq) |
| 39 | + | ||
| 40 | + val vlReq = PeriodicWorkRequestBuilder<PhotoVlWorker>(15, TimeUnit.MINUTES) | ||
| 41 | + .setConstraints( | ||
| 42 | + Constraints.Builder() | ||
| 43 | + .setRequiredNetworkType(NetworkType.NOT_REQUIRED) | ||
| 44 | + .setRequiresBatteryNotLow(true) | ||
| 45 | + .setRequiresCharging(true) // 全量 VL 很烧电,只在充电时推进 | ||
| 46 | + .build(), | ||
| 47 | + ) | ||
| 30 | .setBackoffCriteria(BackoffPolicy.LINEAR, 1, TimeUnit.MINUTES) | 48 | .setBackoffCriteria(BackoffPolicy.LINEAR, 1, TimeUnit.MINUTES) |
| 31 | .build() | 49 | .build() |
| 32 | - WorkManager.getInstance(ctx).enqueueUniquePeriodicWork( | ||
| 33 | - PERIODIC_TAG, ExistingPeriodicWorkPolicy.UPDATE, request, | ||
| 34 | - ) | 50 | + wm.enqueueUniquePeriodicWork(VL_PERIODIC, ExistingPeriodicWorkPolicy.UPDATE, vlReq) |
| 35 | } | 51 | } |
| 36 | 52 | ||
| 37 | - /** 导入后调一次,让后台尽快开工。已在跑则忽略(KEEP)。 */ | 53 | + /** 导入后催一次:先铺人脸(不要求充电),VL 留给周期/充电时。 */ |
| 38 | fun runNow(ctx: Context) { | 54 | fun runNow(ctx: Context) { |
| 39 | - val constraints = Constraints.Builder() | ||
| 40 | - .setRequiresBatteryNotLow(true) | 55 | + val req = OneTimeWorkRequestBuilder<PhotoFaceWorker>() |
| 56 | + .setConstraints(Constraints.Builder().setRequiresBatteryNotLow(true).build()) | ||
| 57 | + .setBackoffCriteria(BackoffPolicy.LINEAR, 1, TimeUnit.MINUTES) | ||
| 41 | .build() | 58 | .build() |
| 42 | - val request = OneTimeWorkRequestBuilder<PhotoWorker>() | ||
| 43 | - .setConstraints(constraints) | ||
| 44 | - .addTag(ONE_SHOT_TAG) | 59 | + WorkManager.getInstance(ctx).enqueueUniqueWork(FACE_ONCE, ExistingWorkPolicy.KEEP, req) |
| 60 | + } | ||
| 61 | + | ||
| 62 | + /** 进相册页催一次全相册扫描 + Tier-0(同 runNow,仅语义更明确)。 */ | ||
| 63 | + fun scanNow(ctx: Context) = runNow(ctx) | ||
| 64 | + | ||
| 65 | + /** 手动催一次 VL(设置/相册页「立即理解」按钮用,绕开充电约束)。 */ | ||
| 66 | + fun runVlNow(ctx: Context) { | ||
| 67 | + val req = OneTimeWorkRequestBuilder<PhotoVlWorker>() | ||
| 68 | + .setConstraints(Constraints.Builder().setRequiresBatteryNotLow(true).build()) | ||
| 45 | .setBackoffCriteria(BackoffPolicy.LINEAR, 1, TimeUnit.MINUTES) | 69 | .setBackoffCriteria(BackoffPolicy.LINEAR, 1, TimeUnit.MINUTES) |
| 46 | .build() | 70 | .build() |
| 47 | - WorkManager.getInstance(ctx).enqueueUniqueWork( | ||
| 48 | - ONE_SHOT_TAG, ExistingWorkPolicy.KEEP, request, | ||
| 49 | - ) | 71 | + WorkManager.getInstance(ctx).enqueueUniqueWork(VL_ONCE, ExistingWorkPolicy.KEEP, req) |
| 72 | + } | ||
| 73 | + | ||
| 74 | + /** 停止 VL:取消手动的一次性 VL(周期由 paused 标志另行 gate)。 */ | ||
| 75 | + fun cancelVl(ctx: Context) { | ||
| 76 | + WorkManager.getInstance(ctx).cancelUniqueWork(VL_ONCE) | ||
| 50 | } | 77 | } |
| 51 | } | 78 | } |
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/vision/PhotoVlState.kt
0 → 100644
| 1 | +package com.lemon.lifelog.vision | ||
| 2 | + | ||
| 3 | +import android.content.Context | ||
| 4 | +import kotlinx.coroutines.flow.MutableStateFlow | ||
| 5 | +import kotlinx.coroutines.flow.StateFlow | ||
| 6 | + | ||
| 7 | +/** | ||
| 8 | + * Tier-1 VL 处理的运行态(给相册 UI:进度/暂停/停止 + 单张处理指示)。 | ||
| 9 | + * 跟 RecorderState 同形——纯进程内 StateFlow,worker 推、UI 收。paused 持久化到 prefs, | ||
| 10 | + * 这样周期/充电时的自动 worker 也能尊重「暂停」。 | ||
| 11 | + */ | ||
| 12 | +object PhotoVlState { | ||
| 13 | + /** VL worker 是否正在循环处理(拿到锁那个)。 */ | ||
| 14 | + val running = MutableStateFlow(false) | ||
| 15 | + | ||
| 16 | + /** 当前正在出场景描述的照片 id(null = 没有)。UI 在对应格上转圈。 */ | ||
| 17 | + val currentPhotoId = MutableStateFlow<Long?>(null) | ||
| 18 | + | ||
| 19 | + private val _paused = MutableStateFlow(false) | ||
| 20 | + | ||
| 21 | + /** 用户「暂停」标志(持久化)。worker 每张前检查;为 true 时不取新照片。 */ | ||
| 22 | + val paused: StateFlow<Boolean> = _paused | ||
| 23 | + | ||
| 24 | + private const val PREFS = "photo_vl" | ||
| 25 | + private const val KEY_PAUSED = "paused" | ||
| 26 | + | ||
| 27 | + /** App 启动时把持久化的暂停标志载入 StateFlow(在 LifelogApp.onCreate 调)。 */ | ||
| 28 | + fun init(ctx: Context) { | ||
| 29 | + _paused.value = ctx.getSharedPreferences(PREFS, Context.MODE_PRIVATE).getBoolean(KEY_PAUSED, false) | ||
| 30 | + } | ||
| 31 | + | ||
| 32 | + fun setPaused(ctx: Context, v: Boolean) { | ||
| 33 | + ctx.getSharedPreferences(PREFS, Context.MODE_PRIVATE).edit().putBoolean(KEY_PAUSED, v).apply() | ||
| 34 | + _paused.value = v | ||
| 35 | + } | ||
| 36 | + | ||
| 37 | + /** worker 读这个(持久化值为准,跨进程/重启可靠)。 */ | ||
| 38 | + fun isPaused(ctx: Context): Boolean = | ||
| 39 | + ctx.getSharedPreferences(PREFS, Context.MODE_PRIVATE).getBoolean(KEY_PAUSED, false) | ||
| 40 | +} |
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/vision/PhotoVlWorker.kt
0 → 100644
| 1 | +package com.lemon.lifelog.vision | ||
| 2 | + | ||
| 3 | +import android.content.Context | ||
| 4 | +import android.util.Log | ||
| 5 | +import androidx.work.CoroutineWorker | ||
| 6 | +import androidx.work.WorkerParameters | ||
| 7 | +import com.lemon.lifelog.chat.ChatConfig | ||
| 8 | +import com.lemon.lifelog.data.Repository | ||
| 9 | +import java.util.concurrent.atomic.AtomicBoolean | ||
| 10 | + | ||
| 11 | +/** | ||
| 12 | + * Tier-1 worker:给已铺人脸的照片(scanState=1)按优先级跑 Qwen-VL 场景描述 + caption 语义向量。 | ||
| 13 | + * | ||
| 14 | + * 慢(~10-40s/张,打本地 llama-server),所以:充电 + 不低电才跑、优先级队列(有人脸>近期)、 | ||
| 15 | + * 单轮 7min 预算 + retry。全相册全量 VL 是个长期后台任务,进度在相册页可见。 | ||
| 16 | + * | ||
| 17 | + * 并发互斥:periodic + 手动 + retry 可能并发,多 worker 打同一 llama-server 会 502。 | ||
| 18 | + */ | ||
| 19 | +class PhotoVlWorker(ctx: Context, params: WorkerParameters) : CoroutineWorker(ctx, params) { | ||
| 20 | + | ||
| 21 | + override suspend fun doWork(): Result { | ||
| 22 | + // 用户暂停时彻底不跑(含周期/充电触发)。 | ||
| 23 | + if (PhotoVlState.isPaused(applicationContext)) return Result.success() | ||
| 24 | + if (!isRunning.compareAndSet(false, true)) return Result.success() | ||
| 25 | + PhotoVlState.running.value = true | ||
| 26 | + try { | ||
| 27 | + val repo = Repository(applicationContext) | ||
| 28 | + val pending = repo.photos.pendingCaption() | ||
| 29 | + if (pending.isEmpty()) return Result.success() | ||
| 30 | + | ||
| 31 | + val base = ChatConfig.localBase(applicationContext) | ||
| 32 | + val vision = VisionClient(base) | ||
| 33 | + val embed = EmbedClient(base, ChatConfig.modelEmbed(applicationContext)) | ||
| 34 | + val proc = PhotoProcessor(applicationContext, repo) | ||
| 35 | + val deadline = System.currentTimeMillis() + RUN_BUDGET_MS | ||
| 36 | + var done = 0 | ||
| 37 | + for (p in pending) { | ||
| 38 | + if (System.currentTimeMillis() > deadline) break | ||
| 39 | + if (PhotoVlState.isPaused(applicationContext)) break // 暂停:当前张不再开新的 | ||
| 40 | + PhotoVlState.currentPhotoId.value = p.id | ||
| 41 | + try { | ||
| 42 | + proc.captionStage(p, vision, embed) | ||
| 43 | + } catch (t: Throwable) { | ||
| 44 | + Log.w(TAG, "captionStage failed ${p.path}: ${t.message}") | ||
| 45 | + repo.photos.markCaptionSkipped(p.id, System.currentTimeMillis()) // 别卡队列 | ||
| 46 | + } | ||
| 47 | + done++ | ||
| 48 | + } | ||
| 49 | + Log.i(TAG, "tier1 vl: $done / ${pending.size}") | ||
| 50 | + // 暂停时别 retry(否则一直重排);正常没跑完才 retry。 | ||
| 51 | + val paused = PhotoVlState.isPaused(applicationContext) | ||
| 52 | + return if (!paused && done < pending.size) Result.retry() else Result.success() | ||
| 53 | + } catch (t: Throwable) { | ||
| 54 | + Log.e(TAG, "worker failed", t) | ||
| 55 | + return Result.retry() | ||
| 56 | + } finally { | ||
| 57 | + PhotoVlState.currentPhotoId.value = null | ||
| 58 | + PhotoVlState.running.value = false | ||
| 59 | + isRunning.set(false) | ||
| 60 | + } | ||
| 61 | + } | ||
| 62 | + | ||
| 63 | + companion object { | ||
| 64 | + const val TAG = "PhotoVlWorker" | ||
| 65 | + /** 单轮预算 7min(VL ~10-40s/张),避开 WorkManager ~10min 上限。 */ | ||
| 66 | + const val RUN_BUDGET_MS = 7 * 60 * 1000L | ||
| 67 | + private val isRunning = AtomicBoolean(false) | ||
| 68 | + } | ||
| 69 | +} |