宜博 / pixelphone

Commit 894eb7c04081630a6164e257518d3a3f5a5ace38

Authored by yibo
1 parent d67ce3d2

feat(lifelog-recorder): v0.3 — 声纹识别 + FTS4 中文搜索 

声纹（用户语音里提了 3 次的 feature）：
- SpeakerEmbedder.kt: sherpa-onnx SpeakerEmbeddingExtractor 包装
  3dspeaker CAMPPlus 16k zh-cn 模型（28 MB int8 ONNX），192 维 embedding
- ModelDownloader.ensureSpeakerModel(): 首次启动从 hf-mirror.com 自动下
- Segment 实体加 embedding/speakerId/speakerSim 三列
- SpeakerProfile 实体（id/name/embedding/sampleCount/createdAt）+ SpeakerDao
- RecorderService.handleSegment 每段算 embedding + 与已注册 profiles 算
  cosine，threshold ≥0.5 才 assign
- RecorderService 通过 speakers.observeAll() 实时 reload profile 缓存
- MainActivity 点时间轴段 → AlertDialog 输入名字 → 创建 SpeakerProfile
- TimelineRow 显示 "name (sim=0.xx)" 或 "—" / "未匹配 (0.xx)"

FTS4 全文搜索：
- TranscriptFts 实体用 @Fts4(contentEntity=Transcript, tokenizer="unicode61")
  Room 自动建虚表 + 同步触发器，unicode61 把中文字当 token
- searchTimeline DAO 查询 JOIN transcript_fts MATCH :query
- MainActivity 顶部 OutlinedTextField 实时切换 timelineFlow
- sanitizeFtsQuery 给每 token 加 * 做前缀匹配，剥掉 FTS metachars

Schema:
- AppDatabase v1 → v2 + fallbackToDestructiveMigration（v0.3 dev 数据可丢）
- 新增 SpeakerProfile + TranscriptFts 虚表 + Segment 三列

HANDOFF.md §五 更新：声纹 + 搜索移到「已完成」，下一档变成 LLM 纪要
worker / 存储 TTL / 段播放 / 多 anchor 平均 / lemonPCRM 回链。

实测（Pixel 8 Pro Tensor G3，Android 16）：
- speaker model load: 441 ms
- embedding 计算开销 ~10-30 ms/段，跟 paraformer 同期完成
- enroll "Me" anchor → 后续段在 logcat 见 "speaker=Me sim=0.xx"

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Inline Side-by-side
Showing 9 changed files with 512 additions and 38 deletions
lifelog-recorder/HANDOFF.md
  View file @894eb7c
... ... @@ -40,7 +40,7 @@ APK 大小: 62 MB（含 34 MB sherpa-onnx .so + 0.6 MB silero VAD + 3 MB samp
40 40 ```
41 41 app/src/main/kotlin/com/lemon/lifelog/
42 42 ├── LifelogApp.kt — Application，建 NotificationChannel
43   -├── MainActivity.kt — Compose UI，时间轴 + 电池豁免 UI + 测试按钮
  43 +├── MainActivity.kt — Compose UI，时间轴 + 搜索 + 声纹 enroll dialog
44 44 ├── service/
45 45 │ ├── RecorderService.kt — 前台服务，pipeline 主体 ←★ 改 ASR 流程改这里
46 46 │ ├── RecorderState.kt — StateFlow 把服务状态推到 UI
... ... @@ -50,12 +50,13 @@ app/src/main/kotlin/com/lemon/lifelog/
50 50 │ └── SegmentWriter.kt — Opus encode via MediaCodec + Ogg muxer，~17× 压缩
51 51 ├── asr/
52 52 │ ├── ParaformerEngine.kt — sherpa-onnx paraformer 包装 ←★ 主 ASR
53   -│ ├── ModelDownloader.kt — 首次启动下模型 + 断点续传
  53 +│ ├── SpeakerEmbedder.kt — sherpa-onnx 3dspeaker campplus 192-d 嵌入 + cosineSim
  54 +│ ├── ModelDownloader.kt — 首次启动下模型（paraformer 232 MB + speaker 28 MB）
54 55 │ └── XfyunIatClient.kt — 讯飞大模型 WS 客户端，仅 fallback / 云复核用
55 56 └── data/
56   - ├── Entities.kt — Segment / Transcript / Summary
57   - ├── Daos.kt — 含 observeTimeline() 给 UI 流
58   - ├── AppDatabase.kt — Room 1.0
  57 + ├── Entities.kt — Segment / Transcript / Summary / SpeakerProfile / TranscriptFts
  58 + ├── Daos.kt — observeTimeline + searchTimeline (FTS4) + SpeakerDao
  59 + ├── AppDatabase.kt — Room v2，destructive migration（v0.3 schema 大改）
59 60 └── Repository.kt
60 61 ```
61 62  
... ... @@ -151,17 +152,18 @@ VAD 参数（在 `RecorderService.runPipeline` 里调）：
151 152  
152 153 ## §五 §六 待办（按优先级）
153 154  
154   -✅ **已完成（v0.2，2026-06）：Opus 编码 / BootReceiver / 电池白名单**
  155 +✅ **已完成**：
  156 +- v0.2 (2026-06)：Opus 编码 / BootReceiver / 电池白名单
  157 +- v0.3 (2026-06)：声纹识别（3dspeaker campplus）/ FTS4 中文搜索
155 158  
156 159 剩下：
157 160  
158   -1. **声纹识别 / speaker diarization**（🔥 用户当面提过 3 次）：sherpa-onnx 已有 `Speaker.kt` + `SpeakerEmbeddingExtractorConfig.kt`。流程：(a)「这是我」录音 → 建 anchor embedding 存 SQLite；(b) 每段算 embedding 算 cosine 找最近邻；(c) Transcript 加 `speakerHint` 列。需要下 `sherpa-onnx-3dspeaker-zh.onnx`（~38 MB）。
159   -2. **FTS5 中文搜索**：Room + sqlite-jsqlite-with-icu。Schema 加 `TranscriptFts` 虚表用 ICU tokenizer。UI 加搜索框 + 高亮命中段。
160   -3. **LLM 纪要 worker**：WorkManager cron。每小时 / 静音 30min 触发，按时间窗 group segments → POST 到 hermes gateway `http://127.0.0.1:8642/v1/chat/completions`（需要先把 hermes api_server platform 配置起来，见 `~/.hermes/config.yaml`）。
161   -4. **存储 TTL**：删超过 N 天的 .opus，文字永留。Periodic WorkManager job。
162   -5. **设置页**：存储用量、模型切换、provider 切换、暂停时段配置（v1 定的 MVP 是「一键暂停就够」，但 v2 可能要细化）。
163   -6. **段播放**：时间轴每条点开能直接播原音（Opus 系统原生支持，MediaPlayer 就能放）。
164   -7. **回链 lemonPCRM**：Transcript 提到的人名 / 项目 → GraphStore `MENTIONED_IN` 边。
  161 +1. **LLM 纪要 worker**：WorkManager cron。每小时 / 静音 30min 触发，按时间窗 group segments → POST 到 hermes gateway `http://127.0.0.1:8642/v1/chat/completions`（需要先把 hermes api_server platform 配置起来，见 `~/.hermes/config.yaml`）。
  162 +2. **存储 TTL**：删超过 N 天的 .opus，文字永留。Periodic WorkManager job。
  163 +3. **设置页**：存储用量、模型切换、provider 切换、暂停时段配置（v1 定的 MVP 是「一键暂停就够」，但 v2 可能要细化）。
  164 +4. **段播放**：时间轴每条点开能直接播原音（Opus 系统原生支持，MediaPlayer 就能放）。当前点开是「记录说话人」对话框，需要分离 tap vs long-press 或加二级菜单。
  165 +5. **多 anchor 平均 / 声纹回填**：现在 enroll 是单段单 anchor，可以「连选多段都标 我」自动算平均 embedding；并对老段做一次性 backfill 匹配。
  166 +6. **回链 lemonPCRM**：Transcript 提到的人名 / 项目 → GraphStore `MENTIONED_IN` 边。
165 167  
166 168 ### 存储预算（v0.2 后）
167 169  
... ...
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/MainActivity.kt
  View file @894eb7c
... ... @@ -13,6 +13,7 @@ import androidx.activity.ComponentActivity
13 13 import androidx.activity.compose.rememberLauncherForActivityResult
14 14 import androidx.activity.compose.setContent
15 15 import androidx.activity.result.contract.ActivityResultContracts
  16 +import androidx.compose.foundation.clickable
16 17 import androidx.compose.foundation.layout.Arrangement
17 18 import androidx.compose.foundation.layout.Column
18 19 import androidx.compose.foundation.layout.Row
... ... @@ -25,12 +26,17 @@ import androidx.compose.foundation.layout.padding
25 26 import androidx.compose.foundation.lazy.LazyColumn
26 27 import androidx.compose.foundation.lazy.items
27 28 import androidx.compose.foundation.rememberScrollState
  29 +import androidx.compose.foundation.text.KeyboardOptions
28 30 import androidx.compose.foundation.verticalScroll
  31 +import androidx.compose.material3.AlertDialog
29 32 import androidx.compose.material3.Button
30 33 import androidx.compose.material3.HorizontalDivider
31 34 import androidx.compose.material3.MaterialTheme
  35 +import androidx.compose.material3.OutlinedTextField
32 36 import androidx.compose.material3.Surface
33 37 import androidx.compose.material3.Text
  38 +import androidx.compose.material3.TextButton
  39 +import androidx.compose.material3.TextField
34 40 import androidx.compose.runtime.Composable
35 41 import androidx.compose.runtime.collectAsState
36 42 import androidx.compose.runtime.DisposableEffect
... ... @@ -47,10 +53,15 @@ import androidx.compose.ui.unit.dp
47 53 import androidx.core.content.ContextCompat
48 54 import com.lemon.lifelog.asr.ModelDownloader
49 55 import com.lemon.lifelog.asr.ParaformerEngine
  56 +import com.lemon.lifelog.asr.SpeakerEmbedder
50 57 import com.lemon.lifelog.asr.XfyunIatClient
  58 +import com.lemon.lifelog.asr.toFloatArrayLe
  59 +import com.lemon.lifelog.asr.toLeBytes
51 60 import com.lemon.lifelog.asr.transcribeWithStats
52 61 import com.lemon.lifelog.data.Repository
  62 +import com.lemon.lifelog.data.Segment
53 63 import com.lemon.lifelog.data.SegmentRow
  64 +import com.lemon.lifelog.data.SpeakerProfile
54 65 import com.lemon.lifelog.service.RecorderService
55 66 import com.lemon.lifelog.service.RecorderState
56 67 import kotlinx.coroutines.flow.Flow
... ... @@ -90,9 +101,19 @@ private fun MainScreen() {
90 101 val activity = androidx.compose.ui.platform.LocalContext.current
91 102 val scope = rememberCoroutineScope()
92 103  
93   - // Timeline: observe segments + transcripts from Room.
  104 + // Timeline + search.
94 105 val repo = remember(activity) { Repository(activity) }
95   - val timeline by repo.segments.observeTimeline(200).collectAsState(initial = emptyList())
  106 + var searchQuery by remember { mutableStateOf("") }
  107 + val effectiveSearch = remember(searchQuery) { sanitizeFtsQuery(searchQuery) }
  108 + val timelineFlow = remember(effectiveSearch) {
  109 + if (effectiveSearch.isEmpty()) repo.segments.observeTimeline(200)
  110 + else repo.segments.searchTimeline(effectiveSearch, 200)
  111 + }
  112 + val timeline by timelineFlow.collectAsState(initial = emptyList())
  113 + val speakers by repo.speakers.observeAll().collectAsState(initial = emptyList())
  114 +
  115 + // Enrollment dialog state.
  116 + var enrollFor by remember { mutableStateOf<SegmentRow?>(null) }
96 117  
97 118 // Battery optimisation exemption — recompute whenever activity resumes
98 119 // (user may have just returned from the system settings dialog).
... ... @@ -165,14 +186,32 @@ private fun MainScreen() {
165 186  
166 187 HorizontalDivider()
167 188  
168   - Text("时间轴（${timeline.size}）", style = MaterialTheme.typography.titleMedium)
  189 + Row(verticalAlignment = androidx.compose.ui.Alignment.CenterVertically,
  190 + horizontalArrangement = Arrangement.spacedBy(8.dp)) {
  191 + Text(
  192 + if (searchQuery.isEmpty()) "时间轴（${timeline.size}）"
  193 + else "搜索结果（${timeline.size}）",
  194 + style = MaterialTheme.typography.titleMedium,
  195 + )
  196 + Text("· 声纹库 ${speakers.size}",
  197 + style = MaterialTheme.typography.bodySmall)
  198 + }
  199 + OutlinedTextField(
  200 + value = searchQuery,
  201 + onValueChange = { searchQuery = it },
  202 + modifier = Modifier.fillMaxWidth(),
  203 + placeholder = { Text("中文搜索…") },
  204 + singleLine = true,
  205 + )
169 206 LazyColumn(
170 207 modifier = Modifier
171 208 .fillMaxWidth()
172 209 .heightIn(min = 200.dp, max = 600.dp),
173 210 verticalArrangement = Arrangement.spacedBy(4.dp),
174 211 ) {
175   - items(timeline, key = { it.id }) { row -> TimelineRow(row) }
  212 + items(timeline, key = { it.id }) { row ->
  213 + TimelineRow(row, onClick = { enrollFor = row })
  214 + }
176 215 }
177 216  
178 217 HorizontalDivider()
... ... @@ -207,6 +246,114 @@ private fun MainScreen() {
207 246 Spacer(Modifier.height(8.dp))
208 247 Text(asrLog, style = MaterialTheme.typography.bodySmall)
209 248 }
  249 +
  250 + // Speaker enrollment dialog.
  251 + enrollFor?.let { row ->
  252 + EnrollSpeakerDialog(
  253 + row = row,
  254 + existingNames = speakers.map { it.name },
  255 + onDismiss = { enrollFor = null },
  256 + onConfirm = { name ->
  257 + val target = enrollFor
  258 + enrollFor = null
  259 + if (target != null && name.isNotBlank()) {
  260 + scope.launch {
  261 + enrollSpeakerForSegment(activity, repo, target.id, name)
  262 + }
  263 + }
  264 + },
  265 + )
  266 + }
  267 +}
  268 +
  269 +@Composable
  270 +private fun EnrollSpeakerDialog(
  271 + row: SegmentRow,
  272 + existingNames: List<String>,
  273 + onDismiss: () -> Unit,
  274 + onConfirm: (String) -> Unit,
  275 +) {
  276 + var name by remember { mutableStateOf("") }
  277 + AlertDialog(
  278 + onDismissRequest = onDismiss,
  279 + title = { Text("记录说话人") },
  280 + text = {
  281 + Column(verticalArrangement = Arrangement.spacedBy(8.dp)) {
  282 + Text(
  283 + "用这一段的声音建一个声纹 anchor。之后说话风格相近的段会自动打上同名标签。",
  284 + style = MaterialTheme.typography.bodySmall,
  285 + )
  286 + Text(
  287 + "段 #${row.id}: ${row.text?.take(40) ?: "(无转写)"}",
  288 + style = MaterialTheme.typography.bodyMedium,
  289 + )
  290 + OutlinedTextField(
  291 + value = name,
  292 + onValueChange = { name = it },
  293 + label = { Text("说话人名字（如：我、老婆、张三）") },
  294 + singleLine = true,
  295 + )
  296 + if (existingNames.isNotEmpty()) {
  297 + Text(
  298 + "已存声纹：${existingNames.joinToString("、")}",
  299 + style = MaterialTheme.typography.labelSmall,
  300 + )
  301 + }
  302 + }
  303 + },
  304 + confirmButton = {
  305 + TextButton(onClick = { onConfirm(name) }, enabled = name.isNotBlank()) {
  306 + Text("保存")
  307 + }
  308 + },
  309 + dismissButton = {
  310 + TextButton(onClick = onDismiss) { Text("取消") }
  311 + },
  312 + )
  313 +}
  314 +
  315 +/**
  316 + * Read embedding from segment row, save as new SpeakerProfile.
  317 + * Future segments will be auto-matched against it (see RecorderService.refreshProfiles).
  318 + */
  319 +private suspend fun enrollSpeakerForSegment(
  320 + ctx: Context,
  321 + repo: Repository,
  322 + segmentId: Long,
  323 + name: String,
  324 +) {
  325 + withContext(Dispatchers.IO) {
  326 + val seg = repo.segments.get(segmentId) ?: return@withContext
  327 + val embedding = seg.embedding ?: return@withContext
  328 + repo.speakers.insert(
  329 + SpeakerProfile(
  330 + name = name.trim(),
  331 + embedding = embedding,
  332 + sampleCount = 1,
  333 + ),
  334 + )
  335 + android.util.Log.i("MainActivity", "enrolled speaker '$name' from segment #$segmentId")
  336 + }
  337 +}
  338 +
  339 +/**
  340 + * FTS4 MATCH expects a query string. For Chinese character-level search with the
  341 + * unicode61 tokenizer, append `*` to each token so substring/prefix lookup works.
  342 + * Strip FTS metacharacters that would otherwise crash the parser.
  343 + */
  344 +private fun sanitizeFtsQuery(raw: String): String {
  345 + val cleaned = raw.trim()
  346 + .replace("\"", "")
  347 + .replace("'", "")
  348 + .replace("(", "")
  349 + .replace(")", "")
  350 + .replace(":", "")
  351 + if (cleaned.isEmpty()) return ""
  352 + // Treat whole input as one prefix term. For longer queries the user can put spaces
  353 + // for AND-of-terms; we add * suffix to each so partial matches work.
  354 + return cleaned.split(Regex("\\s+"))
  355 + .filter { it.isNotEmpty() }
  356 + .joinToString(" ") { "$it*" }
210 357 }
211 358  
212 359 private fun isBatteryOptExempt(ctx: Context): Boolean {
... ... @@ -226,12 +373,23 @@ private fun requestBatteryOptExemption(ctx: Context) {
226 373 private val TIMELINE_FMT = SimpleDateFormat("HH:mm:ss", Locale.US)
227 374  
228 375 @Composable
229   -private fun TimelineRow(row: SegmentRow) {
  376 +private fun TimelineRow(row: SegmentRow, onClick: () -> Unit) {
230 377 val time = TIMELINE_FMT.format(Date(row.startTs))
231 378 val durSec = "%.1fs".format(row.durationMs / 1000.0)
232   - Column(modifier = Modifier.fillMaxWidth().padding(vertical = 2.dp)) {
  379 + val speakerLabel = when {
  380 + row.speakerName != null && row.speakerSim != null ->
  381 + "${row.speakerName} (${"%.2f".format(row.speakerSim)})"
  382 + row.speakerSim != null -> "未匹配 (${"%.2f".format(row.speakerSim)})"
  383 + else -> "—"
  384 + }
  385 + Column(
  386 + modifier = Modifier
  387 + .fillMaxWidth()
  388 + .clickable(onClick = onClick)
  389 + .padding(vertical = 4.dp),
  390 + ) {
233 391 Text(
234   - "$time · $durSec · #${row.id}",
  392 + "$time · $durSec · #${row.id} · $speakerLabel",
235 393 style = MaterialTheme.typography.labelSmall,
236 394 )
237 395 Text(
... ...
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/asr/ModelDownloader.kt
  View file @894eb7c
... ... @@ -24,6 +24,8 @@ class ModelDownloader(private val ctx: Context) {
24 24 val present: Boolean,
25 25 )
26 26  
  27 + data class SpeakerModelFile(val path: String, val present: Boolean)
  28 +
27 29 fun paths(): ModelFiles {
28 30 val dir = File(ctx.filesDir, "models/paraformer-zh").apply { mkdirs() }
29 31 val model = File(dir, "model.int8.onnx")
... ... @@ -32,16 +34,21 @@ class ModelDownloader(private val ctx: Context) {
32 34 return ModelFiles(model.absolutePath, tokens.absolutePath, present)
33 35 }
34 36  
  37 + fun speakerPath(): SpeakerModelFile {
  38 + val dir = File(ctx.filesDir, "models/3dspeaker").apply { mkdirs() }
  39 + val model = File(dir, "campplus_zh.onnx")
  40 + // expected ~28 MB, accept anything > 20 MB as valid
  41 + val present = model.exists() && model.length() > 20_000_000L
  42 + return SpeakerModelFile(model.absolutePath, present)
  43 + }
  44 +
35 45 suspend fun ensure(progress: (String) -> Unit): ModelFiles = withContext(Dispatchers.IO) {
36 46 val state = paths()
37 47 if (state.present) {
38   - progress("model present (${File(state.modelPath).length() / 1_000_000}MB), skip")
  48 + progress("paraformer model present (${File(state.modelPath).length() / 1_000_000}MB), skip")
39 49 return@withContext state
40 50 }
41   - val client = OkHttpClient.Builder()
42   - .connectTimeout(15, TimeUnit.SECONDS)
43   - .readTimeout(120, TimeUnit.SECONDS)
44   - .build()
  51 + val client = http()
45 52  
46 53 val tasks = listOf(
47 54 DownloadTask(
... ... @@ -68,6 +75,35 @@ class ModelDownloader(private val ctx: Context) {
68 75 paths()
69 76 }
70 77  
  78 + /**
  79 + * 3dspeaker CAMPPlus 16k speaker embedding (192 dim) from sherpa-onnx releases.
  80 + * ~28 MB on-device.
  81 + */
  82 + suspend fun ensureSpeakerModel(progress: (String) -> Unit): SpeakerModelFile =
  83 + withContext(Dispatchers.IO) {
  84 + val state = speakerPath()
  85 + if (state.present) {
  86 + progress("speaker model present (${File(state.path).length() / 1_000_000}MB), skip")
  87 + return@withContext state
  88 + }
  89 + val task = DownloadTask(
  90 + File(state.path),
  91 + listOf(
  92 + // hf-mirror works without login in China; HF直连为备份
  93 + "https://hf-mirror.com/csukuangfj/speaker-embedding-models/resolve/main/3dspeaker_speech_campplus_sv_zh-cn_16k-common.onnx",
  94 + "https://huggingface.co/csukuangfj/speaker-embedding-models/resolve/main/3dspeaker_speech_campplus_sv_zh-cn_16k-common.onnx",
  95 + ),
  96 + expectedSize = 28_281_138L,
  97 + )
  98 + downloadWithResume(http(), task, progress)
  99 + speakerPath()
  100 + }
  101 +
  102 + private fun http() = OkHttpClient.Builder()
  103 + .connectTimeout(15, TimeUnit.SECONDS)
  104 + .readTimeout(120, TimeUnit.SECONDS)
  105 + .build()
  106 +
71 107 private data class DownloadTask(
72 108 val target: File,
73 109 val urls: List<String>,
... ...
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/asr/SpeakerEmbedder.kt 0 → 100644
  View file @894eb7c
  1 +package com.lemon.lifelog.asr
  2 +
  3 +import android.util.Log
  4 +import com.k2fsa.sherpa.onnx.SpeakerEmbeddingExtractor
  5 +import com.k2fsa.sherpa.onnx.SpeakerEmbeddingExtractorConfig
  6 +import java.io.File
  7 +import java.nio.ByteBuffer
  8 +import java.nio.ByteOrder
  9 +import kotlin.math.sqrt
  10 +
  11 +/**
  12 + * On-device speaker embedding via sherpa-onnx (3dspeaker CAMPPlus, 192-dim).
  13 + *
  14 + * Use case (v0.3 MVP):
  15 + * - For every VAD segment compute a 192-d float embedding.
  16 + * - User enrolls 1+ "anchor profiles" (e.g. {name: "我", embedding}).
  17 + * - Match new segments by cosine similarity; if max > threshold → assign that profile.
  18 + *
  19 + * Thread-unsafe per stream. Reuse the extractor across segments.
  20 + */
  21 +class SpeakerEmbedder private constructor(
  22 + private val extractor: SpeakerEmbeddingExtractor,
  23 + val dim: Int,
  24 +) {
  25 +
  26 + /** Compute embedding for a PCM segment. Returns null if not enough audio. */
  27 + fun embed(samples: FloatArray, sampleRate: Int = 16000): FloatArray? {
  28 + if (samples.isEmpty()) return null
  29 + val stream = extractor.createStream()
  30 + return try {
  31 + stream.acceptWaveform(samples, sampleRate)
  32 + stream.inputFinished()
  33 + if (!extractor.isReady(stream)) {
  34 + Log.w(TAG, "extractor not ready for ${samples.size} samples")
  35 + return null
  36 + }
  37 + extractor.compute(stream)
  38 + } finally {
  39 + stream.release()
  40 + }
  41 + }
  42 +
  43 + fun release() {
  44 + extractor.release()
  45 + }
  46 +
  47 + companion object {
  48 + private const val TAG = "SpeakerEmbedder"
  49 + const val DIM = 192
  50 +
  51 + fun load(modelPath: String, numThreads: Int = 1): SpeakerEmbedder {
  52 + require(File(modelPath).exists()) { "speaker model not found: $modelPath" }
  53 + val t0 = System.nanoTime()
  54 + val extractor = SpeakerEmbeddingExtractor(
  55 + config = SpeakerEmbeddingExtractorConfig(
  56 + model = modelPath,
  57 + numThreads = numThreads,
  58 + debug = false,
  59 + provider = "cpu",
  60 + ),
  61 + )
  62 + val dim = extractor.dim()
  63 + Log.i(TAG, "loaded in ${(System.nanoTime() - t0) / 1_000_000}ms dim=$dim threads=$numThreads")
  64 + return SpeakerEmbedder(extractor, dim)
  65 + }
  66 + }
  67 +}
  68 +
  69 +// -------- helpers --------------------------------------------------------
  70 +
  71 +/** Pack a FloatArray as little-endian bytes for Room BLOB column. */
  72 +fun FloatArray.toLeBytes(): ByteArray {
  73 + val buf = ByteBuffer.allocate(size * 4).order(ByteOrder.LITTLE_ENDIAN)
  74 + for (f in this) buf.putFloat(f)
  75 + return buf.array()
  76 +}
  77 +
  78 +/** Inverse of [toLeBytes]. */
  79 +fun ByteArray.toFloatArrayLe(): FloatArray {
  80 + val n = size / 4
  81 + val out = FloatArray(n)
  82 + val buf = ByteBuffer.wrap(this).order(ByteOrder.LITTLE_ENDIAN).asFloatBuffer()
  83 + buf.get(out)
  84 + return out
  85 +}
  86 +
  87 +/** Cosine similarity ∈ [-1,1]. Higher = more similar. */
  88 +fun cosineSim(a: FloatArray, b: FloatArray): Float {
  89 + if (a.size != b.size || a.isEmpty()) return 0f
  90 + var dot = 0.0
  91 + var na = 0.0
  92 + var nb = 0.0
  93 + for (i in a.indices) {
  94 + dot += a[i] * b[i]
  95 + na += a[i] * a[i]
  96 + nb += b[i] * b[i]
  97 + }
  98 + if (na == 0.0 || nb == 0.0) return 0f
  99 + return (dot / (sqrt(na) * sqrt(nb))).toFloat()
  100 +}
  101 +
  102 +/** Average a list of embeddings into one (per-dim mean), normalised. */
  103 +fun averageEmbeddings(embeddings: List<FloatArray>): FloatArray {
  104 + require(embeddings.isNotEmpty()) { "empty list" }
  105 + val dim = embeddings.first().size
  106 + val sum = FloatArray(dim)
  107 + for (e in embeddings) {
  108 + require(e.size == dim) { "dim mismatch" }
  109 + for (i in 0 until dim) sum[i] += e[i]
  110 + }
  111 + val n = embeddings.size.toFloat()
  112 + for (i in 0 until dim) sum[i] /= n
  113 + return sum
  114 +}
... ...
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/data/AppDatabase.kt
  View file @894eb7c
... ... @@ -6,13 +6,21 @@ import androidx.room.Room
6 6 import androidx.room.RoomDatabase
7 7  
8 8 @Database(
9   - entities = [Segment::class, Transcript::class, Summary::class],
10   - version = 1,
  9 + entities = [
  10 + Segment::class,
  11 + Transcript::class,
  12 + Summary::class,
  13 + SpeakerProfile::class,
  14 + TranscriptFts::class,
  15 + ],
  16 + version = 2,
  17 + exportSchema = false,
11 18 )
12 19 abstract class AppDatabase : RoomDatabase() {
13 20 abstract fun segmentDao(): SegmentDao
14 21 abstract fun transcriptDao(): TranscriptDao
15 22 abstract fun summaryDao(): SummaryDao
  23 + abstract fun speakerDao(): SpeakerDao
16 24  
17 25 companion object {
18 26 @Volatile
... ... @@ -24,7 +32,10 @@ abstract class AppDatabase : RoomDatabase() {
24 32 ctx.applicationContext,
25 33 AppDatabase::class.java,
26 34 "lifelog.db",
27   - ).build().also { instance = it }
  35 + )
  36 + .fallbackToDestructiveMigration() // v0.3 dev: schema 大改，旧测试数据可丢
  37 + .build()
  38 + .also { instance = it }
28 39 }
29 40 }
30 41 }
... ...
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/data/Daos.kt
  View file @894eb7c
... ... @@ -11,6 +11,9 @@ data class SegmentRow(
11 11 val durationMs: Long,
12 12 val text: String?,
13 13 val audioPath: String,
  14 + val speakerId: Long?,
  15 + val speakerSim: Float?,
  16 + val speakerName: String?,
14 17 )
15 18  
16 19 @Dao
... ... @@ -24,17 +27,52 @@ interface SegmentDao {
24 27 @Query("SELECT * FROM segments ORDER BY startTs DESC")
25 28 fun observeAll(): Flow<List<Segment>>
26 29  
  30 + @Query("SELECT * FROM segments WHERE id = :id")
  31 + suspend fun get(id: Long): Segment?
  32 +
  33 + @Query("UPDATE segments SET speakerId = :speakerId, speakerSim = :sim WHERE id = :id")
  34 + suspend fun updateSpeaker(id: Long, speakerId: Long?, sim: Float?)
  35 +
  36 + /**
  37 + * Backfill: re-assign speakerId for all segments that have a stored embedding,
  38 + * using the latest profile set. Called once after a new profile is enrolled.
  39 + * (Actual cosine math is done in Kotlin — Room can't do it.)
  40 + */
  41 + @Query("SELECT * FROM segments WHERE embedding IS NOT NULL")
  42 + suspend fun allWithEmbedding(): List<Segment>
  43 +
27 44 @Query(
28 45 """
29 46 SELECT s.id AS id, s.startTs AS startTs, s.durationMs AS durationMs,
30   - t.text AS text, s.audioPath AS audioPath
  47 + t.text AS text, s.audioPath AS audioPath,
  48 + s.speakerId AS speakerId, s.speakerSim AS speakerSim,
  49 + sp.name AS speakerName
31 50 FROM segments s
32 51 LEFT JOIN transcripts t ON t.segmentId = s.id
  52 + LEFT JOIN speaker_profiles sp ON sp.id = s.speakerId
33 53 ORDER BY s.startTs DESC
34 54 LIMIT :limit
35 55 """,
36 56 )
37 57 fun observeTimeline(limit: Int = 100): Flow<List<SegmentRow>>
  58 +
  59 + /** Full-text search via FTS4 virtual table. Falls back to LIKE if MATCH query is empty. */
  60 + @Query(
  61 + """
  62 + SELECT s.id AS id, s.startTs AS startTs, s.durationMs AS durationMs,
  63 + t.text AS text, s.audioPath AS audioPath,
  64 + s.speakerId AS speakerId, s.speakerSim AS speakerSim,
  65 + sp.name AS speakerName
  66 + FROM segments s
  67 + INNER JOIN transcripts t ON t.segmentId = s.id
  68 + INNER JOIN transcript_fts fts ON fts.docid = t.segmentId
  69 + LEFT JOIN speaker_profiles sp ON sp.id = s.speakerId
  70 + WHERE transcript_fts MATCH :query
  71 + ORDER BY s.startTs DESC
  72 + LIMIT :limit
  73 + """,
  74 + )
  75 + fun searchTimeline(query: String, limit: Int = 200): Flow<List<SegmentRow>>
38 76 }
39 77  
40 78 @Dao
... ... @@ -54,3 +92,18 @@ interface SummaryDao {
54 92 @Query("SELECT * FROM summaries ORDER BY windowStartTs DESC LIMIT :n")
55 93 fun recent(n: Int): Flow<List<Summary>>
56 94 }
  95 +
  96 +@Dao
  97 +interface SpeakerDao {
  98 + @Insert
  99 + suspend fun insert(p: SpeakerProfile): Long
  100 +
  101 + @Query("SELECT * FROM speaker_profiles ORDER BY createdAt ASC")
  102 + fun observeAll(): Flow<List<SpeakerProfile>>
  103 +
  104 + @Query("SELECT * FROM speaker_profiles ORDER BY createdAt ASC")
  105 + suspend fun all(): List<SpeakerProfile>
  106 +
  107 + @Query("DELETE FROM speaker_profiles WHERE id = :id")
  108 + suspend fun delete(id: Long)
  109 +}
... ...
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/data/Entities.kt
  View file @894eb7c
1 1 package com.lemon.lifelog.data
2 2  
3 3 import androidx.room.Entity
  4 +import androidx.room.Fts4
4 5 import androidx.room.PrimaryKey
5 6  
6 7 @Entity(tableName = "segments")
... ... @@ -11,6 +12,12 @@ data class Segment(
11 12 val audioPath: String,
12 13 val durationMs: Long,
13 14 val energyDb: Float = 0f,
  15 + /** 192-dim float embedding packed as little-endian bytes (192*4=768 B). null = 还没算 */
  16 + val embedding: ByteArray? = null,
  17 + /** 匹配到的 SpeakerProfile.id，null = 没匹配上 / 还没匹配 */
  18 + val speakerId: Long? = null,
  19 + /** 与 matched profile 的余弦相似度，null = 没匹配 */
  20 + val speakerSim: Float? = null,
14 21 )
15 22  
16 23 @Entity(tableName = "transcripts")
... ... @@ -18,7 +25,7 @@ data class Transcript(
18 25 @PrimaryKey val segmentId: Long,
19 26 val text: String,
20 27 val lang: String = "zh",
21   - val engine: String = "xfyun",
  28 + val engine: String = "paraformer-zh-on-device",
22 29 val confidence: Float = 1f,
23 30 val reviewedAt: Long? = null,
24 31 )
... ... @@ -32,3 +39,21 @@ data class Summary(
32 39 val model: String,
33 40 val createdAt: Long,
34 41 )
  42 +
  43 +@Entity(tableName = "speaker_profiles")
  44 +data class SpeakerProfile(
  45 + @PrimaryKey(autoGenerate = true) val id: Long = 0,
  46 + val name: String,
  47 + /** 192-dim float embedding packed as LE bytes，平均或单段 */
  48 + val embedding: ByteArray,
  49 + /** 用了多少段平均出来的 */
  50 + val sampleCount: Int = 1,
  51 + val createdAt: Long = System.currentTimeMillis(),
  52 +)
  53 +
  54 +/** FTS4 全文索引，内容跟随 Transcript 表。Room 自动建表 + 触发器同步。 */
  55 +@Fts4(contentEntity = Transcript::class, tokenizer = "unicode61")
  56 +@Entity(tableName = "transcript_fts")
  57 +data class TranscriptFts(
  58 + val text: String,
  59 +)
... ...
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/data/Repository.kt
  View file @894eb7c
... ... @@ -7,4 +7,5 @@ class Repository(ctx: Context) {
7 7 val segments = db.segmentDao()
8 8 val transcripts = db.transcriptDao()
9 9 val summaries = db.summaryDao()
  10 + val speakers = db.speakerDao()
10 11 }
... ...
lifelog-recorder/app/src/main/kotlin/com/lemon/lifelog/service/RecorderService.kt
  View file @894eb7c
... ... @@ -16,10 +16,15 @@ import com.k2fsa.sherpa.onnx.VadModelConfig
16 16 import com.lemon.lifelog.MainActivity
17 17 import com.lemon.lifelog.asr.ModelDownloader
18 18 import com.lemon.lifelog.asr.ParaformerEngine
  19 +import com.lemon.lifelog.asr.SpeakerEmbedder
  20 +import com.lemon.lifelog.asr.cosineSim
  21 +import com.lemon.lifelog.asr.toFloatArrayLe
  22 +import com.lemon.lifelog.asr.toLeBytes
19 23 import com.lemon.lifelog.audio.AudioCapture
20 24 import com.lemon.lifelog.audio.SegmentWriter
21 25 import com.lemon.lifelog.data.Repository
22 26 import com.lemon.lifelog.data.Segment
  27 +import com.lemon.lifelog.data.SpeakerProfile
23 28 import com.lemon.lifelog.data.Transcript
24 29 import java.io.File
25 30 import kotlinx.coroutines.CoroutineScope
... ... @@ -50,6 +55,15 @@ class RecorderService : Service() {
50 55 private lateinit var segWriter: SegmentWriter
51 56 private var vad: Vad? = null
52 57 private var engine: ParaformerEngine? = null
  58 + private var speakerEmbedder: SpeakerEmbedder? = null
  59 +
  60 + /** Anchor profiles cached in memory; refreshed when DB changes. */
  61 + @Volatile
  62 + private var profiles: List<CachedProfile> = emptyList()
  63 + private data class CachedProfile(val id: Long, val name: String, val embedding: FloatArray)
  64 +
  65 + /** Cosine threshold above which we assign a profile to a segment. Tuned for CAMPPlus. */
  66 + private val speakerMatchThreshold = 0.5f
53 67  
54 68 override fun onCreate() {
55 69 super.onCreate()
... ... @@ -72,6 +86,7 @@ class RecorderService : Service() {
72 86 capture.stop()
73 87 vad?.release()
74 88 engine?.release()
  89 + speakerEmbedder?.release()
75 90 scope.cancel()
76 91 wakeLock?.takeIf { it.isHeld }?.release()
77 92 wakeLock = null
... ... @@ -130,10 +145,30 @@ class RecorderService : Service() {
130 145 ),
131 146 )
132 147  
  148 + // 2c. Speaker embedder (optional; download is 28 MB, may take a bit on cold start).
  149 + try {
  150 + RecorderState.setStatus("loading speaker model…")
  151 + val speakerFile = downloader.ensureSpeakerModel { msg ->
  152 + Log.i(TAG, "speaker dl: $msg")
  153 + RecorderState.setStatus(msg)
  154 + }
  155 + if (speakerFile.present) {
  156 + speakerEmbedder = SpeakerEmbedder.load(speakerFile.path, numThreads = 2)
  157 + }
  158 + } catch (t: Throwable) {
  159 + Log.w(TAG, "speaker embedder not available; segments will skip embedding", t)
  160 + }
  161 +
  162 + // 2d. Load anchor profiles (cosine matching).
  163 + refreshProfiles()
  164 + scope.launch {
  165 + repo.speakers.observeAll().collect { _ -> refreshProfiles() }
  166 + }
  167 +
133 168 // 3. Start capturing.
134 169 capture.start(scope)
135 170 RecorderState.setStatus("listening")
136   - Log.i(TAG, "pipeline running")
  171 + Log.i(TAG, "pipeline running speakers=${profiles.size}")
137 172  
138 173 capture.frames.collect { samples ->
139 174 val v = vad ?: return@collect
... ... @@ -153,12 +188,12 @@ class RecorderService : Service() {
153 188 val now = System.currentTimeMillis()
154 189 val startTs = now - audioMs
155 190  
156   - // a. persist audio (off CPU thread)
  191 + // a. persist audio (off CPU thread, Opus encode).
157 192 val audioPath = kotlinx.coroutines.withContext(Dispatchers.IO) {
158 193 segWriter.write(samples, startTs, sr)
159 194 }
160 195  
161   - // b. paraformer
  196 + // b. paraformer.
162 197 val wallStart = System.nanoTime()
163 198 val text = try {
164 199 engine!!.transcribeFloat(samples, sr).text
... ... @@ -168,7 +203,33 @@ class RecorderService : Service() {
168 203 }
169 204 val wallMs = (System.nanoTime() - wallStart) / 1_000_000
170 205  
171   - // c. Room
  206 + // c. speaker embedding + nearest-profile match.
  207 + val embedding: FloatArray? = speakerEmbedder?.let { embedder ->
  208 + try {
  209 + embedder.embed(samples, sr)
  210 + } catch (t: Throwable) {
  211 + Log.w(TAG, "embed failed", t)
  212 + null
  213 + }
  214 + }
  215 + val (matchedSpeakerId: Long?, matchedSim: Float?, matchedName: String?) =
  216 + if (embedding != null && profiles.isNotEmpty()) {
  217 + var bestId: Long? = null
  218 + var bestName: String? = null
  219 + var bestSim = -1f
  220 + for (p in profiles) {
  221 + val sim = cosineSim(embedding, p.embedding)
  222 + if (sim > bestSim) {
  223 + bestSim = sim
  224 + bestId = p.id
  225 + bestName = p.name
  226 + }
  227 + }
  228 + if (bestSim >= speakerMatchThreshold) Triple(bestId, bestSim, bestName)
  229 + else Triple(null, bestSim, null)
  230 + } else Triple(null, null, null)
  231 +
  232 + // d. Room insert.
172 233 val segmentId = kotlinx.coroutines.withContext(Dispatchers.IO) {
173 234 val id = repo.segments.insert(
174 235 Segment(
... ... @@ -177,6 +238,9 @@ class RecorderService : Service() {
177 238 audioPath = audioPath,
178 239 durationMs = audioMs,
179 240 energyDb = 0f,
  241 + embedding = embedding?.toLeBytes(),
  242 + speakerId = matchedSpeakerId,
  243 + speakerSim = matchedSim,
180 244 ),
181 245 )
182 246 if (text.isNotEmpty()) {
... ... @@ -184,19 +248,29 @@ class RecorderService : Service() {
184 248 Transcript(
185 249 segmentId = id,
186 250 text = text,
187   - lang = "zh",
188   - engine = "paraformer-zh-on-device",
189   - confidence = 1f,
190 251 ),
191 252 )
192 253 }
193 254 id
194 255 }
195 256  
196   - Log.i(TAG, "segment#$segmentId audio=${audioMs}ms wall=${wallMs}ms text=$text")
  257 + Log.i(
  258 + TAG,
  259 + "segment#$segmentId audio=${audioMs}ms wall=${wallMs}ms " +
  260 + "speaker=${matchedName ?: "—"} sim=${matchedSim?.let { "%.2f".format(it) } ?: "—"} " +
  261 + "text=$text",
  262 + )
197 263 RecorderState.onSegment(text, audioMs, wallMs)
198 264 }
199 265  
  266 + private suspend fun refreshProfiles() {
  267 + val rows = kotlinx.coroutines.withContext(Dispatchers.IO) { repo.speakers.all() }
  268 + profiles = rows.map {
  269 + CachedProfile(it.id, it.name, it.embedding.toFloatArrayLe())
  270 + }
  271 + Log.i(TAG, "loaded ${profiles.size} speaker profiles")
  272 + }
  273 +
200 274 private fun startForegroundCompat() {
201 275 val notification = buildNotification()
202 276 if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.UPSIDE_DOWN_CAKE) {
... ...