U-037 fix: bootstrap a fresh db when prod_db_path doesn't exist yet

The third real self-hosting attempt (first to actually reach stage_deployment itself, past the earlier bootstrap/commit bugs) hit an uncaught FileNotFoundError: _copy_prod_db() assumed prod_db_path always exists and shutil.copy2()'d it unconditionally. graphlemon has never actually run as a persistent service, so ~/.graphlemon/graphlemon.db never existed. stage_deployment() already handles this exact class of "no prior state" bootstrap case for previous_commit_hash/previous_instance_dir -- prod_db_path never got the same treatment, and every existing test seeds a prod db in setUp so this was never exercised. Fix: when prod_db_path doesn't exist, create a fresh schema'd db instead of copying -- semantically correct, since a genuinely-first deployment has no production data to inherit. New regression test (BootstrapProdDbTest) covers it. 555 tests green (435 src + 120 supervisor). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

U-037 fix: bootstrap a fresh db when prod_db_path doesn't exist yet
The third real self-hosting attempt (first to actually reach stage_deployment itself, past the earlier bootstrap/commit bugs) hit an uncaught FileNotFoundError: _copy_prod_db() assumed prod_db_path always exists and shutil.copy2()'d it unconditionally. graphlemon has never actually run as a persistent service, so ~/.graphlemon/graphlemon.db never existed. stage_deployment() already handles this exact class of "no prior state" bootstrap case for previous_commit_hash/previous_instance_dir -- prod_db_path never got the same treatment, and every existing test seeds a prod db in setUp so this was never exercised. Fix: when prod_db_path doesn't exist, create a fresh schema'd db instead of copying -- semantically correct, since a genuinely-first deployment has no production data to inherit. New regression test (BootstrapProdDbTest) covers it. 555 tests green (435 src + 120 supervisor). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
bboonnee
1 parent 438607d6
Showing 4 changed files with 48 additions and 4 deletions
docs/development-plan.md
docs/sdd/unit_matrix.json
supervisor/deploy_stage.py
supervisor/test_deploy_stage.py
@@ -365,6 +365,7 @@ PH2-S1/S2/S3/S4 之间除标注的依赖外相互独立，可并行推进。
   - **实现补充（2026-08-02）**：新增 `supervisor/deploy_stage.py`：`stage_deployment()` 建 `deployment` 行、`git worktree add --detach`（不是 `-b <branch>`，候选检出是只读验证用途，不打算再接受提交，和 `work_node_run.create_worktree()` 的编码用 worktree 是两回事）到全新目录、复制生产 DB 到一次性副本、`subprocess.Popen` 拉起候选进程。**返回值订正（低风险，按判断处理未升级询问）**：03-C `output_contract` 字面写返回 `{deployment_id, status:'staged'}`，但 04-P §4.6 状态转移表明确写"`c_deploy_supervisor_stage` 完成"这个触发条件本身就是把 `deployment.status` 从 `staged` 推进到 `candidate_starting`——两处矛盾，按"更详细的 04-P 状态机表为准"处理，函数完成时返回并持久化的都是 `candidate_starting`，不是字面上的 `staged`。**发现并补上两个真实缺口**：① `deployment` 表 01-M 字段清单里完全没有"候选检出目录存在哪"这一列——`candidate_port`/`candidate_db_path` 都有，唯独检出目录没地方存，Deploy Supervisor 若在 staging 和 cutover 之间重启就会彻底丢失候选目录位置；`schema.py` 补了 `candidate_instance_dir` 列（同一次 finalized 部署的这个字段，就是下一次 staging 时 `previous_instance_dir` 的来源——candidate 一旦顺利转正就是新 Live，回滚要回退到的正是它）。② `backend_server.py` 从 U-007 建骨架以来就没有任何 CLI 入口——`create_server()` 只在测试里以进程内方式实例化过，从没人真正把它作为独立 OS 进程启动过；候选进程恰好需要这个能力，顺手补了 `main()`（`python3 -m src.backend_server --host --port --db-path --skill-repo-path`，`SIGTERM` 走 `server.shutdown()`——必须在另一个线程调用，同线程调用 `serve_forever()` 期间调 `shutdown()` 会按 `http.server` 自己的文档死锁）。`/api/health` 端点本身留给 U-038 自己去加，不在这里超范围实现。
     - **验证方式**：`BootstrapPreviousTest`/`StageDeploymentHappyPathTest`/`CheckoutFailedTest` 用真实 sqlite + 真实 git worktree 操作（只 mock 掉真正拉候选进程这一步，验证 `stage_deployment()` 自己的逻辑）；`RealCandidateLaunchTest` 完全不 mock——对着 graphlemon 自己仓库真实 `git worktree add --detach` 检出 + 真实 `python3 -m src.backend_server` 子进程拉起 + 真实 HTTP 请求验证候选进程确实在真实监听、真实响应，验证完 `SIGTERM`（超时兜底 `SIGKILL`）+ `git worktree remove` 清理干净。这个真实测试意外命中一个开发时序问题：`git worktree add` 只看已提交内容，而这次实现是先写代码再测试再提交（标准流程），测试跑的时候 `main()` 还没提交，导致 checkout 出来的 `backend_server.py` 是没有 `main()` 的旧版本，候选进程"启动"后什么也不做直接退出——用 `git stash create`（不碰工作区/HEAD，只生成一个代表当前工作区状态的 commit 对象）替代直接 `HEAD`，让测试始终对着"当前工作区实际内容"验证，而不是"上一次提交的内容"；提交后本地工作区变干净时 `git stash create` 打印空字符串，自动回退到普通 `HEAD`，两种时序都覆盖。
   - 覆盖：03-C `c_deploy_supervisor_stage`；01-M `deployment.candidate_instance_dir`（补字段）；`backend_server.py` 新增可独立运行的 CLI 入口
+  - **真实 bug（2026-08-03，U-042 第三次真实自举压测才第一次真正跑到这一步）**：前两次真实压测分别卡在"人设文件没部署"和"worktree 改动没提交"这两个更早的环节，这是第一次真正执行到 `stage_deployment()` 本体。`stage_deployment()` 自己已经正确处理了"没有上一条 finalized deployment"这个引导期边界情况（`previous_commit_hash`/`previous_instance_dir` 回退到当前正在跑的 commit/checkout），但 `_copy_prod_db()` 完全没有对应处理——它假设 `prod_db_path` 永远已经存在，直接 `shutil.copy2()`，而 graphlemon 从未真正作为一个长期运行的服务跑过，`~/.graphlemon/graphlemon.db` 根本不存在，第一次真实调用直接抛出未捕获的 `FileNotFoundError`。本文件 `test_deploy_stage.py` 里每一个既有测试的 `setUp()` 都会先建好一份 `prod_db_path`，所以这个真实场景之前从未被测过。修复：`prod_db_path` 不存在时，不再尝试复制，改为直接在候选副本的位置新建一份带完整 schema 的空库（`schema.connect()` + `schema.create_schema()`）——语义上是对的：第一次部署本来就没有生产数据可继承，候选实例应该从一份干净但可用的库开始，这份库后续 cutover 成功后就会真正变成生产库。新增回归测试 `BootstrapProdDbTest`，验证 `prod_db_path` 不存在时不再崩溃，且生成的候选库真的可用（能查询 `task` 表）。
  
 - **U-038 [Risk Gate]** `c_deploy_supervisor_health_check`：`process_alive`/`http_health`/`canary_task` 三项 pre-cutover 检查
   - 依赖：U-037
@@ -54,7 +54,7 @@
     {"id": "U-034", "phase": "PH5", "sub_phase": "PH5-S1", "name": "01-M 扩展：deployment/deploy_health_signal + core_engine 枚举（见 development-plan.md 2026-08-02 实现补充）", "classification": "Enabler", "depends_on": ["U-001"], "coverage": {"model": ["deployment", "deploy_health_signal", "bug_record"], "view": [], "controller": [], "process": [], "requirement": []}, "done": "两张新表建表成功（共 17 张）；bug_record 新枚举值向后兼容 U-015/U-018 已有测试；deployment 状态机用 01-M 1.15 完整 15 值枚举；ux_deployment_active 表达式索引强制全局单例非终态约束；bug_record.py 应用层白名单刻意未跟着放开 core_engine，classification+路由逻辑留给 U-035"},
     {"id": "U-035", "phase": "PH5", "sub_phase": "PH5-S1", "name": "self-target worktree 无条件强制 + core_engine 分类审计（见 development-plan.md 2026-08-02 实现补充）", "classification": "Risk Gate", "depends_on": ["U-013", "U-016", "U-034"], "coverage": {"model": [], "view": [], "controller": [], "process": [], "requirement": ["REQ-03"]}, "done": "self-target 时无条件走 worktree（work_graph_orchestrate.run_graph 内强制）；core_engine 分类（src/core_engine_classify.py，路径规则）在 self-target 任务 PASS 收尾时记一条审计用 bug_record；且改动落地后重跑 U-013/REQ-03（worktree 隔离）现有回归用例全部通过。**2026-08-02 用户决策修正**：Deploy Supervisor 触发条件按 supervisor-auto-rollback-design.md §1 字面设计（self-target+PASS 即触发，bug_record 是旁路产物不是前提），而非 03-C/04-P 字面文本要求的\"先有 core_engine bug_record\"——后者会让从不产生 bug_record 的 goal 型自我改进任务永远触发不了 Deploy Supervisor，已用 AskUserQuestion 与用户确认此订正。c_org_loop_route 未改动（fast_lane/full_replan 是 FAIL 路径重试路由，与此处 PASS 路径的蓝绿触发正交），deployment 行创建仍是 U-037 的范围，不在本 Unit 提前做"},
     {"id": "U-036", "phase": "PH5", "sub_phase": "PH5-S2", "name": "supervisor/isolation 补齐单测 + Deploy Supervisor 进程骨架（见 development-plan.md 2026-08-02 实现补充）", "classification": "Enabler", "depends_on": [], "coverage": {"model": [], "view": [], "controller": [], "process": [], "requirement": []}, "done": "四模块各有 test_*.py（供 python3 -m unittest discover -s supervisor/isolation 运行，52 测试全绿）；Deploy Supervisor 骨架（supervisor/deploy_supervisor.py）独立进程可启停（真实子进程验证，非 mock），不受 Live/Candidate 崩溃影响；本 Unit 只做进程生命周期骨架，不含任何真实部署决策逻辑（U-037+ 的 poll_fn 挂载点）"},
-    {"id": "U-037", "phase": "PH5", "sub_phase": "PH5-S2", "name": "c_deploy_supervisor_stage（见 development-plan.md 2026-08-02 实现补充）", "classification": "Risk Gate", "depends_on": ["U-034", "U-036"], "coverage": {"model": ["deployment"], "view": [], "controller": ["c_deploy_supervisor_stage"], "process": [], "requirement": []}, "done": "能生成 deployment 记录并拉起监听临时端口、连一次性 DB 副本的候选进程（supervisor/deploy_stage.py，真实 git worktree + 真实子进程验证过）；顺带补上 deployment.candidate_instance_dir 缺字段 + backend_server.py 首个 CLI 入口（此前只能进程内启动，从未真正作为独立 OS 进程跑过）"},
+    {"id": "U-037", "phase": "PH5", "sub_phase": "PH5-S2", "name": "c_deploy_supervisor_stage（见 development-plan.md 2026-08-02/03 实现补充）", "classification": "Risk Gate", "depends_on": ["U-034", "U-036"], "coverage": {"model": ["deployment"], "view": [], "controller": ["c_deploy_supervisor_stage"], "process": [], "requirement": []}, "done": "能生成 deployment 记录并拉起监听临时端口、连一次性 DB 副本的候选进程（supervisor/deploy_stage.py，真实 git worktree + 真实子进程验证过）；顺带补上 deployment.candidate_instance_dir 缺字段 + backend_server.py 首个 CLI 入口。2026-08-03：U-042 第三次真实自举压测发现 _copy_prod_db() 没有处理 prod_db_path 首次不存在的引导期场景（真实 FileNotFoundError），已修复为自动新建带 schema 的空库并补测试"},
     {"id": "U-038", "phase": "PH5", "sub_phase": "PH5-S2", "name": "c_deploy_supervisor_health_check（见 development-plan.md 2026-08-02 实现补充）", "classification": "Risk Gate", "depends_on": ["U-037"], "coverage": {"model": ["deploy_health_signal"], "view": [], "controller": ["c_deploy_supervisor_health_check"], "process": [], "requirement": []}, "done": "process_alive/http_health/canary_task 三项检查全过 -> health_check_passed；任一失败 -> health_check_failed 且 Live 未受影响（supervisor/deploy_health_check.py，真实 HTTP+真实 project/task 创建验证过）；backend_server.py 新增 GET /api/health（真查一次 SELECT 1，不只是进程存活）"},
     {"id": "U-039", "phase": "PH5", "sub_phase": "PH5-S2", "name": "c_deploy_supervisor_cutover（见 development-plan.md 2026-08-02 实现补充）", "classification": "Risk Gate", "depends_on": ["U-038"], "coverage": {"model": ["deployment"], "view": [], "controller": ["c_deploy_supervisor_cutover"], "process": [], "requirement": []}, "done": "健康检查通过后完成真实切换（supervisor/deploy_cutover.py，真实进程停止+重启验证过）；在途 work_node_execution 重启后可恢复（复用既有 REQ-06 机制）；previous_instance_dir 保留至监控窗口结束；新增 LIVE_PORT=8420 常量（05 §5.9 缺的一个字段）；真实测试发现并修复 deploy_supervisor.is_alive() 的僵尸进程判活缺陷（新增 reap_if_exited()）"},
     {"id": "U-040", "phase": "PH5", "sub_phase": "PH5-S3", "name": "c_deploy_supervisor_monitor（见 development-plan.md 2026-08-02 实现补充）", "classification": "Risk Gate", "depends_on": ["U-039"], "coverage": {"model": ["deploy_health_signal"], "view": [], "controller": ["c_deploy_supervisor_monitor"], "process": [], "requirement": []}, "done": "四类信号采样正常（supervisor/deploy_monitor.py）；validator_approval_rate 偏离基线阈值时触发 anomaly_detected；approve 占比算法基于对 scheduler.finish_task_lifecycle 真实代码逻辑的核实（PASS 从不产生 bug_record，FAIL 必然产生），非字面 SDD 定义"},
@@ -32,7 +32,7 @@ import subprocess
 import sys
 import uuid
  
-from src import paths
+from src import paths, schema
  
  
 class CheckoutFailedError(Exception):
@@ -97,9 +97,24 @@ def _checkout_candidate(repo_root, git_commit_hash, deployment_id, checkout_pare
 def _copy_prod_db(prod_db_path, deployment_id):
     """MVP: whole-file copy (supervisor-auto-rollback-design.md §10 open
     question 2 already flags this as the simple-first choice, revisit only
-    if a large production DB makes this slow)."""
+    if a large production DB makes this slow).
+
+    **Bootstrap (2026-08-03, first real self-hosting run)**: the very first
+    deployment ever has no `prod_db_path` to copy from yet -- nothing has
+    run as a persistent service before, the same class of "no prior state"
+    gap `stage_deployment()` already handles for `previous_commit_hash`/
+    `previous_instance_dir`. Copying a nonexistent file raised
+    `FileNotFoundError` uncaught the first time this actually ran for real.
+    A fresh, correctly-schema'd db is the right substitute -- there's no
+    production data to inherit yet, so an empty-but-valid db is exactly
+    what a genuinely-first candidate should start from."""
     dest = os.path.join(_candidate_dbs_dir(), deployment_id + ".db")
-    shutil.copy2(prod_db_path, dest)
+    if os.path.exists(prod_db_path):
+        shutil.copy2(prod_db_path, dest)
+    else:
+        conn = schema.connect(dest)
+        schema.create_schema(conn)
+        conn.close()
     return dest
  
  
@@ -108,6 +108,34 @@ class BootstrapPreviousTest(DeployStageTestBase):
         self.assertEqual(row[1], "/old/candidate/dir")
  
  
+class BootstrapProdDbTest(DeployStageTestBase):
+    """A real self-hosting run (2026-08-03) hit this uncaught: the very
+    first deployment ever has no prod_db_path file yet (nothing has run as
+    a persistent service before) -- _copy_prod_db() used to assume it
+    always exists and crashed with a raw FileNotFoundError. Every other
+    test in this file seeds prod_db_path in setUp, so this exact case had
+    never been exercised."""
+
+    def test_missing_prod_db_bootstraps_a_fresh_schema_instead_of_crashing(self):
+        os.remove(self.prod_db_path)
+        self.assertFalse(os.path.exists(self.prod_db_path))
+
+        with patch.object(deploy_stage, "_launch_candidate") as mock_launch:
+            mock_launch.return_value.pid = 12345
+            result = deploy_stage.stage_deployment(
+                self.conn, self.commit_v2, repo_root=self.repo_dir, prod_db_path=self.prod_db_path, port=9999,
+            )
+
+        candidate_db_path = self.conn.execute(
+            "SELECT candidate_db_path FROM deployment WHERE deployment_id = ?", (result["deployment_id"],)
+        ).fetchone()[0]
+        self.assertTrue(os.path.exists(candidate_db_path))
+        # the candidate db must be a real, usable schema, not an empty file
+        candidate_conn = schema.connect(candidate_db_path)
+        candidate_conn.execute("SELECT COUNT(*) FROM task").fetchone()
+        candidate_conn.close()
+
+
 class StageDeploymentHappyPathTest(DeployStageTestBase):
     def test_creates_candidate_starting_row_with_real_worktree_and_db_copy(self):
         with patch.object(deploy_stage, "_launch_candidate") as mock_launch: