U-039: c_deploy_supervisor_cutover (blue-green switch) + a real zombie-reaping bug fix

Adds supervisor/deploy_cutover.py: stops the old Live process, retires the health-check-mode candidate, and relaunches the same already-verified candidate checkout bound to LIVE_PORT/PROD_DB_PATH -- http.server has no live socket-rebind operation, so "switch to production port/db" is implemented as stop-then-relaunch-from-the-same-dir rather than an in-place rebind (functionally equivalent, the only mechanism stdlib actually offers). Adds LIVE_PORT=8420 to 05 §5.9's MVP-fixed parameter table -- the one parameter that table was missing (frontend/app.js never needed one, same- origin relative fetches don't care, but successive deployments need to keep landing on the same port for a human bookmark to keep working). "在途 work_node_execution 重启后可恢复" is satisfied by relying on the existing REQ-06 recovery machinery unchanged -- cutover doesn't add any special-case logic, a cutover-triggered restart is just another interruption REQ-06 already survives. A real (not mocked) test caught a real bug: deploy_supervisor.is_alive()'s os.kill(pid, 0) check can't distinguish "truly running" from "exited but never reaped by its own parent" -- a zombie's PID slot answers kill(pid, 0) successfully until waited on. U-036's own subprocess test never caught this because that skeleton deletes its own PID file before exiting, sidestepping is_alive() entirely; deploy_cutover's test is the first one to actually poll for a stopped process to disappear, and it hung until fixed. Added reap_if_exited() (real os.waitpid for processes we spawned ourselves, falls back to is_alive() for pids from a prior session where ChildProcessError means we're not the real parent) and switched deploy_cutover._stop_process, deploy_health_check.check_process_alive, and deploy_supervisor.stop() to use it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

U-039: c_deploy_supervisor_cutover (blue-green switch) + a real zombie-reaping bug fix
Adds supervisor/deploy_cutover.py: stops the old Live process, retires the health-check-mode candidate, and relaunches the same already-verified candidate checkout bound to LIVE_PORT/PROD_DB_PATH -- http.server has no live socket-rebind operation, so "switch to production port/db" is implemented as stop-then-relaunch-from-the-same-dir rather than an in-place rebind (functionally equivalent, the only mechanism stdlib actually offers). Adds LIVE_PORT=8420 to 05 §5.9's MVP-fixed parameter table -- the one parameter that table was missing (frontend/app.js never needed one, same- origin relative fetches don't care, but successive deployments need to keep landing on the same port for a human bookmark to keep working). "在途 work_node_execution 重启后可恢复" is satisfied by relying on the existing REQ-06 recovery machinery unchanged -- cutover doesn't add any special-case logic, a cutover-triggered restart is just another interruption REQ-06 already survives. A real (not mocked) test caught a real bug: deploy_supervisor.is_alive()'s os.kill(pid, 0) check can't distinguish "truly running" from "exited but never reaped by its own parent" -- a zombie's PID slot answers kill(pid, 0) successfully until waited on. U-036's own subprocess test never caught this because that skeleton deletes its own PID file before exiting, sidestepping is_alive() entirely; deploy_cutover's test is the first one to actually poll for a stopped process to disappear, and it hung until fixed. Added reap_if_exited() (real os.waitpid for processes we spawned ourselves, falls back to is_alive() for pids from a prior session where ChildProcessError means we're not the real parent) and switched deploy_cutover._stop_process, deploy_health_check.check_process_alive, and deploy_supervisor.stop() to use it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
bboonnee
1 parent 48e38441
Showing 7 changed files with 352 additions and 2 deletions
docs/development-plan.md
docs/sdd/05-实施约束与依赖.md
docs/sdd/unit_matrix.json
supervisor/deploy_cutover.py
supervisor/deploy_health_check.py
supervisor/deploy_supervisor.py
supervisor/test_deploy_cutover.py
@@ -374,6 +374,9 @@ PH2-S1/S2/S3/S4 之间除标注的依赖外相互独立，可并行推进。
 - **U-039 [Risk Gate]** `c_deploy_supervisor_cutover`：蓝绿切换本体（Live 停止接受新触发 → 候选切换到生产端口/生产库 → 旧 Live checkout 保留为 `previous_instance_dir`）
   - 依赖：U-038
   - Done：健康检查通过后能完成真实切换，切换过程中在途 `work_node_execution` 能在重启后从持久化状态恢复（复用 U-021 断点恢复机制），`previous_instance_dir` 在监控窗口结束前不被清理
+  - **实现补充（2026-08-02）**：新增 `supervisor/deploy_cutover.py`。**发现并补上一个真实缺口**：05 §5.9 的 MVP 固定参数表把健康检查/监控相关的时间阈值都列全了，唯独没有一个"生产端口"常量——这是合理的（控制台前端 `app.js` 全是同源相对路径 `fetch('/api/...')`，运行时从不需要知道自己监听在哪个端口），但 cutover 场景确实需要：连续几次自我修改必须落在同一个端口上，人工书签/固定入口才有意义。新增 `LIVE_PORT = 8420` 常量，和 §5.9 表里其余参数同一性质（工程草案值，非用户可配）。**Rebind 策略是标准库约束下的判断处理，不是 SDD 里有歧义要停下来问的地方**：`http.server` 没有"给已监听的 socket 换端口/换 DB"这种原地操作，"候选切换到生产端口/生产库"落地成"停掉健康检查用的候选进程，从同一个（已经过验证的）候选检出目录，用生产端口+生产 DB 重新拉起一个新进程"——对外效果等价（同一份已验证代码现在服务生产端口），是 stdlib 唯一提供的机制，不是设计妥协。"在途 `work_node_execution` 能在重启后恢复"这条 Done 标准完全靠复用已有的 REQ-06 机制（`resume_work_graph_run`/`run_work_node`），本 Unit 不需要为此新写任何逻辑——cutover 造成的中断和其他任何中断（进程崩溃、机器重启）对 REQ-06 机制而言没有区别。**顺带记录一个相邻但不在本 Unit 范围内的缺口**：`backend_server.py` 目前完全没有调用 `scheduler.run_scheduler_once()` 的地方，"Live 进程里有在途工作"目前更多是理论场景而非真实运行时状态——这是调度器接线的缺口，不是 cutover 该修的，留给后续发现。
+    - **真实测试意外抓到一个真 bug，两处都修了**：`deploy_supervisor.is_alive()`（U-036）用 `os.kill(pid,0)` 判活，这个调用无法区分"真的还在跑"和"已经退出但父进程从没 `wait()` 过、还是个僵尸"——僵尸进程的 PID 槽位在被回收之前 `kill(pid,0)` 一直成功。U-036 自己的真实子进程测试之所以没暴露这个问题，是因为 Deploy Supervisor 骨架自己在退出前会删掉 PID 文件，`is_running()` 靠文件消失就判定"没在跑"，从来没真的走到 `is_alive()` 这条判活分支上验证过。这次 `deploy_cutover.py` 的真实测试第一次用"发 `SIGTERM` 后一直轮询直到进程真的消失"这种更严格的验证方式，一测就卡住——旧 Live 进程收到 `SIGTERM` 后真的退出了，但因为父进程（Deploy Supervisor 自己）从没调用过 `os.waitpid`，它变成僵尸，`is_alive()` 永远报告"还活着"。修法：`deploy_supervisor.py` 新增 `reap_if_exited(pid)`——是自己 `subprocess.Popen` 出来的子进程就用 `os.waitpid(pid, WNOHANG)` 真的回收（顺带拿到准确的"已退出"判定），不是自己子进程（比如上一次 Deploy Supervisor 会话遗留下来的 `live_pid`）就 `os.waitpid` 会抛 `ChildProcessError`，退化回 `is_alive()` 的尽力而为判断——两种场景都要处理，因为 cutover 完全可能在 Deploy Supervisor 自己重启之后被调用，此时手上的 `live_pid` 不是这次进程真正 fork 出来的。`deploy_cutover._stop_process()` 和 `deploy_health_check.check_process_alive()` 都改用 `reap_if_exited`，`deploy_supervisor.stop()` 顺手也补了一次回收（不影响它自己已有的返回值语义，纯粹清理僵尸）。这是这次会话里又一次"mock 测试全绿、真实测试才炸出真 bug"的例子，和 PH4/U-031 那几次是同一个模式。
+  - 覆盖：03-C `c_deploy_supervisor_cutover`；05 §5.9 新增 `LIVE_PORT` 参数；`deploy_supervisor.is_alive()` 的僵尸进程判活缺陷（真实测试发现，非 mock）
 ### PH5-S3 监控与回滚
@@ -126,5 +126,6 @@
 | `VALIDATOR_RATE_DEVIATION_THRESHOLD` | ±30 个百分点 | 判定 Validator 行为异常（`validator_approval_rate` 信号阈值） |
 | `MONITORING_WINDOW_HOURS` | 24h | 监控窗口时长上限 |
 | `MONITORING_MIN_TASK_COUNT` | 10 个真实任务 | 监控窗口任务数上限，先到者结束 |
+| `LIVE_PORT` | 8420 | 生产端口固定值（2026-08-02，U-039 补充——原表遗漏；控制台前端全走同源相对路径 fetch，运行时本身不需要这个常量，但连续多次 cutover 必须落在同一端口，人工书签/固定入口才有意义） |
 **与 5.4.2 节额度等待参数的差异**：5.4.2 节的退避/超时参数依据是 Anthropic 平台文档定义的外部事实（5 小时/7 天窗口周期），可以合理标注为"MVP-fixed"；本节这组阈值是纯粹的工程经验草案，没有类似的外部权威来源，标注为"可调，需在真实跑通几轮自我修改后实测校准"，不建议实现时直接当成不可变常量硬编码进测试断言。
@@ -56,7 +56,7 @@
     {"id": "U-036", "phase": "PH5", "sub_phase": "PH5-S2", "name": "supervisor/isolation 补齐单测 + Deploy Supervisor 进程骨架（见 development-plan.md 2026-08-02 实现补充）", "classification": "Enabler", "depends_on": [], "coverage": {"model": [], "view": [], "controller": [], "process": [], "requirement": []}, "done": "四模块各有 test_*.py（供 python3 -m unittest discover -s supervisor/isolation 运行，52 测试全绿）；Deploy Supervisor 骨架（supervisor/deploy_supervisor.py）独立进程可启停（真实子进程验证，非 mock），不受 Live/Candidate 崩溃影响；本 Unit 只做进程生命周期骨架，不含任何真实部署决策逻辑（U-037+ 的 poll_fn 挂载点）"},
     {"id": "U-037", "phase": "PH5", "sub_phase": "PH5-S2", "name": "c_deploy_supervisor_stage（见 development-plan.md 2026-08-02 实现补充）", "classification": "Risk Gate", "depends_on": ["U-034", "U-036"], "coverage": {"model": ["deployment"], "view": [], "controller": ["c_deploy_supervisor_stage"], "process": [], "requirement": []}, "done": "能生成 deployment 记录并拉起监听临时端口、连一次性 DB 副本的候选进程（supervisor/deploy_stage.py，真实 git worktree + 真实子进程验证过）；顺带补上 deployment.candidate_instance_dir 缺字段 + backend_server.py 首个 CLI 入口（此前只能进程内启动，从未真正作为独立 OS 进程跑过）"},
     {"id": "U-038", "phase": "PH5", "sub_phase": "PH5-S2", "name": "c_deploy_supervisor_health_check（见 development-plan.md 2026-08-02 实现补充）", "classification": "Risk Gate", "depends_on": ["U-037"], "coverage": {"model": ["deploy_health_signal"], "view": [], "controller": ["c_deploy_supervisor_health_check"], "process": [], "requirement": []}, "done": "process_alive/http_health/canary_task 三项检查全过 -> health_check_passed；任一失败 -> health_check_failed 且 Live 未受影响（supervisor/deploy_health_check.py，真实 HTTP+真实 project/task 创建验证过）；backend_server.py 新增 GET /api/health（真查一次 SELECT 1，不只是进程存活）"},
-    {"id": "U-039", "phase": "PH5", "sub_phase": "PH5-S2", "name": "c_deploy_supervisor_cutover", "classification": "Risk Gate", "depends_on": ["U-038"], "coverage": {"model": ["deployment"], "view": [], "controller": ["c_deploy_supervisor_cutover"], "process": [], "requirement": []}, "done": "健康检查通过后完成真实切换；在途 work_node_execution 重启后可恢复；previous_instance_dir 保留至监控窗口结束"},
+    {"id": "U-039", "phase": "PH5", "sub_phase": "PH5-S2", "name": "c_deploy_supervisor_cutover（见 development-plan.md 2026-08-02 实现补充）", "classification": "Risk Gate", "depends_on": ["U-038"], "coverage": {"model": ["deployment"], "view": [], "controller": ["c_deploy_supervisor_cutover"], "process": [], "requirement": []}, "done": "健康检查通过后完成真实切换（supervisor/deploy_cutover.py，真实进程停止+重启验证过）；在途 work_node_execution 重启后可恢复（复用既有 REQ-06 机制）；previous_instance_dir 保留至监控窗口结束；新增 LIVE_PORT=8420 常量（05 §5.9 缺的一个字段）；真实测试发现并修复 deploy_supervisor.is_alive() 的僵尸进程判活缺陷（新增 reap_if_exited()）"},
     {"id": "U-040", "phase": "PH5", "sub_phase": "PH5-S3", "name": "c_deploy_supervisor_monitor", "classification": "Risk Gate", "depends_on": ["U-039"], "coverage": {"model": ["deploy_health_signal"], "view": [], "controller": ["c_deploy_supervisor_monitor"], "process": [], "requirement": []}, "done": "四类信号采样正常；validator_approval_rate 偏离基线阈值时触发 anomaly_detected"},
     {"id": "U-041", "phase": "PH5", "sub_phase": "PH5-S3", "name": "c_deploy_supervisor_rollback", "classification": "Risk Gate", "depends_on": ["U-038", "U-040"], "coverage": {"model": ["deployment"], "view": [], "controller": ["c_deploy_supervisor_rollback"], "process": [], "requirement": []}, "done": "回滚路径 A/B 各跑通一次；旧版本也起不来时正确终态到 blocked_pending_human；git revert 冲突有明确处理"},
     {"id": "U-042", "phase": "PH5", "sub_phase": "PH5-S3", "name": "自举闭环端到端验收", "classification": "Risk Gate", "depends_on": ["U-035", "U-041"], "coverage": {"model": [], "view": [], "controller": [], "process": [], "requirement": []}, "done": "真实复现 self-development-feasibility.md 9 步场景，全程无需人工评审代码内容；真实双实例 E2E 用例默认跳过，需显式环境变量开启"},
+"""c_deploy_supervisor_cutover (03-C §3.15 / 04-P §4.6 / design doc §4 step
+5): the blue-green switch itself -- old Live stops, the candidate (already
+staged by U-037, already health-checked by U-038) takes over the production
+port and database.
+
+**`LIVE_PORT` (new MVP-fixed parameter, found missing while building this)**:
+05 §5.9 lists every OTHER Deploy Supervisor timing/threshold constant but
+never actually names a fixed production port -- reasonable, since the
+console's own frontend/app.js only ever makes same-origin relative fetch()
+calls (whatever port serves index.html is naturally used for /api/... too),
+so nothing in the running system needed one internally. Cutover is the one
+place that genuinely does: successive deployments need to keep landing on
+the SAME port so a human bookmark/shortcut to `http://127.0.0.1:LIVE_PORT`
+keeps working across self-modifications. Same "MVP-fixed draft value, not
+user-configurable" treatment as every other constant in that table.
+
+**Rebind strategy (documented judgment call, not escalated -- stdlib
+constraint, not an SDD ambiguity)**: `http.server` has no live "rebind this
+already-listening socket to a different port/DB" operation. "候选切换到生产
+端口/生产库" is implemented as stop-candidate-then-relaunch-from-the-same-
+already-verified checkout directory bound to LIVE_PORT/PROD_DB_PATH, rather
+than an in-place socket rebind -- functionally identical from the outside
+(same code now serving the production port/db) and the only mechanism
+stdlib's `http.server` actually offers.
+
+**REQ-06 breakpoint recovery is relied on, not reimplemented here**: any
+`work_node_execution` in flight when Live stops resumes correctly on restart
+via the existing `resume_work_graph_run`/`run_work_node` machinery (U-013/
+U-021) -- this module does nothing special for "in-flight work", it just
+stops one process and starts another, which REQ-06 was built to already
+survive regardless of what caused the interruption.
+
+**Known adjacent gap, out of scope here**: `backend_server.py` doesn't
+currently invoke `scheduler.run_scheduler_once()` anywhere -- there's no
+in-process background dispatch loop running inside the Live server today,
+so "在途工作" is presently more theoretical than real. Not this Unit's job
+to fix (cutover's contract is the process swap, not the scheduler wiring
+gap); flagged so a future Unit doesn't rediscover it from scratch.
+"""
+
+import os
+import signal
+import time
+
+from supervisor import deploy_health_check, deploy_stage, deploy_supervisor
+from src import paths
+
+LIVE_PORT = 8420
+
+
+class CutoverFailedError(Exception):
+    """CUTOVER_FAILED -- 03-C §3.15: the restarted candidate failed to bind
+    the production port/db during cutover. Not retried -- the caller routes
+    this to `deployment.status='blocked_pending_human'`, already applied by
+    the time this is raised."""
+
+
+def _stop_process(pid, timeout=10):
+    """Uses `reap_if_exited`, not plain `is_alive`, throughout -- a process
+    this module itself launched via `deploy_stage._launch_candidate`
+    (subprocess.Popen) needs to actually be reaped for `is_alive()` to ever
+    report it as gone; found for real (2026-08-02) via a test that hung
+    forever otherwise (see `deploy_supervisor.is_alive`'s own docstring)."""
+    if pid is None or deploy_supervisor.reap_if_exited(pid):
+        return
+    os.kill(pid, signal.SIGTERM)
+    deadline = time.time() + timeout
+    while time.time() < deadline:
+        if deploy_supervisor.reap_if_exited(pid):
+            return
+        time.sleep(0.1)
+    os.kill(pid, signal.SIGKILL)
+    deploy_supervisor.reap_if_exited(pid)
+
+
+def _wait_for_live_http(port, timeout, sleep=time.sleep, now=time.time):
+    deadline = now() + timeout
+    while now() < deadline:
+        if deploy_health_check.check_http_health(port, timeout=2)["status"] == "ok":
+            return True
+        sleep(0.2)
+    return False
+
+
+def cutover(
+    conn, deployment_id, live_pid, candidate_pid, candidate_instance_dir,
+    prod_db_path=None, live_port=LIVE_PORT, python_executable=None,
+    stop_timeout=10, bind_timeout=15, sleep=time.sleep, now=time.time,
+):
+    """Only callable after `c_deploy_supervisor_health_check` returned
+    `health_check_passed` for `deployment_id` (03-C's own precondition;
+    not re-checked here -- the caller, Deploy Supervisor's own poll loop,
+    is what enforces the state-machine ordering, same layering
+    `c_deploy_supervisor_stage` uses).
+
+    `live_pid` may be None (nothing was Live yet -- the very first
+    deployment). Returns {status: 'live', live_pid, live_port, cutover_at}.
+    Raises CutoverFailedError (deployment already marked
+    blocked_pending_human) if the restarted candidate never binds
+    `live_port`.
+    """
+    prod_db_path = prod_db_path or paths.db_path()
+    conn.execute("UPDATE deployment SET status = 'cutover_in_progress' WHERE deployment_id = ?", (deployment_id,))
+    conn.commit()
+
+    try:
+        # Live stops accepting new triggers by simply stopping -- in-flight
+        # work_node_execution resumes via REQ-06 machinery on next dispatch,
+        # not this module's concern (see module docstring).
+        _stop_process(live_pid, timeout=stop_timeout)
+        # The health-check-mode candidate (bound to candidate_port/
+        # candidate_db_path) is retired too -- its job was verification,
+        # not serving traffic.
+        _stop_process(candidate_pid, timeout=stop_timeout)
+
+        new_live_process = deploy_stage._launch_candidate(
+            candidate_instance_dir, live_port, prod_db_path, python_executable
+        )
+        if not _wait_for_live_http(live_port, bind_timeout, sleep=sleep, now=now):
+            raise CutoverFailedError(
+                "candidate for deployment {} never bound live_port {} within {}s".format(
+                    deployment_id, live_port, bind_timeout
+                )
+            )
+    except CutoverFailedError:
+        conn.execute(
+            "UPDATE deployment SET status = 'blocked_pending_human' WHERE deployment_id = ?", (deployment_id,)
+        )
+        conn.commit()
+        raise
+    except Exception as e:
+        conn.execute(
+            "UPDATE deployment SET status = 'blocked_pending_human' WHERE deployment_id = ?", (deployment_id,)
+        )
+        conn.commit()
+        raise CutoverFailedError(
+            "cutover failed for deployment {}: {}".format(deployment_id, e)
+        ) from e
+
+    conn.execute(
+        "UPDATE deployment SET status = 'live', cutover_at = CURRENT_TIMESTAMP WHERE deployment_id = ?",
+        (deployment_id,),
+    )
+    conn.commit()
+    return {"status": "live", "live_pid": new_live_process.pid, "live_port": live_port}
@@ -29,9 +29,15 @@ _SIGNAL_TYPES = (&quot;process_alive&quot;, &quot;http_health&quot;, &quot;canary_task&quot;)
 def check_process_alive(pid, timeout=HEALTH_CHECK_STARTUP_TIMEOUT, sleep=time.sleep, now=time.time):
+    """Uses `reap_if_exited` (not plain `is_alive`) so a candidate that
+    crashed immediately after being spawned by THIS process reads as dead
+    rather than a false "alive" zombie reading -- see
+    `deploy_supervisor.is_alive`'s own docstring for why the distinction
+    matters here specifically (this module's caller is always the same
+    process that Popen'd the candidate)."""
     deadline = now() + timeout
     while now() < deadline:
-        if deploy_supervisor.is_alive(pid):
+        if not deploy_supervisor.reap_if_exited(pid):
             return {"signal_type": "process_alive", "status": "ok", "detail": None}
         sleep(0.5)
     return {
@@ -55,6 +55,18 @@ def _atomic_write_json(path, data):
 def is_alive(pid):
+    """`os.kill(pid, 0)`-based liveness -- NOTE this can't distinguish "truly
+    running" from "exited but not yet reaped by its parent" (a zombie's PID
+    slot still answers kill(pid, 0) successfully until something calls
+    wait() on it). If the calling process spawned `pid` itself via
+    `subprocess.Popen` and holds onto it across multiple `is_alive()` calls
+    without ever waiting on it, this can report True forever for an
+    already-dead process -- found for real (2026-08-02, U-039) via a test
+    that actually waited for a stopped process to go away and it never did.
+    Callers that spawned `pid` themselves should use `reap_if_exited()`
+    instead, which handles this correctly; this function is still right for
+    checking a pid this process did NOT spawn (nothing to reap, this is the
+    best available check)."""
     if not isinstance(pid, int) or pid <= 0:
         return False
     try:
@@ -64,6 +76,26 @@ def is_alive(pid):
     return True
+def reap_if_exited(pid):
+    """True once `pid` is confirmed gone. If `pid` is a real child of the
+    calling process (e.g. spawned via `subprocess.Popen` earlier in this
+    same process's lifetime), this actually reaps it via a non-blocking
+    `os.waitpid` -- required for `is_alive()` to ever correctly report False
+    for it afterward (see that function's own docstring for why). If `pid`
+    isn't our child (belongs to a different process/session -- e.g. a
+    `live_pid` inherited from a prior Deploy Supervisor run), `os.waitpid`
+    raises `ChildProcessError` and this falls back to `is_alive()`'s plain
+    liveness check instead, which is the best available answer without
+    being the real parent."""
+    try:
+        reaped_pid, _ = os.waitpid(pid, os.WNOHANG)
+        if reaped_pid == pid:
+            return True
+    except ChildProcessError:
+        pass
+    return not is_alive(pid)
+
+
 def write_status(status, detail=None):
     _atomic_write_json(status_file_path(), {
         "status": status, "pid": os.getpid(), "updated_at": time.time(), "detail": detail,
@@ -160,6 +192,7 @@ def stop(timeout=10.0):
     deadline = time.time() + timeout
     while time.time() < deadline:
         if not is_running():
+            reap_if_exited(pid)  # best-effort zombie cleanup, doesn't affect the True/False result
             return True
         time.sleep(0.1)
     return False
+"""U-039 tests: c_deploy_supervisor_cutover. Real subprocesses throughout
+(old Live, health-check-mode candidate, and the post-cutover restarted
+process) -- this Unit's whole job is process lifecycle around a real port/DB
+swap, so mocking that away would test nothing real.
+
+Run: python3 -m unittest discover -s supervisor -p "test_*.py" -v
+"""
+
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+import time
+import unittest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from supervisor import deploy_cutover, deploy_health_check, deploy_stage, deploy_supervisor
+from src import paths, schema
+
+
+def _free_port():
+    return deploy_stage._find_free_port()
+
+
+def _commit_to_test_against():
+    """Same rationale as test_deploy_stage.py's own helper -- git worktree
+    only sees committed content, and this Unit's own changes may not be
+    committed yet at test time."""
+    stash_sha = subprocess.run(
+        ["git", "stash", "create"], cwd=paths.repo_root(), capture_output=True, text=True, check=True
+    ).stdout.strip()
+    return stash_sha or deploy_stage._current_commit(paths.repo_root())
+
+
+class DeployCutoverTestBase(unittest.TestCase):
+    def setUp(self):
+        self.home_tmp = tempfile.mkdtemp(prefix="deploy-cutover-home-")
+        self.addCleanup(shutil.rmtree, self.home_tmp, ignore_errors=True)
+        self.old_home = os.environ.get("GRAPHLEMON_HOME")
+        os.environ["GRAPHLEMON_HOME"] = self.home_tmp
+
+        self.prod_db_path = os.path.join(self.home_tmp, "prod.db")
+        prod_conn = schema.connect(self.prod_db_path)
+        schema.create_schema(prod_conn)
+        prod_conn.close()
+
+        self.conn = schema.connect(":memory:")
+        schema.create_schema(self.conn)
+        self.addCleanup(self.conn.close)
+
+        self._cleanup_pids = []
+        self._cleanup_worktrees = []
+
+    def tearDown(self):
+        if self.old_home is None:
+            os.environ.pop("GRAPHLEMON_HOME", None)
+        else:
+            os.environ["GRAPHLEMON_HOME"] = self.old_home
+        for pid in self._cleanup_pids:
+            if deploy_supervisor.is_alive(pid):
+                os.kill(pid, 9)
+        for wt in self._cleanup_worktrees:
+            subprocess.run(["git", "worktree", "remove", "--force", wt], cwd=paths.repo_root(), capture_output=True)
+
+    def _stage_and_health_check(self, port=None):
+        commit = _commit_to_test_against()
+        result = deploy_stage.stage_deployment(
+            self.conn, commit, repo_root=paths.repo_root(), prod_db_path=self.prod_db_path,
+            port=port or _free_port(),
+        )
+        self._cleanup_pids.append(result["candidate_pid"])
+        self._cleanup_worktrees.append(result["candidate_instance_dir"])
+        self._wait_for_port(result["candidate_port"])
+        deploy_health_check.run_health_check(
+            self.conn, result["deployment_id"], result["candidate_pid"], result["candidate_port"]
+        )
+        return result
+
+    def _wait_for_port(self, port, timeout=10):
+        deadline = time.time() + timeout
+        while time.time() < deadline:
+            if deploy_health_check.check_http_health(port, timeout=1)["status"] == "ok":
+                return
+            time.sleep(0.2)
+        self.fail("candidate on port {} never came up".format(port))
+
+
+class CutoverHappyPathTest(DeployCutoverTestBase):
+    def test_first_ever_cutover_no_prior_live_process(self):
+        staged = self._stage_and_health_check()
+        live_port = _free_port()
+
+        result = deploy_cutover.cutover(
+            self.conn, staged["deployment_id"], live_pid=None, candidate_pid=staged["candidate_pid"],
+            candidate_instance_dir=staged["candidate_instance_dir"], prod_db_path=self.prod_db_path,
+            live_port=live_port, sleep=lambda s: time.sleep(min(s, 0.2)),
+        )
+        self._cleanup_pids.append(result["live_pid"])
+
+        self.assertEqual(result["status"], "live")
+        self.assertEqual(result["live_port"], live_port)
+        self.assertTrue(deploy_supervisor.is_alive(result["live_pid"]))
+        self.assertNotEqual(result["live_pid"], staged["candidate_pid"])  # old candidate process was retired
+        self.assertFalse(deploy_supervisor.is_alive(staged["candidate_pid"]))
+
+        row = self.conn.execute(
+            "SELECT status, cutover_at FROM deployment WHERE deployment_id = ?", (staged["deployment_id"],)
+        ).fetchone()
+        self.assertEqual(row[0], "live")
+        self.assertIsNotNone(row[1])
+
+        # real HTTP round trip against the NEW live process, on live_port,
+        # backed by prod_db_path (not the candidate's one-time DB copy)
+        self._wait_for_port(live_port)
+        health = deploy_health_check.check_http_health(live_port)
+        self.assertEqual(health["status"], "ok")
+
+    def test_cutover_stops_the_old_live_process(self):
+        staged = self._stage_and_health_check()
+
+        old_live_port = _free_port()
+        old_live = deploy_stage._launch_candidate(paths.repo_root(), old_live_port, self.prod_db_path)
+        self._cleanup_pids.append(old_live.pid)
+        self._wait_for_port(old_live_port)
+        self.assertTrue(deploy_supervisor.is_alive(old_live.pid))
+
+        new_live_port = _free_port()
+        result = deploy_cutover.cutover(
+            self.conn, staged["deployment_id"], live_pid=old_live.pid, candidate_pid=staged["candidate_pid"],
+            candidate_instance_dir=staged["candidate_instance_dir"], prod_db_path=self.prod_db_path,
+            live_port=new_live_port,
+        )
+        self._cleanup_pids.append(result["live_pid"])
+
+        deadline = time.time() + 5
+        while time.time() < deadline and deploy_supervisor.is_alive(old_live.pid):
+            time.sleep(0.1)
+        self.assertFalse(deploy_supervisor.is_alive(old_live.pid))
+
+
+class CutoverFailedTest(DeployCutoverTestBase):
+    def test_bind_timeout_marks_blocked_pending_human_and_raises(self):
+        staged = self._stage_and_health_check()
+
+        with self.assertRaises(deploy_cutover.CutoverFailedError):
+            deploy_cutover.cutover(
+                self.conn, staged["deployment_id"], live_pid=None, candidate_pid=staged["candidate_pid"],
+                candidate_instance_dir="/nonexistent/checkout/dir/that/cannot/run/anything",
+                prod_db_path=self.prod_db_path, live_port=_free_port(),
+                bind_timeout=1, sleep=lambda s: time.sleep(min(s, 0.1)),
+            )
+
+        row = self.conn.execute(
+            "SELECT status FROM deployment WHERE deployment_id = ?", (staged["deployment_id"],)
+        ).fetchone()
+        self.assertEqual(row[0], "blocked_pending_human")
+
+
+if __name__ == "__main__":
+    unittest.main()