diff --git a/docs/development-plan.md b/docs/development-plan.md index c8ace3a..0c67d7b 100644 --- a/docs/development-plan.md +++ b/docs/development-plan.md @@ -374,6 +374,9 @@ PH2-S1/S2/S3/S4 之间除标注的依赖外相互独立，可并行推进。 - **U-039 [Risk Gate]** `c_deploy_supervisor_cutover`：蓝绿切换本体（Live 停止接受新触发 → 候选切换到生产端口/生产库 → 旧 Live checkout 保留为 `previous_instance_dir`） - 依赖：U-038 - Done：健康检查通过后能完成真实切换，切换过程中在途 `work_node_execution` 能在重启后从持久化状态恢复（复用 U-021 断点恢复机制），`previous_instance_dir` 在监控窗口结束前不被清理 + - **实现补充（2026-08-02）**：新增 `supervisor/deploy_cutover.py`。**发现并补上一个真实缺口**：05 §5.9 的 MVP 固定参数表把健康检查/监控相关的时间阈值都列全了，唯独没有一个"生产端口"常量——这是合理的（控制台前端 `app.js` 全是同源相对路径 `fetch('/api/...')`，运行时从不需要知道自己监听在哪个端口），但 cutover 场景确实需要：连续几次自我修改必须落在同一个端口上，人工书签/固定入口才有意义。新增 `LIVE_PORT = 8420` 常量，和 §5.9 表里其余参数同一性质（工程草案值，非用户可配）。**Rebind 策略是标准库约束下的判断处理，不是 SDD 里有歧义要停下来问的地方**：`http.server` 没有"给已监听的 socket 换端口/换 DB"这种原地操作，"候选切换到生产端口/生产库"落地成"停掉健康检查用的候选进程，从同一个（已经过验证的）候选检出目录，用生产端口+生产 DB 重新拉起一个新进程"——对外效果等价（同一份已验证代码现在服务生产端口），是 stdlib 唯一提供的机制，不是设计妥协。"在途 `work_node_execution` 能在重启后恢复"这条 Done 标准完全靠复用已有的 REQ-06 机制（`resume_work_graph_run`/`run_work_node`），本 Unit 不需要为此新写任何逻辑——cutover 造成的中断和其他任何中断（进程崩溃、机器重启）对 REQ-06 机制而言没有区别。**顺带记录一个相邻但不在本 Unit 范围内的缺口**：`backend_server.py` 目前完全没有调用 `scheduler.run_scheduler_once()` 的地方，"Live 进程里有在途工作"目前更多是理论场景而非真实运行时状态——这是调度器接线的缺口，不是 cutover 该修的，留给后续发现。 + - **真实测试意外抓到一个真 bug，两处都修了**：`deploy_supervisor.is_alive()`（U-036）用 `os.kill(pid,0)` 判活，这个调用无法区分"真的还在跑"和"已经退出但父进程从没 `wait()` 过、还是个僵尸"——僵尸进程的 PID 槽位在被回收之前 `kill(pid,0)` 一直成功。U-036 自己的真实子进程测试之所以没暴露这个问题，是因为 Deploy Supervisor 骨架自己在退出前会删掉 PID 文件，`is_running()` 靠文件消失就判定"没在跑"，从来没真的走到 `is_alive()` 这条判活分支上验证过。这次 `deploy_cutover.py` 的真实测试第一次用"发 `SIGTERM` 后一直轮询直到进程真的消失"这种更严格的验证方式，一测就卡住——旧 Live 进程收到 `SIGTERM` 后真的退出了，但因为父进程（Deploy Supervisor 自己）从没调用过 `os.waitpid`，它变成僵尸，`is_alive()` 永远报告"还活着"。修法：`deploy_supervisor.py` 新增 `reap_if_exited(pid)`——是自己 `subprocess.Popen` 出来的子进程就用 `os.waitpid(pid, WNOHANG)` 真的回收（顺带拿到准确的"已退出"判定），不是自己子进程（比如上一次 Deploy Supervisor 会话遗留下来的 `live_pid`）就 `os.waitpid` 会抛 `ChildProcessError`，退化回 `is_alive()` 的尽力而为判断——两种场景都要处理，因为 cutover 完全可能在 Deploy Supervisor 自己重启之后被调用，此时手上的 `live_pid` 不是这次进程真正 fork 出来的。`deploy_cutover._stop_process()` 和 `deploy_health_check.check_process_alive()` 都改用 `reap_if_exited`，`deploy_supervisor.stop()` 顺手也补了一次回收（不影响它自己已有的返回值语义，纯粹清理僵尸）。这是这次会话里又一次"mock 测试全绿、真实测试才炸出真 bug"的例子，和 PH4/U-031 那几次是同一个模式。 + - 覆盖：03-C `c_deploy_supervisor_cutover`；05 §5.9 新增 `LIVE_PORT` 参数；`deploy_supervisor.is_alive()` 的僵尸进程判活缺陷（真实测试发现，非 mock） ### PH5-S3 监控与回滚 diff --git a/docs/sdd/05-实施约束与依赖.md b/docs/sdd/05-实施约束与依赖.md index 55dcc06..8b160b6 100644 --- a/docs/sdd/05-实施约束与依赖.md +++ b/docs/sdd/05-实施约束与依赖.md @@ -126,5 +126,6 @@ | `VALIDATOR_RATE_DEVIATION_THRESHOLD` | ±30 个百分点 | 判定 Validator 行为异常（`validator_approval_rate` 信号阈值） | | `MONITORING_WINDOW_HOURS` | 24h | 监控窗口时长上限 | | `MONITORING_MIN_TASK_COUNT` | 10 个真实任务 | 监控窗口任务数上限，先到者结束 | +| `LIVE_PORT` | 8420 | 生产端口固定值（2026-08-02，U-039 补充——原表遗漏；控制台前端全走同源相对路径 fetch，运行时本身不需要这个常量，但连续多次 cutover 必须落在同一端口，人工书签/固定入口才有意义） | **与 5.4.2 节额度等待参数的差异**：5.4.2 节的退避/超时参数依据是 Anthropic 平台文档定义的外部事实（5 小时/7 天窗口周期），可以合理标注为"MVP-fixed"；本节这组阈值是纯粹的工程经验草案，没有类似的外部权威来源，标注为"可调，需在真实跑通几轮自我修改后实测校准"，不建议实现时直接当成不可变常量硬编码进测试断言。 diff --git a/docs/sdd/unit_matrix.json b/docs/sdd/unit_matrix.json index efda63e..98ede73 100644 --- a/docs/sdd/unit_matrix.json +++ b/docs/sdd/unit_matrix.json @@ -56,7 +56,7 @@ {"id": "U-036", "phase": "PH5", "sub_phase": "PH5-S2", "name": "supervisor/isolation 补齐单测 + Deploy Supervisor 进程骨架（见 development-plan.md 2026-08-02 实现补充）", "classification": "Enabler", "depends_on": [], "coverage": {"model": [], "view": [], "controller": [], "process": [], "requirement": []}, "done": "四模块各有 test_*.py（供 python3 -m unittest discover -s supervisor/isolation 运行，52 测试全绿）；Deploy Supervisor 骨架（supervisor/deploy_supervisor.py）独立进程可启停（真实子进程验证，非 mock），不受 Live/Candidate 崩溃影响；本 Unit 只做进程生命周期骨架，不含任何真实部署决策逻辑（U-037+ 的 poll_fn 挂载点）"}, {"id": "U-037", "phase": "PH5", "sub_phase": "PH5-S2", "name": "c_deploy_supervisor_stage（见 development-plan.md 2026-08-02 实现补充）", "classification": "Risk Gate", "depends_on": ["U-034", "U-036"], "coverage": {"model": ["deployment"], "view": [], "controller": ["c_deploy_supervisor_stage"], "process": [], "requirement": []}, "done": "能生成 deployment 记录并拉起监听临时端口、连一次性 DB 副本的候选进程（supervisor/deploy_stage.py，真实 git worktree + 真实子进程验证过）；顺带补上 deployment.candidate_instance_dir 缺字段 + backend_server.py 首个 CLI 入口（此前只能进程内启动，从未真正作为独立 OS 进程跑过）"}, {"id": "U-038", "phase": "PH5", "sub_phase": "PH5-S2", "name": "c_deploy_supervisor_health_check（见 development-plan.md 2026-08-02 实现补充）", "classification": "Risk Gate", "depends_on": ["U-037"], "coverage": {"model": ["deploy_health_signal"], "view": [], "controller": ["c_deploy_supervisor_health_check"], "process": [], "requirement": []}, "done": "process_alive/http_health/canary_task 三项检查全过 -> health_check_passed；任一失败 -> health_check_failed 且 Live 未受影响（supervisor/deploy_health_check.py，真实 HTTP+真实 project/task 创建验证过）；backend_server.py 新增 GET /api/health（真查一次 SELECT 1，不只是进程存活）"}, - {"id": "U-039", "phase": "PH5", "sub_phase": "PH5-S2", "name": "c_deploy_supervisor_cutover", "classification": "Risk Gate", "depends_on": ["U-038"], "coverage": {"model": ["deployment"], "view": [], "controller": ["c_deploy_supervisor_cutover"], "process": [], "requirement": []}, "done": "健康检查通过后完成真实切换；在途 work_node_execution 重启后可恢复；previous_instance_dir 保留至监控窗口结束"}, + {"id": "U-039", "phase": "PH5", "sub_phase": "PH5-S2", "name": "c_deploy_supervisor_cutover（见 development-plan.md 2026-08-02 实现补充）", "classification": "Risk Gate", "depends_on": ["U-038"], "coverage": {"model": ["deployment"], "view": [], "controller": ["c_deploy_supervisor_cutover"], "process": [], "requirement": []}, "done": "健康检查通过后完成真实切换（supervisor/deploy_cutover.py，真实进程停止+重启验证过）；在途 work_node_execution 重启后可恢复（复用既有 REQ-06 机制）；previous_instance_dir 保留至监控窗口结束；新增 LIVE_PORT=8420 常量（05 §5.9 缺的一个字段）；真实测试发现并修复 deploy_supervisor.is_alive() 的僵尸进程判活缺陷（新增 reap_if_exited()）"}, {"id": "U-040", "phase": "PH5", "sub_phase": "PH5-S3", "name": "c_deploy_supervisor_monitor", "classification": "Risk Gate", "depends_on": ["U-039"], "coverage": {"model": ["deploy_health_signal"], "view": [], "controller": ["c_deploy_supervisor_monitor"], "process": [], "requirement": []}, "done": "四类信号采样正常；validator_approval_rate 偏离基线阈值时触发 anomaly_detected"}, {"id": "U-041", "phase": "PH5", "sub_phase": "PH5-S3", "name": "c_deploy_supervisor_rollback", "classification": "Risk Gate", "depends_on": ["U-038", "U-040"], "coverage": {"model": ["deployment"], "view": [], "controller": ["c_deploy_supervisor_rollback"], "process": [], "requirement": []}, "done": "回滚路径 A/B 各跑通一次；旧版本也起不来时正确终态到 blocked_pending_human；git revert 冲突有明确处理"}, {"id": "U-042", "phase": "PH5", "sub_phase": "PH5-S3", "name": "自举闭环端到端验收", "classification": "Risk Gate", "depends_on": ["U-035", "U-041"], "coverage": {"model": [], "view": [], "controller": [], "process": [], "requirement": []}, "done": "真实复现 self-development-feasibility.md 9 步场景，全程无需人工评审代码内容；真实双实例 E2E 用例默认跳过，需显式环境变量开启"}, diff --git a/supervisor/deploy_cutover.py b/supervisor/deploy_cutover.py new file mode 100644 index 0000000..7693e08 --- /dev/null +++ b/supervisor/deploy_cutover.py @@ -0,0 +1,145 @@ +"""c_deploy_supervisor_cutover (03-C §3.15 / 04-P §4.6 / design doc §4 step +5): the blue-green switch itself -- old Live stops, the candidate (already +staged by U-037, already health-checked by U-038) takes over the production +port and database. + +**`LIVE_PORT` (new MVP-fixed parameter, found missing while building this)**: +05 §5.9 lists every OTHER Deploy Supervisor timing/threshold constant but +never actually names a fixed production port -- reasonable, since the +console's own frontend/app.js only ever makes same-origin relative fetch() +calls (whatever port serves index.html is naturally used for /api/... too), +so nothing in the running system needed one internally. Cutover is the one +place that genuinely does: successive deployments need to keep landing on +the SAME port so a human bookmark/shortcut to `http://127.0.0.1:LIVE_PORT` +keeps working across self-modifications. Same "MVP-fixed draft value, not +user-configurable" treatment as every other constant in that table. + +**Rebind strategy (documented judgment call, not escalated -- stdlib +constraint, not an SDD ambiguity)**: `http.server` has no live "rebind this +already-listening socket to a different port/DB" operation. "候选切换到生产 +端口/生产库" is implemented as stop-candidate-then-relaunch-from-the-same- +already-verified checkout directory bound to LIVE_PORT/PROD_DB_PATH, rather +than an in-place socket rebind -- functionally identical from the outside +(same code now serving the production port/db) and the only mechanism +stdlib's `http.server` actually offers. + +**REQ-06 breakpoint recovery is relied on, not reimplemented here**: any +`work_node_execution` in flight when Live stops resumes correctly on restart +via the existing `resume_work_graph_run`/`run_work_node` machinery (U-013/ +U-021) -- this module does nothing special for "in-flight work", it just +stops one process and starts another, which REQ-06 was built to already +survive regardless of what caused the interruption. + +**Known adjacent gap, out of scope here**: `backend_server.py` doesn't +currently invoke `scheduler.run_scheduler_once()` anywhere -- there's no +in-process background dispatch loop running inside the Live server today, +so "在途工作" is presently more theoretical than real. Not this Unit's job +to fix (cutover's contract is the process swap, not the scheduler wiring +gap); flagged so a future Unit doesn't rediscover it from scratch. +""" + +import os +import signal +import time + +from supervisor import deploy_health_check, deploy_stage, deploy_supervisor +from src import paths + +LIVE_PORT = 8420 + + +class CutoverFailedError(Exception): + """CUTOVER_FAILED -- 03-C §3.15: the restarted candidate failed to bind + the production port/db during cutover. Not retried -- the caller routes + this to `deployment.status='blocked_pending_human'`, already applied by + the time this is raised.""" + + +def _stop_process(pid, timeout=10): + """Uses `reap_if_exited`, not plain `is_alive`, throughout -- a process + this module itself launched via `deploy_stage._launch_candidate` + (subprocess.Popen) needs to actually be reaped for `is_alive()` to ever + report it as gone; found for real (2026-08-02) via a test that hung + forever otherwise (see `deploy_supervisor.is_alive`'s own docstring).""" + if pid is None or deploy_supervisor.reap_if_exited(pid): + return + os.kill(pid, signal.SIGTERM) + deadline = time.time() + timeout + while time.time() < deadline: + if deploy_supervisor.reap_if_exited(pid): + return + time.sleep(0.1) + os.kill(pid, signal.SIGKILL) + deploy_supervisor.reap_if_exited(pid) + + +def _wait_for_live_http(port, timeout, sleep=time.sleep, now=time.time): + deadline = now() + timeout + while now() < deadline: + if deploy_health_check.check_http_health(port, timeout=2)["status"] == "ok": + return True + sleep(0.2) + return False + + +def cutover( + conn, deployment_id, live_pid, candidate_pid, candidate_instance_dir, + prod_db_path=None, live_port=LIVE_PORT, python_executable=None, + stop_timeout=10, bind_timeout=15, sleep=time.sleep, now=time.time, +): + """Only callable after `c_deploy_supervisor_health_check` returned + `health_check_passed` for `deployment_id` (03-C's own precondition; + not re-checked here -- the caller, Deploy Supervisor's own poll loop, + is what enforces the state-machine ordering, same layering + `c_deploy_supervisor_stage` uses). + + `live_pid` may be None (nothing was Live yet -- the very first + deployment). Returns {status: 'live', live_pid, live_port, cutover_at}. + Raises CutoverFailedError (deployment already marked + blocked_pending_human) if the restarted candidate never binds + `live_port`. + """ + prod_db_path = prod_db_path or paths.db_path() + conn.execute("UPDATE deployment SET status = 'cutover_in_progress' WHERE deployment_id = ?", (deployment_id,)) + conn.commit() + + try: + # Live stops accepting new triggers by simply stopping -- in-flight + # work_node_execution resumes via REQ-06 machinery on next dispatch, + # not this module's concern (see module docstring). + _stop_process(live_pid, timeout=stop_timeout) + # The health-check-mode candidate (bound to candidate_port/ + # candidate_db_path) is retired too -- its job was verification, + # not serving traffic. + _stop_process(candidate_pid, timeout=stop_timeout) + + new_live_process = deploy_stage._launch_candidate( + candidate_instance_dir, live_port, prod_db_path, python_executable + ) + if not _wait_for_live_http(live_port, bind_timeout, sleep=sleep, now=now): + raise CutoverFailedError( + "candidate for deployment {} never bound live_port {} within {}s".format( + deployment_id, live_port, bind_timeout + ) + ) + except CutoverFailedError: + conn.execute( + "UPDATE deployment SET status = 'blocked_pending_human' WHERE deployment_id = ?", (deployment_id,) + ) + conn.commit() + raise + except Exception as e: + conn.execute( + "UPDATE deployment SET status = 'blocked_pending_human' WHERE deployment_id = ?", (deployment_id,) + ) + conn.commit() + raise CutoverFailedError( + "cutover failed for deployment {}: {}".format(deployment_id, e) + ) from e + + conn.execute( + "UPDATE deployment SET status = 'live', cutover_at = CURRENT_TIMESTAMP WHERE deployment_id = ?", + (deployment_id,), + ) + conn.commit() + return {"status": "live", "live_pid": new_live_process.pid, "live_port": live_port} diff --git a/supervisor/deploy_health_check.py b/supervisor/deploy_health_check.py index a6e0388..31eb69a 100644 --- a/supervisor/deploy_health_check.py +++ b/supervisor/deploy_health_check.py @@ -29,9 +29,15 @@ _SIGNAL_TYPES = ("process_alive", "http_health", "canary_task") def check_process_alive(pid, timeout=HEALTH_CHECK_STARTUP_TIMEOUT, sleep=time.sleep, now=time.time): + """Uses `reap_if_exited` (not plain `is_alive`) so a candidate that + crashed immediately after being spawned by THIS process reads as dead + rather than a false "alive" zombie reading -- see + `deploy_supervisor.is_alive`'s own docstring for why the distinction + matters here specifically (this module's caller is always the same + process that Popen'd the candidate).""" deadline = now() + timeout while now() < deadline: - if deploy_supervisor.is_alive(pid): + if not deploy_supervisor.reap_if_exited(pid): return {"signal_type": "process_alive", "status": "ok", "detail": None} sleep(0.5) return { diff --git a/supervisor/deploy_supervisor.py b/supervisor/deploy_supervisor.py index a5f984b..b33e0cd 100644 --- a/supervisor/deploy_supervisor.py +++ b/supervisor/deploy_supervisor.py @@ -55,6 +55,18 @@ def _atomic_write_json(path, data): def is_alive(pid): + """`os.kill(pid, 0)`-based liveness -- NOTE this can't distinguish "truly + running" from "exited but not yet reaped by its parent" (a zombie's PID + slot still answers kill(pid, 0) successfully until something calls + wait() on it). If the calling process spawned `pid` itself via + `subprocess.Popen` and holds onto it across multiple `is_alive()` calls + without ever waiting on it, this can report True forever for an + already-dead process -- found for real (2026-08-02, U-039) via a test + that actually waited for a stopped process to go away and it never did. + Callers that spawned `pid` themselves should use `reap_if_exited()` + instead, which handles this correctly; this function is still right for + checking a pid this process did NOT spawn (nothing to reap, this is the + best available check).""" if not isinstance(pid, int) or pid <= 0: return False try: @@ -64,6 +76,26 @@ def is_alive(pid): return True +def reap_if_exited(pid): + """True once `pid` is confirmed gone. If `pid` is a real child of the + calling process (e.g. spawned via `subprocess.Popen` earlier in this + same process's lifetime), this actually reaps it via a non-blocking + `os.waitpid` -- required for `is_alive()` to ever correctly report False + for it afterward (see that function's own docstring for why). If `pid` + isn't our child (belongs to a different process/session -- e.g. a + `live_pid` inherited from a prior Deploy Supervisor run), `os.waitpid` + raises `ChildProcessError` and this falls back to `is_alive()`'s plain + liveness check instead, which is the best available answer without + being the real parent.""" + try: + reaped_pid, _ = os.waitpid(pid, os.WNOHANG) + if reaped_pid == pid: + return True + except ChildProcessError: + pass + return not is_alive(pid) + + def write_status(status, detail=None): _atomic_write_json(status_file_path(), { "status": status, "pid": os.getpid(), "updated_at": time.time(), "detail": detail, @@ -160,6 +192,7 @@ def stop(timeout=10.0): deadline = time.time() + timeout while time.time() < deadline: if not is_running(): + reap_if_exited(pid) # best-effort zombie cleanup, doesn't affect the True/False result return True time.sleep(0.1) return False diff --git a/supervisor/test_deploy_cutover.py b/supervisor/test_deploy_cutover.py new file mode 100644 index 0000000..e735550 --- /dev/null +++ b/supervisor/test_deploy_cutover.py @@ -0,0 +1,162 @@ +"""U-039 tests: c_deploy_supervisor_cutover. Real subprocesses throughout +(old Live, health-check-mode candidate, and the post-cutover restarted +process) -- this Unit's whole job is process lifecycle around a real port/DB +swap, so mocking that away would test nothing real. + +Run: python3 -m unittest discover -s supervisor -p "test_*.py" -v +""" + +import os +import shutil +import subprocess +import sys +import tempfile +import time +import unittest + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + +from supervisor import deploy_cutover, deploy_health_check, deploy_stage, deploy_supervisor +from src import paths, schema + + +def _free_port(): + return deploy_stage._find_free_port() + + +def _commit_to_test_against(): + """Same rationale as test_deploy_stage.py's own helper -- git worktree + only sees committed content, and this Unit's own changes may not be + committed yet at test time.""" + stash_sha = subprocess.run( + ["git", "stash", "create"], cwd=paths.repo_root(), capture_output=True, text=True, check=True + ).stdout.strip() + return stash_sha or deploy_stage._current_commit(paths.repo_root()) + + +class DeployCutoverTestBase(unittest.TestCase): + def setUp(self): + self.home_tmp = tempfile.mkdtemp(prefix="deploy-cutover-home-") + self.addCleanup(shutil.rmtree, self.home_tmp, ignore_errors=True) + self.old_home = os.environ.get("GRAPHLEMON_HOME") + os.environ["GRAPHLEMON_HOME"] = self.home_tmp + + self.prod_db_path = os.path.join(self.home_tmp, "prod.db") + prod_conn = schema.connect(self.prod_db_path) + schema.create_schema(prod_conn) + prod_conn.close() + + self.conn = schema.connect(":memory:") + schema.create_schema(self.conn) + self.addCleanup(self.conn.close) + + self._cleanup_pids = [] + self._cleanup_worktrees = [] + + def tearDown(self): + if self.old_home is None: + os.environ.pop("GRAPHLEMON_HOME", None) + else: + os.environ["GRAPHLEMON_HOME"] = self.old_home + for pid in self._cleanup_pids: + if deploy_supervisor.is_alive(pid): + os.kill(pid, 9) + for wt in self._cleanup_worktrees: + subprocess.run(["git", "worktree", "remove", "--force", wt], cwd=paths.repo_root(), capture_output=True) + + def _stage_and_health_check(self, port=None): + commit = _commit_to_test_against() + result = deploy_stage.stage_deployment( + self.conn, commit, repo_root=paths.repo_root(), prod_db_path=self.prod_db_path, + port=port or _free_port(), + ) + self._cleanup_pids.append(result["candidate_pid"]) + self._cleanup_worktrees.append(result["candidate_instance_dir"]) + self._wait_for_port(result["candidate_port"]) + deploy_health_check.run_health_check( + self.conn, result["deployment_id"], result["candidate_pid"], result["candidate_port"] + ) + return result + + def _wait_for_port(self, port, timeout=10): + deadline = time.time() + timeout + while time.time() < deadline: + if deploy_health_check.check_http_health(port, timeout=1)["status"] == "ok": + return + time.sleep(0.2) + self.fail("candidate on port {} never came up".format(port)) + + +class CutoverHappyPathTest(DeployCutoverTestBase): + def test_first_ever_cutover_no_prior_live_process(self): + staged = self._stage_and_health_check() + live_port = _free_port() + + result = deploy_cutover.cutover( + self.conn, staged["deployment_id"], live_pid=None, candidate_pid=staged["candidate_pid"], + candidate_instance_dir=staged["candidate_instance_dir"], prod_db_path=self.prod_db_path, + live_port=live_port, sleep=lambda s: time.sleep(min(s, 0.2)), + ) + self._cleanup_pids.append(result["live_pid"]) + + self.assertEqual(result["status"], "live") + self.assertEqual(result["live_port"], live_port) + self.assertTrue(deploy_supervisor.is_alive(result["live_pid"])) + self.assertNotEqual(result["live_pid"], staged["candidate_pid"]) # old candidate process was retired + self.assertFalse(deploy_supervisor.is_alive(staged["candidate_pid"])) + + row = self.conn.execute( + "SELECT status, cutover_at FROM deployment WHERE deployment_id = ?", (staged["deployment_id"],) + ).fetchone() + self.assertEqual(row[0], "live") + self.assertIsNotNone(row[1]) + + # real HTTP round trip against the NEW live process, on live_port, + # backed by prod_db_path (not the candidate's one-time DB copy) + self._wait_for_port(live_port) + health = deploy_health_check.check_http_health(live_port) + self.assertEqual(health["status"], "ok") + + def test_cutover_stops_the_old_live_process(self): + staged = self._stage_and_health_check() + + old_live_port = _free_port() + old_live = deploy_stage._launch_candidate(paths.repo_root(), old_live_port, self.prod_db_path) + self._cleanup_pids.append(old_live.pid) + self._wait_for_port(old_live_port) + self.assertTrue(deploy_supervisor.is_alive(old_live.pid)) + + new_live_port = _free_port() + result = deploy_cutover.cutover( + self.conn, staged["deployment_id"], live_pid=old_live.pid, candidate_pid=staged["candidate_pid"], + candidate_instance_dir=staged["candidate_instance_dir"], prod_db_path=self.prod_db_path, + live_port=new_live_port, + ) + self._cleanup_pids.append(result["live_pid"]) + + deadline = time.time() + 5 + while time.time() < deadline and deploy_supervisor.is_alive(old_live.pid): + time.sleep(0.1) + self.assertFalse(deploy_supervisor.is_alive(old_live.pid)) + + +class CutoverFailedTest(DeployCutoverTestBase): + def test_bind_timeout_marks_blocked_pending_human_and_raises(self): + staged = self._stage_and_health_check() + + with self.assertRaises(deploy_cutover.CutoverFailedError): + deploy_cutover.cutover( + self.conn, staged["deployment_id"], live_pid=None, candidate_pid=staged["candidate_pid"], + candidate_instance_dir="/nonexistent/checkout/dir/that/cannot/run/anything", + prod_db_path=self.prod_db_path, live_port=_free_port(), + bind_timeout=1, sleep=lambda s: time.sleep(min(s, 0.1)), + ) + + row = self.conn.execute( + "SELECT status FROM deployment WHERE deployment_id = ?", (staged["deployment_id"],) + ).fetchone() + self.assertEqual(row[0], "blocked_pending_human") + + +if __name__ == "__main__": + unittest.main()