宜博 / lemongraph

Commit 002bba468e74b283ee61a890d5aa1fcafa7d4e12

Authored by bboonnee
1 parent 48e38441

U-039: c_deploy_supervisor_cutover (blue-green switch) + a real zombie-reaping bug fix 

Adds supervisor/deploy_cutover.py: stops the old Live process, retires the
health-check-mode candidate, and relaunches the same already-verified
candidate checkout bound to LIVE_PORT/PROD_DB_PATH -- http.server has no
live socket-rebind operation, so "switch to production port/db" is
implemented as stop-then-relaunch-from-the-same-dir rather than an in-place
rebind (functionally equivalent, the only mechanism stdlib actually offers).

Adds LIVE_PORT=8420 to 05 §5.9's MVP-fixed parameter table -- the one
parameter that table was missing (frontend/app.js never needed one, same-
origin relative fetches don't care, but successive deployments need to keep
landing on the same port for a human bookmark to keep working).

"在途 work_node_execution 重启后可恢复" is satisfied by relying on the
existing REQ-06 recovery machinery unchanged -- cutover doesn't add any
special-case logic, a cutover-triggered restart is just another interruption
REQ-06 already survives.

A real (not mocked) test caught a real bug: deploy_supervisor.is_alive()'s
os.kill(pid, 0) check can't distinguish "truly running" from "exited but
never reaped by its own parent" -- a zombie's PID slot answers kill(pid, 0)
successfully until waited on. U-036's own subprocess test never caught this
because that skeleton deletes its own PID file before exiting, sidestepping
is_alive() entirely; deploy_cutover's test is the first one to actually poll
for a stopped process to disappear, and it hung until fixed. Added
reap_if_exited() (real os.waitpid for processes we spawned ourselves, falls
back to is_alive() for pids from a prior session where ChildProcessError
means we're not the real parent) and switched deploy_cutover._stop_process,
deploy_health_check.check_process_alive, and deploy_supervisor.stop() to use
it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Inline Side-by-side
Showing 7 changed files with 352 additions and 2 deletions
docs/development-plan.md
  View file @002bba4
... ... @@ -374,6 +374,9 @@ PH2-S1/S2/S3/S4 之间除标注的依赖外相互独立，可并行推进。
374 374 - **U-039 [Risk Gate]** `c_deploy_supervisor_cutover`：蓝绿切换本体（Live 停止接受新触发 → 候选切换到生产端口/生产库 → 旧 Live checkout 保留为 `previous_instance_dir`）
375 375 - 依赖：U-038
376 376 - Done：健康检查通过后能完成真实切换，切换过程中在途 `work_node_execution` 能在重启后从持久化状态恢复（复用 U-021 断点恢复机制），`previous_instance_dir` 在监控窗口结束前不被清理
  377 + - **实现补充（2026-08-02）**：新增 `supervisor/deploy_cutover.py`。**发现并补上一个真实缺口**：05 §5.9 的 MVP 固定参数表把健康检查/监控相关的时间阈值都列全了，唯独没有一个"生产端口"常量——这是合理的（控制台前端 `app.js` 全是同源相对路径 `fetch('/api/...')`，运行时从不需要知道自己监听在哪个端口），但 cutover 场景确实需要：连续几次自我修改必须落在同一个端口上，人工书签/固定入口才有意义。新增 `LIVE_PORT = 8420` 常量，和 §5.9 表里其余参数同一性质（工程草案值，非用户可配）。**Rebind 策略是标准库约束下的判断处理，不是 SDD 里有歧义要停下来问的地方**：`http.server` 没有"给已监听的 socket 换端口/换 DB"这种原地操作，"候选切换到生产端口/生产库"落地成"停掉健康检查用的候选进程，从同一个（已经过验证的）候选检出目录，用生产端口+生产 DB 重新拉起一个新进程"——对外效果等价（同一份已验证代码现在服务生产端口），是 stdlib 唯一提供的机制，不是设计妥协。"在途 `work_node_execution` 能在重启后恢复"这条 Done 标准完全靠复用已有的 REQ-06 机制（`resume_work_graph_run`/`run_work_node`），本 Unit 不需要为此新写任何逻辑——cutover 造成的中断和其他任何中断（进程崩溃、机器重启）对 REQ-06 机制而言没有区别。**顺带记录一个相邻但不在本 Unit 范围内的缺口**：`backend_server.py` 目前完全没有调用 `scheduler.run_scheduler_once()` 的地方，"Live 进程里有在途工作"目前更多是理论场景而非真实运行时状态——这是调度器接线的缺口，不是 cutover 该修的，留给后续发现。
  378 + - **真实测试意外抓到一个真 bug，两处都修了**：`deploy_supervisor.is_alive()`（U-036）用 `os.kill(pid,0)` 判活，这个调用无法区分"真的还在跑"和"已经退出但父进程从没 `wait()` 过、还是个僵尸"——僵尸进程的 PID 槽位在被回收之前 `kill(pid,0)` 一直成功。U-036 自己的真实子进程测试之所以没暴露这个问题，是因为 Deploy Supervisor 骨架自己在退出前会删掉 PID 文件，`is_running()` 靠文件消失就判定"没在跑"，从来没真的走到 `is_alive()` 这条判活分支上验证过。这次 `deploy_cutover.py` 的真实测试第一次用"发 `SIGTERM` 后一直轮询直到进程真的消失"这种更严格的验证方式，一测就卡住——旧 Live 进程收到 `SIGTERM` 后真的退出了，但因为父进程（Deploy Supervisor 自己）从没调用过 `os.waitpid`，它变成僵尸，`is_alive()` 永远报告"还活着"。修法：`deploy_supervisor.py` 新增 `reap_if_exited(pid)`——是自己 `subprocess.Popen` 出来的子进程就用 `os.waitpid(pid, WNOHANG)` 真的回收（顺带拿到准确的"已退出"判定），不是自己子进程（比如上一次 Deploy Supervisor 会话遗留下来的 `live_pid`）就 `os.waitpid` 会抛 `ChildProcessError`，退化回 `is_alive()` 的尽力而为判断——两种场景都要处理，因为 cutover 完全可能在 Deploy Supervisor 自己重启之后被调用，此时手上的 `live_pid` 不是这次进程真正 fork 出来的。`deploy_cutover._stop_process()` 和 `deploy_health_check.check_process_alive()` 都改用 `reap_if_exited`，`deploy_supervisor.stop()` 顺手也补了一次回收（不影响它自己已有的返回值语义，纯粹清理僵尸）。这是这次会话里又一次"mock 测试全绿、真实测试才炸出真 bug"的例子，和 PH4/U-031 那几次是同一个模式。
  379 + - 覆盖：03-C `c_deploy_supervisor_cutover`；05 §5.9 新增 `LIVE_PORT` 参数；`deploy_supervisor.is_alive()` 的僵尸进程判活缺陷（真实测试发现，非 mock）
377 380  
378 381 ### PH5-S3 监控与回滚
379 382  
... ...
docs/sdd/05-实施约束与依赖.md
  View file @002bba4
... ... @@ -126,5 +126,6 @@
126 126 | `VALIDATOR_RATE_DEVIATION_THRESHOLD` | ±30 个百分点 | 判定 Validator 行为异常（`validator_approval_rate` 信号阈值） |
127 127 | `MONITORING_WINDOW_HOURS` | 24h | 监控窗口时长上限 |
128 128 | `MONITORING_MIN_TASK_COUNT` | 10 个真实任务 | 监控窗口任务数上限，先到者结束 |
  129 +| `LIVE_PORT` | 8420 | 生产端口固定值（2026-08-02，U-039 补充——原表遗漏；控制台前端全走同源相对路径 fetch，运行时本身不需要这个常量，但连续多次 cutover 必须落在同一端口，人工书签/固定入口才有意义） |
129 130  
130 131 **与 5.4.2 节额度等待参数的差异**：5.4.2 节的退避/超时参数依据是 Anthropic 平台文档定义的外部事实（5 小时/7 天窗口周期），可以合理标注为"MVP-fixed"；本节这组阈值是纯粹的工程经验草案，没有类似的外部权威来源，标注为"可调，需在真实跑通几轮自我修改后实测校准"，不建议实现时直接当成不可变常量硬编码进测试断言。
... ...
docs/sdd/unit_matrix.json
  View file @002bba4
... ... @@ -56,7 +56,7 @@
56 56 {"id": "U-036", "phase": "PH5", "sub_phase": "PH5-S2", "name": "supervisor/isolation 补齐单测 + Deploy Supervisor 进程骨架（见 development-plan.md 2026-08-02 实现补充）", "classification": "Enabler", "depends_on": [], "coverage": {"model": [], "view": [], "controller": [], "process": [], "requirement": []}, "done": "四模块各有 test_*.py（供 python3 -m unittest discover -s supervisor/isolation 运行，52 测试全绿）；Deploy Supervisor 骨架（supervisor/deploy_supervisor.py）独立进程可启停（真实子进程验证，非 mock），不受 Live/Candidate 崩溃影响；本 Unit 只做进程生命周期骨架，不含任何真实部署决策逻辑（U-037+ 的 poll_fn 挂载点）"},
57 57 {"id": "U-037", "phase": "PH5", "sub_phase": "PH5-S2", "name": "c_deploy_supervisor_stage（见 development-plan.md 2026-08-02 实现补充）", "classification": "Risk Gate", "depends_on": ["U-034", "U-036"], "coverage": {"model": ["deployment"], "view": [], "controller": ["c_deploy_supervisor_stage"], "process": [], "requirement": []}, "done": "能生成 deployment 记录并拉起监听临时端口、连一次性 DB 副本的候选进程（supervisor/deploy_stage.py，真实 git worktree + 真实子进程验证过）；顺带补上 deployment.candidate_instance_dir 缺字段 + backend_server.py 首个 CLI 入口（此前只能进程内启动，从未真正作为独立 OS 进程跑过）"},
58 58 {"id": "U-038", "phase": "PH5", "sub_phase": "PH5-S2", "name": "c_deploy_supervisor_health_check（见 development-plan.md 2026-08-02 实现补充）", "classification": "Risk Gate", "depends_on": ["U-037"], "coverage": {"model": ["deploy_health_signal"], "view": [], "controller": ["c_deploy_supervisor_health_check"], "process": [], "requirement": []}, "done": "process_alive/http_health/canary_task 三项检查全过 -> health_check_passed；任一失败 -> health_check_failed 且 Live 未受影响（supervisor/deploy_health_check.py，真实 HTTP+真实 project/task 创建验证过）；backend_server.py 新增 GET /api/health（真查一次 SELECT 1，不只是进程存活）"},
59   - {"id": "U-039", "phase": "PH5", "sub_phase": "PH5-S2", "name": "c_deploy_supervisor_cutover", "classification": "Risk Gate", "depends_on": ["U-038"], "coverage": {"model": ["deployment"], "view": [], "controller": ["c_deploy_supervisor_cutover"], "process": [], "requirement": []}, "done": "健康检查通过后完成真实切换；在途 work_node_execution 重启后可恢复；previous_instance_dir 保留至监控窗口结束"},
  59 + {"id": "U-039", "phase": "PH5", "sub_phase": "PH5-S2", "name": "c_deploy_supervisor_cutover（见 development-plan.md 2026-08-02 实现补充）", "classification": "Risk Gate", "depends_on": ["U-038"], "coverage": {"model": ["deployment"], "view": [], "controller": ["c_deploy_supervisor_cutover"], "process": [], "requirement": []}, "done": "健康检查通过后完成真实切换（supervisor/deploy_cutover.py，真实进程停止+重启验证过）；在途 work_node_execution 重启后可恢复（复用既有 REQ-06 机制）；previous_instance_dir 保留至监控窗口结束；新增 LIVE_PORT=8420 常量（05 §5.9 缺的一个字段）；真实测试发现并修复 deploy_supervisor.is_alive() 的僵尸进程判活缺陷（新增 reap_if_exited()）"},
60 60 {"id": "U-040", "phase": "PH5", "sub_phase": "PH5-S3", "name": "c_deploy_supervisor_monitor", "classification": "Risk Gate", "depends_on": ["U-039"], "coverage": {"model": ["deploy_health_signal"], "view": [], "controller": ["c_deploy_supervisor_monitor"], "process": [], "requirement": []}, "done": "四类信号采样正常；validator_approval_rate 偏离基线阈值时触发 anomaly_detected"},
61 61 {"id": "U-041", "phase": "PH5", "sub_phase": "PH5-S3", "name": "c_deploy_supervisor_rollback", "classification": "Risk Gate", "depends_on": ["U-038", "U-040"], "coverage": {"model": ["deployment"], "view": [], "controller": ["c_deploy_supervisor_rollback"], "process": [], "requirement": []}, "done": "回滚路径 A/B 各跑通一次；旧版本也起不来时正确终态到 blocked_pending_human；git revert 冲突有明确处理"},
62 62 {"id": "U-042", "phase": "PH5", "sub_phase": "PH5-S3", "name": "自举闭环端到端验收", "classification": "Risk Gate", "depends_on": ["U-035", "U-041"], "coverage": {"model": [], "view": [], "controller": [], "process": [], "requirement": []}, "done": "真实复现 self-development-feasibility.md 9 步场景，全程无需人工评审代码内容；真实双实例 E2E 用例默认跳过，需显式环境变量开启"},
... ...
supervisor/deploy_cutover.py 0 → 100644
  View file @002bba4
  1 +"""c_deploy_supervisor_cutover (03-C §3.15 / 04-P §4.6 / design doc §4 step
  2 +5): the blue-green switch itself -- old Live stops, the candidate (already
  3 +staged by U-037, already health-checked by U-038) takes over the production
  4 +port and database.
  5 +
  6 +**`LIVE_PORT` (new MVP-fixed parameter, found missing while building this)**:
  7 +05 §5.9 lists every OTHER Deploy Supervisor timing/threshold constant but
  8 +never actually names a fixed production port -- reasonable, since the
  9 +console's own frontend/app.js only ever makes same-origin relative fetch()
  10 +calls (whatever port serves index.html is naturally used for /api/... too),
  11 +so nothing in the running system needed one internally. Cutover is the one
  12 +place that genuinely does: successive deployments need to keep landing on
  13 +the SAME port so a human bookmark/shortcut to `http://127.0.0.1:LIVE_PORT`
  14 +keeps working across self-modifications. Same "MVP-fixed draft value, not
  15 +user-configurable" treatment as every other constant in that table.
  16 +
  17 +**Rebind strategy (documented judgment call, not escalated -- stdlib
  18 +constraint, not an SDD ambiguity)**: `http.server` has no live "rebind this
  19 +already-listening socket to a different port/DB" operation. "候选切换到生产
  20 +端口/生产库" is implemented as stop-candidate-then-relaunch-from-the-same-
  21 +already-verified checkout directory bound to LIVE_PORT/PROD_DB_PATH, rather
  22 +than an in-place socket rebind -- functionally identical from the outside
  23 +(same code now serving the production port/db) and the only mechanism
  24 +stdlib's `http.server` actually offers.
  25 +
  26 +**REQ-06 breakpoint recovery is relied on, not reimplemented here**: any
  27 +`work_node_execution` in flight when Live stops resumes correctly on restart
  28 +via the existing `resume_work_graph_run`/`run_work_node` machinery (U-013/
  29 +U-021) -- this module does nothing special for "in-flight work", it just
  30 +stops one process and starts another, which REQ-06 was built to already
  31 +survive regardless of what caused the interruption.
  32 +
  33 +**Known adjacent gap, out of scope here**: `backend_server.py` doesn't
  34 +currently invoke `scheduler.run_scheduler_once()` anywhere -- there's no
  35 +in-process background dispatch loop running inside the Live server today,
  36 +so "在途工作" is presently more theoretical than real. Not this Unit's job
  37 +to fix (cutover's contract is the process swap, not the scheduler wiring
  38 +gap); flagged so a future Unit doesn't rediscover it from scratch.
  39 +"""
  40 +
  41 +import os
  42 +import signal
  43 +import time
  44 +
  45 +from supervisor import deploy_health_check, deploy_stage, deploy_supervisor
  46 +from src import paths
  47 +
  48 +LIVE_PORT = 8420
  49 +
  50 +
  51 +class CutoverFailedError(Exception):
  52 + """CUTOVER_FAILED -- 03-C §3.15: the restarted candidate failed to bind
  53 + the production port/db during cutover. Not retried -- the caller routes
  54 + this to `deployment.status='blocked_pending_human'`, already applied by
  55 + the time this is raised."""
  56 +
  57 +
  58 +def _stop_process(pid, timeout=10):
  59 + """Uses `reap_if_exited`, not plain `is_alive`, throughout -- a process
  60 + this module itself launched via `deploy_stage._launch_candidate`
  61 + (subprocess.Popen) needs to actually be reaped for `is_alive()` to ever
  62 + report it as gone; found for real (2026-08-02) via a test that hung
  63 + forever otherwise (see `deploy_supervisor.is_alive`'s own docstring)."""
  64 + if pid is None or deploy_supervisor.reap_if_exited(pid):
  65 + return
  66 + os.kill(pid, signal.SIGTERM)
  67 + deadline = time.time() + timeout
  68 + while time.time() < deadline:
  69 + if deploy_supervisor.reap_if_exited(pid):
  70 + return
  71 + time.sleep(0.1)
  72 + os.kill(pid, signal.SIGKILL)
  73 + deploy_supervisor.reap_if_exited(pid)
  74 +
  75 +
  76 +def _wait_for_live_http(port, timeout, sleep=time.sleep, now=time.time):
  77 + deadline = now() + timeout
  78 + while now() < deadline:
  79 + if deploy_health_check.check_http_health(port, timeout=2)["status"] == "ok":
  80 + return True
  81 + sleep(0.2)
  82 + return False
  83 +
  84 +
  85 +def cutover(
  86 + conn, deployment_id, live_pid, candidate_pid, candidate_instance_dir,
  87 + prod_db_path=None, live_port=LIVE_PORT, python_executable=None,
  88 + stop_timeout=10, bind_timeout=15, sleep=time.sleep, now=time.time,
  89 +):
  90 + """Only callable after `c_deploy_supervisor_health_check` returned
  91 + `health_check_passed` for `deployment_id` (03-C's own precondition;
  92 + not re-checked here -- the caller, Deploy Supervisor's own poll loop,
  93 + is what enforces the state-machine ordering, same layering
  94 + `c_deploy_supervisor_stage` uses).
  95 +
  96 + `live_pid` may be None (nothing was Live yet -- the very first
  97 + deployment). Returns {status: 'live', live_pid, live_port, cutover_at}.
  98 + Raises CutoverFailedError (deployment already marked
  99 + blocked_pending_human) if the restarted candidate never binds
  100 + `live_port`.
  101 + """
  102 + prod_db_path = prod_db_path or paths.db_path()
  103 + conn.execute("UPDATE deployment SET status = 'cutover_in_progress' WHERE deployment_id = ?", (deployment_id,))
  104 + conn.commit()
  105 +
  106 + try:
  107 + # Live stops accepting new triggers by simply stopping -- in-flight
  108 + # work_node_execution resumes via REQ-06 machinery on next dispatch,
  109 + # not this module's concern (see module docstring).
  110 + _stop_process(live_pid, timeout=stop_timeout)
  111 + # The health-check-mode candidate (bound to candidate_port/
  112 + # candidate_db_path) is retired too -- its job was verification,
  113 + # not serving traffic.
  114 + _stop_process(candidate_pid, timeout=stop_timeout)
  115 +
  116 + new_live_process = deploy_stage._launch_candidate(
  117 + candidate_instance_dir, live_port, prod_db_path, python_executable
  118 + )
  119 + if not _wait_for_live_http(live_port, bind_timeout, sleep=sleep, now=now):
  120 + raise CutoverFailedError(
  121 + "candidate for deployment {} never bound live_port {} within {}s".format(
  122 + deployment_id, live_port, bind_timeout
  123 + )
  124 + )
  125 + except CutoverFailedError:
  126 + conn.execute(
  127 + "UPDATE deployment SET status = 'blocked_pending_human' WHERE deployment_id = ?", (deployment_id,)
  128 + )
  129 + conn.commit()
  130 + raise
  131 + except Exception as e:
  132 + conn.execute(
  133 + "UPDATE deployment SET status = 'blocked_pending_human' WHERE deployment_id = ?", (deployment_id,)
  134 + )
  135 + conn.commit()
  136 + raise CutoverFailedError(
  137 + "cutover failed for deployment {}: {}".format(deployment_id, e)
  138 + ) from e
  139 +
  140 + conn.execute(
  141 + "UPDATE deployment SET status = 'live', cutover_at = CURRENT_TIMESTAMP WHERE deployment_id = ?",
  142 + (deployment_id,),
  143 + )
  144 + conn.commit()
  145 + return {"status": "live", "live_pid": new_live_process.pid, "live_port": live_port}
... ...
supervisor/deploy_health_check.py
  View file @002bba4
... ... @@ -29,9 +29,15 @@ _SIGNAL_TYPES = (&quot;process_alive&quot;, &quot;http_health&quot;, &quot;canary_task&quot;)
29 29  
30 30  
31 31 def check_process_alive(pid, timeout=HEALTH_CHECK_STARTUP_TIMEOUT, sleep=time.sleep, now=time.time):
  32 + """Uses `reap_if_exited` (not plain `is_alive`) so a candidate that
  33 + crashed immediately after being spawned by THIS process reads as dead
  34 + rather than a false "alive" zombie reading -- see
  35 + `deploy_supervisor.is_alive`'s own docstring for why the distinction
  36 + matters here specifically (this module's caller is always the same
  37 + process that Popen'd the candidate)."""
32 38 deadline = now() + timeout
33 39 while now() < deadline:
34   - if deploy_supervisor.is_alive(pid):
  40 + if not deploy_supervisor.reap_if_exited(pid):
35 41 return {"signal_type": "process_alive", "status": "ok", "detail": None}
36 42 sleep(0.5)
37 43 return {
... ...
supervisor/deploy_supervisor.py
  View file @002bba4
... ... @@ -55,6 +55,18 @@ def _atomic_write_json(path, data):
55 55  
56 56  
57 57 def is_alive(pid):
  58 + """`os.kill(pid, 0)`-based liveness -- NOTE this can't distinguish "truly
  59 + running" from "exited but not yet reaped by its parent" (a zombie's PID
  60 + slot still answers kill(pid, 0) successfully until something calls
  61 + wait() on it). If the calling process spawned `pid` itself via
  62 + `subprocess.Popen` and holds onto it across multiple `is_alive()` calls
  63 + without ever waiting on it, this can report True forever for an
  64 + already-dead process -- found for real (2026-08-02, U-039) via a test
  65 + that actually waited for a stopped process to go away and it never did.
  66 + Callers that spawned `pid` themselves should use `reap_if_exited()`
  67 + instead, which handles this correctly; this function is still right for
  68 + checking a pid this process did NOT spawn (nothing to reap, this is the
  69 + best available check)."""
58 70 if not isinstance(pid, int) or pid <= 0:
59 71 return False
60 72 try:
... ... @@ -64,6 +76,26 @@ def is_alive(pid):
64 76 return True
65 77  
66 78  
  79 +def reap_if_exited(pid):
  80 + """True once `pid` is confirmed gone. If `pid` is a real child of the
  81 + calling process (e.g. spawned via `subprocess.Popen` earlier in this
  82 + same process's lifetime), this actually reaps it via a non-blocking
  83 + `os.waitpid` -- required for `is_alive()` to ever correctly report False
  84 + for it afterward (see that function's own docstring for why). If `pid`
  85 + isn't our child (belongs to a different process/session -- e.g. a
  86 + `live_pid` inherited from a prior Deploy Supervisor run), `os.waitpid`
  87 + raises `ChildProcessError` and this falls back to `is_alive()`'s plain
  88 + liveness check instead, which is the best available answer without
  89 + being the real parent."""
  90 + try:
  91 + reaped_pid, _ = os.waitpid(pid, os.WNOHANG)
  92 + if reaped_pid == pid:
  93 + return True
  94 + except ChildProcessError:
  95 + pass
  96 + return not is_alive(pid)
  97 +
  98 +
67 99 def write_status(status, detail=None):
68 100 _atomic_write_json(status_file_path(), {
69 101 "status": status, "pid": os.getpid(), "updated_at": time.time(), "detail": detail,
... ... @@ -160,6 +192,7 @@ def stop(timeout=10.0):
160 192 deadline = time.time() + timeout
161 193 while time.time() < deadline:
162 194 if not is_running():
  195 + reap_if_exited(pid) # best-effort zombie cleanup, doesn't affect the True/False result
163 196 return True
164 197 time.sleep(0.1)
165 198 return False
... ...
supervisor/test_deploy_cutover.py 0 → 100644
  View file @002bba4
  1 +"""U-039 tests: c_deploy_supervisor_cutover. Real subprocesses throughout
  2 +(old Live, health-check-mode candidate, and the post-cutover restarted
  3 +process) -- this Unit's whole job is process lifecycle around a real port/DB
  4 +swap, so mocking that away would test nothing real.
  5 +
  6 +Run: python3 -m unittest discover -s supervisor -p "test_*.py" -v
  7 +"""
  8 +
  9 +import os
  10 +import shutil
  11 +import subprocess
  12 +import sys
  13 +import tempfile
  14 +import time
  15 +import unittest
  16 +
  17 +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
  18 +
  19 +from supervisor import deploy_cutover, deploy_health_check, deploy_stage, deploy_supervisor
  20 +from src import paths, schema
  21 +
  22 +
  23 +def _free_port():
  24 + return deploy_stage._find_free_port()
  25 +
  26 +
  27 +def _commit_to_test_against():
  28 + """Same rationale as test_deploy_stage.py's own helper -- git worktree
  29 + only sees committed content, and this Unit's own changes may not be
  30 + committed yet at test time."""
  31 + stash_sha = subprocess.run(
  32 + ["git", "stash", "create"], cwd=paths.repo_root(), capture_output=True, text=True, check=True
  33 + ).stdout.strip()
  34 + return stash_sha or deploy_stage._current_commit(paths.repo_root())
  35 +
  36 +
  37 +class DeployCutoverTestBase(unittest.TestCase):
  38 + def setUp(self):
  39 + self.home_tmp = tempfile.mkdtemp(prefix="deploy-cutover-home-")
  40 + self.addCleanup(shutil.rmtree, self.home_tmp, ignore_errors=True)
  41 + self.old_home = os.environ.get("GRAPHLEMON_HOME")
  42 + os.environ["GRAPHLEMON_HOME"] = self.home_tmp
  43 +
  44 + self.prod_db_path = os.path.join(self.home_tmp, "prod.db")
  45 + prod_conn = schema.connect(self.prod_db_path)
  46 + schema.create_schema(prod_conn)
  47 + prod_conn.close()
  48 +
  49 + self.conn = schema.connect(":memory:")
  50 + schema.create_schema(self.conn)
  51 + self.addCleanup(self.conn.close)
  52 +
  53 + self._cleanup_pids = []
  54 + self._cleanup_worktrees = []
  55 +
  56 + def tearDown(self):
  57 + if self.old_home is None:
  58 + os.environ.pop("GRAPHLEMON_HOME", None)
  59 + else:
  60 + os.environ["GRAPHLEMON_HOME"] = self.old_home
  61 + for pid in self._cleanup_pids:
  62 + if deploy_supervisor.is_alive(pid):
  63 + os.kill(pid, 9)
  64 + for wt in self._cleanup_worktrees:
  65 + subprocess.run(["git", "worktree", "remove", "--force", wt], cwd=paths.repo_root(), capture_output=True)
  66 +
  67 + def _stage_and_health_check(self, port=None):
  68 + commit = _commit_to_test_against()
  69 + result = deploy_stage.stage_deployment(
  70 + self.conn, commit, repo_root=paths.repo_root(), prod_db_path=self.prod_db_path,
  71 + port=port or _free_port(),
  72 + )
  73 + self._cleanup_pids.append(result["candidate_pid"])
  74 + self._cleanup_worktrees.append(result["candidate_instance_dir"])
  75 + self._wait_for_port(result["candidate_port"])
  76 + deploy_health_check.run_health_check(
  77 + self.conn, result["deployment_id"], result["candidate_pid"], result["candidate_port"]
  78 + )
  79 + return result
  80 +
  81 + def _wait_for_port(self, port, timeout=10):
  82 + deadline = time.time() + timeout
  83 + while time.time() < deadline:
  84 + if deploy_health_check.check_http_health(port, timeout=1)["status"] == "ok":
  85 + return
  86 + time.sleep(0.2)
  87 + self.fail("candidate on port {} never came up".format(port))
  88 +
  89 +
  90 +class CutoverHappyPathTest(DeployCutoverTestBase):
  91 + def test_first_ever_cutover_no_prior_live_process(self):
  92 + staged = self._stage_and_health_check()
  93 + live_port = _free_port()
  94 +
  95 + result = deploy_cutover.cutover(
  96 + self.conn, staged["deployment_id"], live_pid=None, candidate_pid=staged["candidate_pid"],
  97 + candidate_instance_dir=staged["candidate_instance_dir"], prod_db_path=self.prod_db_path,
  98 + live_port=live_port, sleep=lambda s: time.sleep(min(s, 0.2)),
  99 + )
  100 + self._cleanup_pids.append(result["live_pid"])
  101 +
  102 + self.assertEqual(result["status"], "live")
  103 + self.assertEqual(result["live_port"], live_port)
  104 + self.assertTrue(deploy_supervisor.is_alive(result["live_pid"]))
  105 + self.assertNotEqual(result["live_pid"], staged["candidate_pid"]) # old candidate process was retired
  106 + self.assertFalse(deploy_supervisor.is_alive(staged["candidate_pid"]))
  107 +
  108 + row = self.conn.execute(
  109 + "SELECT status, cutover_at FROM deployment WHERE deployment_id = ?", (staged["deployment_id"],)
  110 + ).fetchone()
  111 + self.assertEqual(row[0], "live")
  112 + self.assertIsNotNone(row[1])
  113 +
  114 + # real HTTP round trip against the NEW live process, on live_port,
  115 + # backed by prod_db_path (not the candidate's one-time DB copy)
  116 + self._wait_for_port(live_port)
  117 + health = deploy_health_check.check_http_health(live_port)
  118 + self.assertEqual(health["status"], "ok")
  119 +
  120 + def test_cutover_stops_the_old_live_process(self):
  121 + staged = self._stage_and_health_check()
  122 +
  123 + old_live_port = _free_port()
  124 + old_live = deploy_stage._launch_candidate(paths.repo_root(), old_live_port, self.prod_db_path)
  125 + self._cleanup_pids.append(old_live.pid)
  126 + self._wait_for_port(old_live_port)
  127 + self.assertTrue(deploy_supervisor.is_alive(old_live.pid))
  128 +
  129 + new_live_port = _free_port()
  130 + result = deploy_cutover.cutover(
  131 + self.conn, staged["deployment_id"], live_pid=old_live.pid, candidate_pid=staged["candidate_pid"],
  132 + candidate_instance_dir=staged["candidate_instance_dir"], prod_db_path=self.prod_db_path,
  133 + live_port=new_live_port,
  134 + )
  135 + self._cleanup_pids.append(result["live_pid"])
  136 +
  137 + deadline = time.time() + 5
  138 + while time.time() < deadline and deploy_supervisor.is_alive(old_live.pid):
  139 + time.sleep(0.1)
  140 + self.assertFalse(deploy_supervisor.is_alive(old_live.pid))
  141 +
  142 +
  143 +class CutoverFailedTest(DeployCutoverTestBase):
  144 + def test_bind_timeout_marks_blocked_pending_human_and_raises(self):
  145 + staged = self._stage_and_health_check()
  146 +
  147 + with self.assertRaises(deploy_cutover.CutoverFailedError):
  148 + deploy_cutover.cutover(
  149 + self.conn, staged["deployment_id"], live_pid=None, candidate_pid=staged["candidate_pid"],
  150 + candidate_instance_dir="/nonexistent/checkout/dir/that/cannot/run/anything",
  151 + prod_db_path=self.prod_db_path, live_port=_free_port(),
  152 + bind_timeout=1, sleep=lambda s: time.sleep(min(s, 0.1)),
  153 + )
  154 +
  155 + row = self.conn.execute(
  156 + "SELECT status FROM deployment WHERE deployment_id = ?", (staged["deployment_id"],)
  157 + ).fetchone()
  158 + self.assertEqual(row[0], "blocked_pending_human")
  159 +
  160 +
  161 +if __name__ == "__main__":
  162 + unittest.main()
... ...