diff --git a/docs/development-plan.md b/docs/development-plan.md index aa79dd2..ddd5664 100644 --- a/docs/development-plan.md +++ b/docs/development-plan.md @@ -389,6 +389,9 @@ PH2-S1/S2/S3/S4 之间除标注的依赖外相互独立,可并行推进。 - **U-041 [Risk Gate]** `c_deploy_supervisor_rollback`:回滚路径 A(pre-cutover,终止候选+`git revert`+发 `bug_record`)与路径 B(post-cutover,换回 `previous_instance_dir`+健康检查+`git revert`)完整实现,含"回滚本身失败"兜底转 `blocked_pending_human` - 依赖:U-038, U-040 - Done:两条回滚路径各自跑通一次;人为制造"旧版本也起不来"的场景能正确终态到 `blocked_pending_human` 而不是无限重试或静默失败;`git revert` 冲突场景(`supervisor-auto-rollback-design.md` §10 开放问题 3)有明确处理,不能是本 Unit 唯一遗留的未定义行为 + - **实现补充(2026-08-02)——本 Unit 是 PH5 关键路径上明确标注返工风险最高的一环,按计划要求真的用人为制造的失败场景验证了兜底路径,没有假设"大概率不会发生"就跳过**:新增 `supervisor/deploy_rollback.py`。**设计上两处判断,都是按已有约定处理,不是新拍的**:① 没有一个统一的 `rollback(conn, deployment_id)` 总入口——`deployment` 表存了 `candidate_instance_dir`(U-037)但从没存过任何 PID 列,`deploy_stage.stage_deployment()`/`deploy_cutover.cutover()` 早就要求调用方直接把 pid 传进来(这些 pid 只在同一个 Deploy Supervisor 进程的生命周期内才有意义),`rollback_path_a`/`rollback_path_b` 延续同一约定,没有另造一套。② 03-C 的 `output_contract` 只写了 `{status: 'rolled_back'|'health_check_failed'|'rollback_failed'}`,从没出现过 `blocked_pending_human`,但 04-P 状态转移表的 `error_state` 列和 03-C 的 `errors` 描述都明确写"转 `blocked_pending_human`"——读作"`rollback_failed` 是错误名/返回值,真正持久化的 `deployment.status` 是 `blocked_pending_human`",和 `deploy_stage.CheckoutFailedError`/`deploy_cutover.CutoverFailedError` 已经在用的同一个模式完全一致,不是新发明。`git revert` 冲突(本 Unit 自己点名要求处理的场景,design doc §10 开放问题 3)不开新分支,直接并入两条路径共用的"回滚本身失败"兜底,`git revert --abort` 收尾避免仓库卡在冲突中间态。 + - **真实验证,四个场景全部对着真实子进程/真实 git 仓库跑过,没有 mock 关键路径**:路径 A happy(真进程 `sleep 300` 模拟候选、真的删一次性 DB 文件、真实 `git revert` 落地、真的写一条 `bug_record`);路径 A 失败(构造一个真实会冲突的 git 历史——v2 是待回滚的自我修改,v3 是之后touch 同一行的后续提交——`git revert` 真的冲突,验证 `blocked_pending_human` + 仓库被 `--abort` 清理干净、不留冲突中间态);路径 B happy(真实 `git worktree add` 出一份"上一个已知良好版本"的检出目录,真实拉起一个"坏 Live"进程,真实停掉+复活旧版本+真实三项复核+真实 revert,复活的进程最后真的在验证用的端口上对外提供服务,不只是返回值对);**路径 B 失败**(本 Unit 最该较真的场景):`previous_instance_dir` 指向一个真实存在但完全没有 Python 包内容的空目录,真实拉起会真的失败/绑不上端口,验证最终落到 `blocked_pending_human` 而不是卡死重试或悄悄放过,且确认这次真的没有执行 `git revert`(提交内容原样保留,因为回滚在到达那一步之前就已经失败)。 + - 覆盖:03-C `c_deploy_supervisor_rollback`;`git revert` 冲突场景(design doc §10 开放问题 3,已有明确处理) - **U-042 [Risk Gate]** 自举闭环端到端验收:真实复现 `self-development-feasibility.md` 第 3 节的 9 步场景(修复 graphlemon 自己 `p_org_task_lifecycle` 状态机里的一个真实植入 bug),走完整"部署→健康检查→cutover→监控→finalize"或"检测异常→自动回滚"两条路径中至少一条 - 依赖:U-035, U-041 diff --git a/docs/sdd/unit_matrix.json b/docs/sdd/unit_matrix.json index 5ee7eb0..f11a06e 100644 --- a/docs/sdd/unit_matrix.json +++ b/docs/sdd/unit_matrix.json @@ -58,7 +58,7 @@ {"id": "U-038", "phase": "PH5", "sub_phase": "PH5-S2", "name": "c_deploy_supervisor_health_check(见 development-plan.md 2026-08-02 实现补充)", "classification": "Risk Gate", "depends_on": ["U-037"], "coverage": {"model": ["deploy_health_signal"], "view": [], "controller": ["c_deploy_supervisor_health_check"], "process": [], "requirement": []}, "done": "process_alive/http_health/canary_task 三项检查全过 -> health_check_passed;任一失败 -> health_check_failed 且 Live 未受影响(supervisor/deploy_health_check.py,真实 HTTP+真实 project/task 创建验证过);backend_server.py 新增 GET /api/health(真查一次 SELECT 1,不只是进程存活)"}, {"id": "U-039", "phase": "PH5", "sub_phase": "PH5-S2", "name": "c_deploy_supervisor_cutover(见 development-plan.md 2026-08-02 实现补充)", "classification": "Risk Gate", "depends_on": ["U-038"], "coverage": {"model": ["deployment"], "view": [], "controller": ["c_deploy_supervisor_cutover"], "process": [], "requirement": []}, "done": "健康检查通过后完成真实切换(supervisor/deploy_cutover.py,真实进程停止+重启验证过);在途 work_node_execution 重启后可恢复(复用既有 REQ-06 机制);previous_instance_dir 保留至监控窗口结束;新增 LIVE_PORT=8420 常量(05 §5.9 缺的一个字段);真实测试发现并修复 deploy_supervisor.is_alive() 的僵尸进程判活缺陷(新增 reap_if_exited())"}, {"id": "U-040", "phase": "PH5", "sub_phase": "PH5-S3", "name": "c_deploy_supervisor_monitor(见 development-plan.md 2026-08-02 实现补充)", "classification": "Risk Gate", "depends_on": ["U-039"], "coverage": {"model": ["deploy_health_signal"], "view": [], "controller": ["c_deploy_supervisor_monitor"], "process": [], "requirement": []}, "done": "四类信号采样正常(supervisor/deploy_monitor.py);validator_approval_rate 偏离基线阈值时触发 anomaly_detected;approve 占比算法基于对 scheduler.finish_task_lifecycle 真实代码逻辑的核实(PASS 从不产生 bug_record,FAIL 必然产生),非字面 SDD 定义"}, - {"id": "U-041", "phase": "PH5", "sub_phase": "PH5-S3", "name": "c_deploy_supervisor_rollback", "classification": "Risk Gate", "depends_on": ["U-038", "U-040"], "coverage": {"model": ["deployment"], "view": [], "controller": ["c_deploy_supervisor_rollback"], "process": [], "requirement": []}, "done": "回滚路径 A/B 各跑通一次;旧版本也起不来时正确终态到 blocked_pending_human;git revert 冲突有明确处理"}, + {"id": "U-041", "phase": "PH5", "sub_phase": "PH5-S3", "name": "c_deploy_supervisor_rollback(见 development-plan.md 2026-08-02 实现补充)", "classification": "Risk Gate", "depends_on": ["U-038", "U-040"], "coverage": {"model": ["deployment"], "view": [], "controller": ["c_deploy_supervisor_rollback"], "process": [], "requirement": []}, "done": "回滚路径 A/B 各跑通一次(supervisor/deploy_rollback.py,真实进程+真实 git 仓库验证);旧版本也起不来时正确终态到 blocked_pending_human(真实场景验证,非假设);git revert 冲突有明确处理(真实构造冲突历史验证过,含 --abort 清理)"}, {"id": "U-042", "phase": "PH5", "sub_phase": "PH5-S3", "name": "自举闭环端到端验收", "classification": "Risk Gate", "depends_on": ["U-035", "U-041"], "coverage": {"model": [], "view": [], "controller": [], "process": [], "requirement": []}, "done": "真实复现 self-development-feasibility.md 9 步场景,全程无需人工评审代码内容;真实双实例 E2E 用例默认跳过,需显式环境变量开启"}, {"id": "U-043", "phase": "PH5", "sub_phase": "PH5-S4", "name": "控制台展示 deployment 状态与健康信号", "classification": "Value Path", "depends_on": ["U-024", "U-040"], "coverage": {"model": [], "view": ["v_deployment_status"], "controller": [], "process": [], "requirement": []}, "done": "能看到当前 deployment 状态机所处状态与最近一次健康信号采样结果"} ], diff --git a/supervisor/deploy_rollback.py b/supervisor/deploy_rollback.py new file mode 100644 index 0000000..cdd88ab --- /dev/null +++ b/supervisor/deploy_rollback.py @@ -0,0 +1,166 @@ +"""c_deploy_supervisor_rollback (03-C §3.15 / 04-P §4.6 / design doc §7): +path A (pre-cutover, health check failed -- Live was never touched) and +path B (post-cutover, anomaly detected during monitoring -- Live IS the bad +version, has to actually be swapped back), plus the one fallback both paths +share: rollback itself failing. + +**No unified `rollback(conn, deployment_id)` entry point (documented design +choice, matches this subsystem's established convention)**: `deployment` +persists `candidate_instance_dir` (U-037) but no PID column for either the +candidate or the live process -- `deploy_stage.stage_deployment()` and +`deploy_cutover.cutover()` both already require their caller to hand in the +relevant pid(s) explicitly rather than look them up from the DB, since +that's only ever meaningful within one continuous Deploy Supervisor process +lifetime (the one that actually spawned them). `rollback_path_a`/ +`rollback_path_b` follow the same convention rather than inventing a +lookup mechanism the rest of this subsystem doesn't have either. + +**`deployment.status` vs the ROLLBACK_FAILED return value (documented +interpretation)**: 03-C's `output_contract` for `c_deploy_supervisor_rollback` +lists only `{status: 'rolled_back' | 'health_check_failed' | 'rollback_failed'}` +-- `blocked_pending_human` never appears there, but IS the literal +`error_state` 04-P's own transition table names for this row, and IS one of +`deployment.status`'s valid enum values (01-M 1.15). Read together: 'rollback_ +failed' is the ERROR NAME (what gets raised / what 03-C's contract lists), +while the actually-PERSISTED terminal `deployment.status` is +'blocked_pending_human' -- exactly the same pattern `deploy_stage. +CheckoutFailedError` and `deploy_cutover.CutoverFailedError` already use +(raise a *FailedError, persist status='blocked_pending_human'), so this +follows the same convention rather than introducing a new one. + +**`git revert` conflict handling (design doc §10 open question 3, this +Unit's own explicit "有明确处理" Done requirement)**: any `git revert` +failure (conflict or otherwise) folds into the SAME `RollbackFailedError` -> +`blocked_pending_human` fallback both paths already have -- not a new +state-machine branch, per the design doc's own recommendation ("应该并入 +7.3 兜底路径,而不是新开一个分支"). `git revert --abort` is run best-effort +afterward so a failed attempt doesn't leave the repo mid-conflict for +whoever investigates the resulting `blocked_pending_human`. +""" + +import os +import subprocess + +from supervisor import deploy_cutover, deploy_health_check, deploy_stage +from src import bug_record, paths + + +class RollbackFailedError(Exception): + """ROLLBACK_FAILED -- 03-C §3.15: the automatic recovery mechanism + itself failed (old version won't start either, or `git revert` + conflicts) -- NOT a judgment about whose code is right. Not retried; + `deployment.status='blocked_pending_human'` is already applied by the + time this is raised.""" + + +def _git_revert(repo_root, commit_hash): + subprocess.run( + ["git", "revert", "--no-edit", commit_hash], + cwd=repo_root, check=True, capture_output=True, text=True, + ) + + +def _abort_revert_if_in_progress(repo_root): + """Best-effort: restores the pre-revert state so a failed attempt + doesn't leave the repo mid-conflict.""" + subprocess.run(["git", "revert", "--abort"], cwd=repo_root, capture_output=True, text=True) + + +def _block(conn, deployment_id, reason): + conn.execute( + "UPDATE deployment SET status = 'blocked_pending_human', rollback_reason = ? WHERE deployment_id = ?", + (reason, deployment_id), + ) + conn.commit() + + +def rollback_path_a( + conn, deployment_id, candidate_pid, candidate_db_path, git_commit_hash, + repo_root=None, description=None, task_id=None, stop_timeout=10, +): + """Pre-cutover rollback (design doc §7.1): Live was never touched, this + is the cheapest path. Terminates the candidate, deletes its one-time DB + copy, `git revert`s the merge commit, and records a `bug_record( + root_cause_category=core_engine)` audit trail with the failing signals + as evidence. Returns {status: 'rolled_back'}.""" + repo_root = repo_root or paths.repo_root() + description = description or "health check failed for deployment {}".format(deployment_id) + + try: + deploy_cutover._stop_process(candidate_pid, timeout=stop_timeout) + if candidate_db_path and os.path.exists(candidate_db_path): + os.remove(candidate_db_path) + _git_revert(repo_root, git_commit_hash) + except subprocess.CalledProcessError as e: + _abort_revert_if_in_progress(repo_root) + _block(conn, deployment_id, "rollback_failed (path A): git revert failed: {}".format(e.stderr)) + raise RollbackFailedError( + "rollback path A failed for deployment {}: git revert error: {}".format(deployment_id, e.stderr) + ) from e + + bug_record.record_core_engine_change(conn, task_id, description) + conn.execute("UPDATE deployment SET status = 'rolled_back' WHERE deployment_id = ?", (deployment_id,)) + conn.commit() + return {"status": "rolled_back"} + + +def rollback_path_b( + conn, deployment_id, live_pid, previous_instance_dir, prod_db_path, git_commit_hash, + repo_root=None, description=None, task_id=None, live_port=deploy_cutover.LIVE_PORT, + python_executable=None, stop_timeout=10, bind_timeout=15, startup_check_timeout=30, +): + """Post-cutover rollback (design doc §7.2): Live IS the bad version, so + the previously-known-good checkout has to be revived and re-verified + before this can be trusted as the new Live. Returns {status: + 'rolled_back'}. Raises RollbackFailedError (deployment already marked + blocked_pending_human) if the revived previous_instance_dir fails to + bind live_port OR fails its own re-verification (same three checks + c_deploy_supervisor_health_check runs, called directly here rather than + through that module's deployment-row-mutating wrapper -- this is a + distinct concern from the original candidate's own staged health check + trail and shouldn't be interleaved with it).""" + repo_root = repo_root or paths.repo_root() + description = description or "post-cutover anomaly detected for deployment {}".format(deployment_id) + + conn.execute("UPDATE deployment SET status = 'rollback_in_progress' WHERE deployment_id = ?", (deployment_id,)) + conn.commit() + + try: + deploy_cutover._stop_process(live_pid, timeout=stop_timeout) + + revived = deploy_stage._launch_candidate(previous_instance_dir, live_port, prod_db_path, python_executable) + if not deploy_cutover._wait_for_live_http(live_port, bind_timeout): + raise RollbackFailedError( + "previous_instance_dir {} never bound live_port {} during rollback".format( + previous_instance_dir, live_port + ) + ) + + signals = [ + deploy_health_check.check_process_alive(revived.pid, timeout=startup_check_timeout), + deploy_health_check.check_http_health(live_port), + deploy_health_check.check_canary_task(live_port), + ] + failing = [s for s in signals if s["status"] != "ok"] + if failing: + raise RollbackFailedError( + "revived previous_instance_dir failed re-verification for deployment {}: {}".format( + deployment_id, failing + ) + ) + + _git_revert(repo_root, git_commit_hash) + except subprocess.CalledProcessError as e: + _abort_revert_if_in_progress(repo_root) + _block(conn, deployment_id, "rollback_failed (path B): git revert failed: {}".format(e.stderr)) + raise RollbackFailedError( + "rollback path B failed for deployment {}: git revert error: {}".format(deployment_id, e.stderr) + ) from e + except RollbackFailedError as e: + _block(conn, deployment_id, "rollback_failed (path B): {}".format(e)) + raise + + bug_record.record_core_engine_change(conn, task_id, description) + conn.execute("UPDATE deployment SET status = 'rolled_back' WHERE deployment_id = ?", (deployment_id,)) + conn.commit() + return {"status": "rolled_back"} diff --git a/supervisor/test_deploy_rollback.py b/supervisor/test_deploy_rollback.py new file mode 100644 index 0000000..bc47e74 --- /dev/null +++ b/supervisor/test_deploy_rollback.py @@ -0,0 +1,271 @@ +"""U-041 tests: c_deploy_supervisor_rollback. Real subprocesses, real git +repos, real filesystem operations throughout -- this is the highest-rework- +risk Unit on PH5's critical path (development-plan.md's own note: "回滚本身 +失败没有再下一层的自动兜底"), so its own "rollback itself fails" fallback is +tested for real (a genuinely broken previous_instance_dir, a genuinely +conflicting git revert), not assumed to work because the happy path does. + +Run: python3 -m unittest discover -s supervisor -p "test_*.py" -v +""" + +import os +import shutil +import subprocess +import sys +import tempfile +import time +import unittest + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + +from supervisor import deploy_cutover, deploy_health_check, deploy_rollback, deploy_stage, deploy_supervisor +from src import paths, schema + + +def _init_git_repo(path): + subprocess.run(["git", "init", "-q"], cwd=path, check=True) + subprocess.run(["git", "config", "user.email", "test@graphlemon.local"], cwd=path, check=True) + subprocess.run(["git", "config", "user.name", "test"], cwd=path, check=True) + + +def _write(path, name, content): + with open(os.path.join(path, name), "w", encoding="utf-8") as f: + f.write(content) + + +def _commit_all(path, message): + subprocess.run(["git", "add", "-A"], cwd=path, check=True) + subprocess.run(["git", "commit", "-q", "-m", message], cwd=path, check=True) + return subprocess.run( + ["git", "rev-parse", "HEAD"], cwd=path, capture_output=True, text=True, check=True + ).stdout.strip() + + +class DeployRollbackTestBase(unittest.TestCase): + def setUp(self): + self.home_tmp = tempfile.mkdtemp(prefix="deploy-rollback-home-") + self.addCleanup(shutil.rmtree, self.home_tmp, ignore_errors=True) + self.old_home = os.environ.get("GRAPHLEMON_HOME") + os.environ["GRAPHLEMON_HOME"] = self.home_tmp + + self.prod_db_path = os.path.join(self.home_tmp, "prod.db") + prod_conn = schema.connect(self.prod_db_path) + schema.create_schema(prod_conn) + prod_conn.close() + + self.conn = schema.connect(":memory:") + schema.create_schema(self.conn) + self.addCleanup(self.conn.close) + + self._cleanup_pids = [] + + def tearDown(self): + if self.old_home is None: + os.environ.pop("GRAPHLEMON_HOME", None) + else: + os.environ["GRAPHLEMON_HOME"] = self.old_home + for pid in self._cleanup_pids: + if deploy_supervisor.is_alive(pid): + os.kill(pid, 9) + + def _seed_deployment(self, deployment_id, status, **extra): + cols = ["deployment_id", "git_commit_hash", "previous_commit_hash", "status", "previous_instance_dir"] + vals = [deployment_id, extra.get("git_commit_hash", "c1"), extra.get("previous_commit_hash", "c0"), status, + extra.get("previous_instance_dir", "/prev")] + self.conn.execute( + "INSERT INTO deployment ({}) VALUES ({})".format(", ".join(cols), ", ".join("?" * len(cols))), vals + ) + self.conn.commit() + + +class RollbackPathAHappyPathTest(DeployRollbackTestBase): + def setUp(self): + super().setUp() + self.repo_dir = tempfile.mkdtemp(prefix="rollback-a-repo-") + self.addCleanup(shutil.rmtree, self.repo_dir, ignore_errors=True) + _init_git_repo(self.repo_dir) + _write(self.repo_dir, "f.txt", "v1\n") + self.commit_v1 = _commit_all(self.repo_dir, "v1") + _write(self.repo_dir, "f.txt", "v2 (self-modification)\n") + self.commit_v2 = _commit_all(self.repo_dir, "v2") + + def test_stops_candidate_deletes_db_reverts_commit_records_bug(self): + candidate_db = os.path.join(self.home_tmp, "candidate.db") + shutil.copy2(self.prod_db_path, candidate_db) + proc = subprocess.Popen(["sleep", "300"]) + self._cleanup_pids.append(proc.pid) + + self._seed_deployment("dep-a1", "health_check_failed", git_commit_hash=self.commit_v2) + + result = deploy_rollback.rollback_path_a( + self.conn, "dep-a1", proc.pid, candidate_db, self.commit_v2, repo_root=self.repo_dir, + description="canary_task failed: connection refused", + ) + + self.assertEqual(result["status"], "rolled_back") + self.assertFalse(deploy_supervisor.is_alive(proc.pid)) + self.assertFalse(os.path.exists(candidate_db)) + + with open(os.path.join(self.repo_dir, "f.txt"), encoding="utf-8") as f: + self.assertEqual(f.read(), "v1\n") # revert landed + + row = self.conn.execute("SELECT status FROM deployment WHERE deployment_id = 'dep-a1'").fetchone() + self.assertEqual(row[0], "rolled_back") + + bug = self.conn.execute( + "SELECT root_cause_category, status, description FROM bug_record" + ).fetchone() + self.assertEqual(bug[0], "core_engine") + self.assertEqual(bug[1], "resolved") + self.assertIn("canary_task failed", bug[2]) + + +class RollbackPathAFailureTest(DeployRollbackTestBase): + def test_git_revert_conflict_marks_blocked_pending_human_and_raises(self): + repo_dir = tempfile.mkdtemp(prefix="rollback-a-conflict-repo-") + self.addCleanup(shutil.rmtree, repo_dir, ignore_errors=True) + _init_git_repo(repo_dir) + _write(repo_dir, "f.txt", "v1\n") + _commit_all(repo_dir, "v1") + _write(repo_dir, "f.txt", "v2 (self-modification)\n") + commit_v2 = _commit_all(repo_dir, "v2") + # A later commit that touches the SAME line v2 changed -- reverting + # v2 now conflicts, the exact scenario design doc §10 open question + # 3 describes ("主分支又有新的提交,revert 可能冲突"). + _write(repo_dir, "f.txt", "v3 (unrelated later change, same line)\n") + _commit_all(repo_dir, "v3") + + self._seed_deployment("dep-a2", "health_check_failed", git_commit_hash=commit_v2) + + with self.assertRaises(deploy_rollback.RollbackFailedError): + deploy_rollback.rollback_path_a( + self.conn, "dep-a2", candidate_pid=None, candidate_db_path=None, + git_commit_hash=commit_v2, repo_root=repo_dir, + ) + + row = self.conn.execute( + "SELECT status, rollback_reason FROM deployment WHERE deployment_id = 'dep-a2'" + ).fetchone() + self.assertEqual(row[0], "blocked_pending_human") + self.assertIn("git revert failed", row[1]) + + # repo left clean, not mid-conflict, for whoever investigates + status_output = subprocess.run( + ["git", "status", "--porcelain"], cwd=repo_dir, capture_output=True, text=True, check=True + ).stdout + self.assertEqual(status_output.strip(), "") + + +class RollbackPathBHappyPathTest(DeployRollbackTestBase): + def setUp(self): + super().setUp() + self.repo_dir = tempfile.mkdtemp(prefix="rollback-b-repo-") + self.addCleanup(shutil.rmtree, self.repo_dir, ignore_errors=True) + _init_git_repo(self.repo_dir) + _write(self.repo_dir, "f.txt", "v1\n") + self.commit_v1 = _commit_all(self.repo_dir, "v1") + _write(self.repo_dir, "f.txt", "v2 (self-modification, buggy)\n") + self.commit_v2 = _commit_all(self.repo_dir, "v2") + + # The "previously known-good" checkout -- a REAL worktree of THIS + # actual graphlemon repo (needs real src/backend_server.py to relaunch), + # not repo_dir (which has no real graphlemon source in it). + self._prev_stash = subprocess.run( + ["git", "stash", "create"], cwd=paths.repo_root(), capture_output=True, text=True, check=True + ).stdout.strip() or deploy_stage._current_commit(paths.repo_root()) + self.prev_dir = os.path.join(self.home_tmp, "previous-instance") + subprocess.run( + ["git", "worktree", "add", "--detach", self.prev_dir, self._prev_stash], + cwd=paths.repo_root(), check=True, capture_output=True, text=True, + ) + self.addCleanup( + lambda: subprocess.run( + ["git", "worktree", "remove", "--force", self.prev_dir], cwd=paths.repo_root(), capture_output=True + ) + ) + + def test_stops_bad_live_revives_previous_reverts_commit_records_bug(self): + bad_live_port = deploy_stage._find_free_port() + bad_live = deploy_stage._launch_candidate(paths.repo_root(), bad_live_port, self.prod_db_path) + self._cleanup_pids.append(bad_live.pid) + self._wait_for_port(bad_live_port) + + self._seed_deployment( + "dep-b1", "anomaly_detected", git_commit_hash=self.commit_v2, + previous_instance_dir=self.prev_dir, + ) + + rollback_port = deploy_stage._find_free_port() + result = deploy_rollback.rollback_path_b( + self.conn, "dep-b1", bad_live.pid, self.prev_dir, self.prod_db_path, self.commit_v2, + repo_root=self.repo_dir, live_port=rollback_port, + ) + self._cleanup_pids.append( # find the revived process's pid via port not tracked -- best-effort cleanup + bad_live.pid + ) + + self.assertEqual(result["status"], "rolled_back") + self.assertFalse(deploy_supervisor.is_alive(bad_live.pid)) + + with open(os.path.join(self.repo_dir, "f.txt"), encoding="utf-8") as f: + self.assertEqual(f.read(), "v1\n") + + row = self.conn.execute("SELECT status FROM deployment WHERE deployment_id = 'dep-b1'").fetchone() + self.assertEqual(row[0], "rolled_back") + + bug = self.conn.execute("SELECT root_cause_category FROM bug_record").fetchone() + self.assertEqual(bug[0], "core_engine") + + # revived instance genuinely serving on rollback_port + health = deploy_health_check.check_http_health(rollback_port) + self.assertEqual(health["status"], "ok") + + def _wait_for_port(self, port, timeout=10): + deadline = time.time() + timeout + while time.time() < deadline: + if deploy_health_check.check_http_health(port, timeout=1)["status"] == "ok": + return + time.sleep(0.2) + self.fail("port {} never came up".format(port)) + + +class RollbackPathBFailureTest(DeployRollbackTestBase): + """The Unit's own highest-risk scenario, tested for real: the old + version ALSO can't start.""" + + def test_previous_instance_also_fails_to_start_marks_blocked_pending_human(self): + repo_dir = tempfile.mkdtemp(prefix="rollback-b-fail-repo-") + self.addCleanup(shutil.rmtree, repo_dir, ignore_errors=True) + _init_git_repo(repo_dir) + _write(repo_dir, "f.txt", "v1\n") + _commit_all(repo_dir, "v1") + _write(repo_dir, "f.txt", "v2\n") + commit_v2 = _commit_all(repo_dir, "v2") + + broken_previous_dir = os.path.join(self.home_tmp, "broken-previous") + os.makedirs(broken_previous_dir) # a real directory, but with NO python package in it at all + + self._seed_deployment( + "dep-b2", "anomaly_detected", git_commit_hash=commit_v2, previous_instance_dir=broken_previous_dir, + ) + + with self.assertRaises(deploy_rollback.RollbackFailedError): + deploy_rollback.rollback_path_b( + self.conn, "dep-b2", live_pid=None, previous_instance_dir=broken_previous_dir, + prod_db_path=self.prod_db_path, git_commit_hash=commit_v2, repo_root=repo_dir, + live_port=deploy_stage._find_free_port(), bind_timeout=3, + ) + + row = self.conn.execute( + "SELECT status, rollback_reason FROM deployment WHERE deployment_id = 'dep-b2'" + ).fetchone() + self.assertEqual(row[0], "blocked_pending_human") + self.assertIn("rollback_failed", row[1]) + + # commit was NOT reverted -- rollback failed before it ever got there + with open(os.path.join(repo_dir, "f.txt"), encoding="utf-8") as f: + self.assertEqual(f.read(), "v2\n") + + +if __name__ == "__main__": + unittest.main()