U-041: c_deploy_supervisor_rollback (paths A/B + the rollback-fails fallback)

Adds supervisor/deploy_rollback.py: rollback_path_a (pre-cutover, health check failed -- terminate candidate, delete its one-time DB copy, git revert, record a bug_record(core_engine) audit trail) and rollback_path_b (post-cutover, anomaly detected -- stop the bad Live, revive previous_instance_dir, re-run the same three health-check signals directly against it, git revert only once re-verification passes). Design choices follow established conventions rather than inventing new ones: no unified rollback() entry point (deployment has no PID columns, same reason deploy_stage/deploy_cutover already require callers to pass pids explicitly); ROLLBACK_FAILED is the raised error name while the persisted deployment.status is 'blocked_pending_human' (same pattern CheckoutFailedError/CutoverFailedError already use). git revert conflicts fold into the same rollback-itself-failed fallback both paths share, per design doc §10 open question 3's own recommendation, with `git revert --abort` run afterward so a failed attempt doesn't leave the repo mid-conflict. This is PH5's highest-rework-risk Unit per development-plan.md's own flag, so its own "rollback itself fails" fallback is tested for real, not assumed: a genuinely broken previous_instance_dir (a real empty directory with no Python package in it) that really fails to bind its port, and a genuinely conflicting git history (a later commit touching the same line the self-modification changed) that really fails `git revert` -- both correctly land on blocked_pending_human without retrying forever or silently succeeding, and the conflict case leaves the repo clean afterward. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

U-041: c_deploy_supervisor_rollback (paths A/B + the rollback-fails fallback)
Adds supervisor/deploy_rollback.py: rollback_path_a (pre-cutover, health check failed -- terminate candidate, delete its one-time DB copy, git revert, record a bug_record(core_engine) audit trail) and rollback_path_b (post-cutover, anomaly detected -- stop the bad Live, revive previous_instance_dir, re-run the same three health-check signals directly against it, git revert only once re-verification passes). Design choices follow established conventions rather than inventing new ones: no unified rollback() entry point (deployment has no PID columns, same reason deploy_stage/deploy_cutover already require callers to pass pids explicitly); ROLLBACK_FAILED is the raised error name while the persisted deployment.status is 'blocked_pending_human' (same pattern CheckoutFailedError/CutoverFailedError already use). git revert conflicts fold into the same rollback-itself-failed fallback both paths share, per design doc §10 open question 3's own recommendation, with `git revert --abort` run afterward so a failed attempt doesn't leave the repo mid-conflict. This is PH5's highest-rework-risk Unit per development-plan.md's own flag, so its own "rollback itself fails" fallback is tested for real, not assumed: a genuinely broken previous_instance_dir (a real empty directory with no Python package in it) that really fails to bind its port, and a genuinely conflicting git history (a later commit touching the same line the self-modification changed) that really fails `git revert` -- both correctly land on blocked_pending_human without retrying forever or silently succeeding, and the conflict case leaves the repo clean afterward. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
bboonnee
1 parent 98f0d5cd
Showing 4 changed files with 441 additions and 1 deletions
docs/development-plan.md
docs/sdd/unit_matrix.json
supervisor/deploy_rollback.py
supervisor/test_deploy_rollback.py
@@ -389,6 +389,9 @@ PH2-S1/S2/S3/S4 之间除标注的依赖外相互独立，可并行推进。
 - **U-041 [Risk Gate]** `c_deploy_supervisor_rollback`：回滚路径 A（pre-cutover，终止候选+`git revert`+发 `bug_record`）与路径 B（post-cutover，换回 `previous_instance_dir`+健康检查+`git revert`）完整实现，含"回滚本身失败"兜底转 `blocked_pending_human`
   - 依赖：U-038, U-040
   - Done：两条回滚路径各自跑通一次；人为制造"旧版本也起不来"的场景能正确终态到 `blocked_pending_human` 而不是无限重试或静默失败；`git revert` 冲突场景（`supervisor-auto-rollback-design.md` §10 开放问题 3）有明确处理，不能是本 Unit 唯一遗留的未定义行为
+  - **实现补充（2026-08-02）——本 Unit 是 PH5 关键路径上明确标注返工风险最高的一环，按计划要求真的用人为制造的失败场景验证了兜底路径，没有假设"大概率不会发生"就跳过**：新增 `supervisor/deploy_rollback.py`。**设计上两处判断，都是按已有约定处理，不是新拍的**：① 没有一个统一的 `rollback(conn, deployment_id)` 总入口——`deployment` 表存了 `candidate_instance_dir`（U-037）但从没存过任何 PID 列，`deploy_stage.stage_deployment()`/`deploy_cutover.cutover()` 早就要求调用方直接把 pid 传进来（这些 pid 只在同一个 Deploy Supervisor 进程的生命周期内才有意义），`rollback_path_a`/`rollback_path_b` 延续同一约定，没有另造一套。② 03-C 的 `output_contract` 只写了 `{status: 'rolled_back'|'health_check_failed'|'rollback_failed'}`，从没出现过 `blocked_pending_human`，但 04-P 状态转移表的 `error_state` 列和 03-C 的 `errors` 描述都明确写"转 `blocked_pending_human`"——读作"`rollback_failed` 是错误名/返回值，真正持久化的 `deployment.status` 是 `blocked_pending_human`"，和 `deploy_stage.CheckoutFailedError`/`deploy_cutover.CutoverFailedError` 已经在用的同一个模式完全一致，不是新发明。`git revert` 冲突（本 Unit 自己点名要求处理的场景，design doc §10 开放问题 3）不开新分支，直接并入两条路径共用的"回滚本身失败"兜底，`git revert --abort` 收尾避免仓库卡在冲突中间态。
+    - **真实验证，四个场景全部对着真实子进程/真实 git 仓库跑过，没有 mock 关键路径**：路径 A happy（真进程 `sleep 300` 模拟候选、真的删一次性 DB 文件、真实 `git revert` 落地、真的写一条 `bug_record`）；路径 A 失败（构造一个真实会冲突的 git 历史——v2 是待回滚的自我修改，v3 是之后touch 同一行的后续提交——`git revert` 真的冲突，验证 `blocked_pending_human` + 仓库被 `--abort` 清理干净、不留冲突中间态）；路径 B happy（真实 `git worktree add` 出一份"上一个已知良好版本"的检出目录，真实拉起一个"坏 Live"进程，真实停掉+复活旧版本+真实三项复核+真实 revert，复活的进程最后真的在验证用的端口上对外提供服务，不只是返回值对）；**路径 B 失败**（本 Unit 最该较真的场景）：`previous_instance_dir` 指向一个真实存在但完全没有 Python 包内容的空目录，真实拉起会真的失败/绑不上端口，验证最终落到 `blocked_pending_human` 而不是卡死重试或悄悄放过，且确认这次真的没有执行 `git revert`（提交内容原样保留，因为回滚在到达那一步之前就已经失败）。
+  - 覆盖：03-C `c_deploy_supervisor_rollback`；`git revert` 冲突场景（design doc §10 开放问题 3，已有明确处理）
  
 - **U-042 [Risk Gate]** 自举闭环端到端验收：真实复现 `self-development-feasibility.md` 第 3 节的 9 步场景（修复 graphlemon 自己 `p_org_task_lifecycle` 状态机里的一个真实植入 bug），走完整"部署→健康检查→cutover→监控→finalize"或"检测异常→自动回滚"两条路径中至少一条
   - 依赖：U-035, U-041
@@ -58,7 +58,7 @@
     {"id": "U-038", "phase": "PH5", "sub_phase": "PH5-S2", "name": "c_deploy_supervisor_health_check（见 development-plan.md 2026-08-02 实现补充）", "classification": "Risk Gate", "depends_on": ["U-037"], "coverage": {"model": ["deploy_health_signal"], "view": [], "controller": ["c_deploy_supervisor_health_check"], "process": [], "requirement": []}, "done": "process_alive/http_health/canary_task 三项检查全过 -> health_check_passed；任一失败 -> health_check_failed 且 Live 未受影响（supervisor/deploy_health_check.py，真实 HTTP+真实 project/task 创建验证过）；backend_server.py 新增 GET /api/health（真查一次 SELECT 1，不只是进程存活）"},
     {"id": "U-039", "phase": "PH5", "sub_phase": "PH5-S2", "name": "c_deploy_supervisor_cutover（见 development-plan.md 2026-08-02 实现补充）", "classification": "Risk Gate", "depends_on": ["U-038"], "coverage": {"model": ["deployment"], "view": [], "controller": ["c_deploy_supervisor_cutover"], "process": [], "requirement": []}, "done": "健康检查通过后完成真实切换（supervisor/deploy_cutover.py，真实进程停止+重启验证过）；在途 work_node_execution 重启后可恢复（复用既有 REQ-06 机制）；previous_instance_dir 保留至监控窗口结束；新增 LIVE_PORT=8420 常量（05 §5.9 缺的一个字段）；真实测试发现并修复 deploy_supervisor.is_alive() 的僵尸进程判活缺陷（新增 reap_if_exited()）"},
     {"id": "U-040", "phase": "PH5", "sub_phase": "PH5-S3", "name": "c_deploy_supervisor_monitor（见 development-plan.md 2026-08-02 实现补充）", "classification": "Risk Gate", "depends_on": ["U-039"], "coverage": {"model": ["deploy_health_signal"], "view": [], "controller": ["c_deploy_supervisor_monitor"], "process": [], "requirement": []}, "done": "四类信号采样正常（supervisor/deploy_monitor.py）；validator_approval_rate 偏离基线阈值时触发 anomaly_detected；approve 占比算法基于对 scheduler.finish_task_lifecycle 真实代码逻辑的核实（PASS 从不产生 bug_record，FAIL 必然产生），非字面 SDD 定义"},
-    {"id": "U-041", "phase": "PH5", "sub_phase": "PH5-S3", "name": "c_deploy_supervisor_rollback", "classification": "Risk Gate", "depends_on": ["U-038", "U-040"], "coverage": {"model": ["deployment"], "view": [], "controller": ["c_deploy_supervisor_rollback"], "process": [], "requirement": []}, "done": "回滚路径 A/B 各跑通一次；旧版本也起不来时正确终态到 blocked_pending_human；git revert 冲突有明确处理"},
+    {"id": "U-041", "phase": "PH5", "sub_phase": "PH5-S3", "name": "c_deploy_supervisor_rollback（见 development-plan.md 2026-08-02 实现补充）", "classification": "Risk Gate", "depends_on": ["U-038", "U-040"], "coverage": {"model": ["deployment"], "view": [], "controller": ["c_deploy_supervisor_rollback"], "process": [], "requirement": []}, "done": "回滚路径 A/B 各跑通一次（supervisor/deploy_rollback.py，真实进程+真实 git 仓库验证）；旧版本也起不来时正确终态到 blocked_pending_human（真实场景验证，非假设）；git revert 冲突有明确处理（真实构造冲突历史验证过，含 --abort 清理）"},
     {"id": "U-042", "phase": "PH5", "sub_phase": "PH5-S3", "name": "自举闭环端到端验收", "classification": "Risk Gate", "depends_on": ["U-035", "U-041"], "coverage": {"model": [], "view": [], "controller": [], "process": [], "requirement": []}, "done": "真实复现 self-development-feasibility.md 9 步场景，全程无需人工评审代码内容；真实双实例 E2E 用例默认跳过，需显式环境变量开启"},
     {"id": "U-043", "phase": "PH5", "sub_phase": "PH5-S4", "name": "控制台展示 deployment 状态与健康信号", "classification": "Value Path", "depends_on": ["U-024", "U-040"], "coverage": {"model": [], "view": ["v_deployment_status"], "controller": [], "process": [], "requirement": []}, "done": "能看到当前 deployment 状态机所处状态与最近一次健康信号采样结果"}
   ],
+"""c_deploy_supervisor_rollback (03-C §3.15 / 04-P §4.6 / design doc §7):
+path A (pre-cutover, health check failed -- Live was never touched) and
+path B (post-cutover, anomaly detected during monitoring -- Live IS the bad
+version, has to actually be swapped back), plus the one fallback both paths
+share: rollback itself failing.
+
+**No unified `rollback(conn, deployment_id)` entry point (documented design
+choice, matches this subsystem's established convention)**: `deployment`
+persists `candidate_instance_dir` (U-037) but no PID column for either the
+candidate or the live process -- `deploy_stage.stage_deployment()` and
+`deploy_cutover.cutover()` both already require their caller to hand in the
+relevant pid(s) explicitly rather than look them up from the DB, since
+that's only ever meaningful within one continuous Deploy Supervisor process
+lifetime (the one that actually spawned them). `rollback_path_a`/
+`rollback_path_b` follow the same convention rather than inventing a
+lookup mechanism the rest of this subsystem doesn't have either.
+
+**`deployment.status` vs the ROLLBACK_FAILED return value (documented
+interpretation)**: 03-C's `output_contract` for `c_deploy_supervisor_rollback`
+lists only `{status: 'rolled_back' | 'health_check_failed' | 'rollback_failed'}`
+-- `blocked_pending_human` never appears there, but IS the literal
+`error_state` 04-P's own transition table names for this row, and IS one of
+`deployment.status`'s valid enum values (01-M 1.15). Read together: 'rollback_
+failed' is the ERROR NAME (what gets raised / what 03-C's contract lists),
+while the actually-PERSISTED terminal `deployment.status` is
+'blocked_pending_human' -- exactly the same pattern `deploy_stage.
+CheckoutFailedError` and `deploy_cutover.CutoverFailedError` already use
+(raise a *FailedError, persist status='blocked_pending_human'), so this
+follows the same convention rather than introducing a new one.
+
+**`git revert` conflict handling (design doc §10 open question 3, this
+Unit's own explicit "有明确处理" Done requirement)**: any `git revert`
+failure (conflict or otherwise) folds into the SAME `RollbackFailedError` ->
+`blocked_pending_human` fallback both paths already have -- not a new
+state-machine branch, per the design doc's own recommendation ("应该并入
+7.3 兜底路径，而不是新开一个分支"). `git revert --abort` is run best-effort
+afterward so a failed attempt doesn't leave the repo mid-conflict for
+whoever investigates the resulting `blocked_pending_human`.
+"""
+
+import os
+import subprocess
+
+from supervisor import deploy_cutover, deploy_health_check, deploy_stage
+from src import bug_record, paths
+
+
+class RollbackFailedError(Exception):
+    """ROLLBACK_FAILED -- 03-C §3.15: the automatic recovery mechanism
+    itself failed (old version won't start either, or `git revert`
+    conflicts) -- NOT a judgment about whose code is right. Not retried;
+    `deployment.status='blocked_pending_human'` is already applied by the
+    time this is raised."""
+
+
+def _git_revert(repo_root, commit_hash):
+    subprocess.run(
+        ["git", "revert", "--no-edit", commit_hash],
+        cwd=repo_root, check=True, capture_output=True, text=True,
+    )
+
+
+def _abort_revert_if_in_progress(repo_root):
+    """Best-effort: restores the pre-revert state so a failed attempt
+    doesn't leave the repo mid-conflict."""
+    subprocess.run(["git", "revert", "--abort"], cwd=repo_root, capture_output=True, text=True)
+
+
+def _block(conn, deployment_id, reason):
+    conn.execute(
+        "UPDATE deployment SET status = 'blocked_pending_human', rollback_reason = ? WHERE deployment_id = ?",
+        (reason, deployment_id),
+    )
+    conn.commit()
+
+
+def rollback_path_a(
+    conn, deployment_id, candidate_pid, candidate_db_path, git_commit_hash,
+    repo_root=None, description=None, task_id=None, stop_timeout=10,
+):
+    """Pre-cutover rollback (design doc §7.1): Live was never touched, this
+    is the cheapest path. Terminates the candidate, deletes its one-time DB
+    copy, `git revert`s the merge commit, and records a `bug_record(
+    root_cause_category=core_engine)` audit trail with the failing signals
+    as evidence. Returns {status: 'rolled_back'}."""
+    repo_root = repo_root or paths.repo_root()
+    description = description or "health check failed for deployment {}".format(deployment_id)
+
+    try:
+        deploy_cutover._stop_process(candidate_pid, timeout=stop_timeout)
+        if candidate_db_path and os.path.exists(candidate_db_path):
+            os.remove(candidate_db_path)
+        _git_revert(repo_root, git_commit_hash)
+    except subprocess.CalledProcessError as e:
+        _abort_revert_if_in_progress(repo_root)
+        _block(conn, deployment_id, "rollback_failed (path A): git revert failed: {}".format(e.stderr))
+        raise RollbackFailedError(
+            "rollback path A failed for deployment {}: git revert error: {}".format(deployment_id, e.stderr)
+        ) from e
+
+    bug_record.record_core_engine_change(conn, task_id, description)
+    conn.execute("UPDATE deployment SET status = 'rolled_back' WHERE deployment_id = ?", (deployment_id,))
+    conn.commit()
+    return {"status": "rolled_back"}
+
+
+def rollback_path_b(
+    conn, deployment_id, live_pid, previous_instance_dir, prod_db_path, git_commit_hash,
+    repo_root=None, description=None, task_id=None, live_port=deploy_cutover.LIVE_PORT,
+    python_executable=None, stop_timeout=10, bind_timeout=15, startup_check_timeout=30,
+):
+    """Post-cutover rollback (design doc §7.2): Live IS the bad version, so
+    the previously-known-good checkout has to be revived and re-verified
+    before this can be trusted as the new Live. Returns {status:
+    'rolled_back'}. Raises RollbackFailedError (deployment already marked
+    blocked_pending_human) if the revived previous_instance_dir fails to
+    bind live_port OR fails its own re-verification (same three checks
+    c_deploy_supervisor_health_check runs, called directly here rather than
+    through that module's deployment-row-mutating wrapper -- this is a
+    distinct concern from the original candidate's own staged health check
+    trail and shouldn't be interleaved with it)."""
+    repo_root = repo_root or paths.repo_root()
+    description = description or "post-cutover anomaly detected for deployment {}".format(deployment_id)
+
+    conn.execute("UPDATE deployment SET status = 'rollback_in_progress' WHERE deployment_id = ?", (deployment_id,))
+    conn.commit()
+
+    try:
+        deploy_cutover._stop_process(live_pid, timeout=stop_timeout)
+
+        revived = deploy_stage._launch_candidate(previous_instance_dir, live_port, prod_db_path, python_executable)
+        if not deploy_cutover._wait_for_live_http(live_port, bind_timeout):
+            raise RollbackFailedError(
+                "previous_instance_dir {} never bound live_port {} during rollback".format(
+                    previous_instance_dir, live_port
+                )
+            )
+
+        signals = [
+            deploy_health_check.check_process_alive(revived.pid, timeout=startup_check_timeout),
+            deploy_health_check.check_http_health(live_port),
+            deploy_health_check.check_canary_task(live_port),
+        ]
+        failing = [s for s in signals if s["status"] != "ok"]
+        if failing:
+            raise RollbackFailedError(
+                "revived previous_instance_dir failed re-verification for deployment {}: {}".format(
+                    deployment_id, failing
+                )
+            )
+
+        _git_revert(repo_root, git_commit_hash)
+    except subprocess.CalledProcessError as e:
+        _abort_revert_if_in_progress(repo_root)
+        _block(conn, deployment_id, "rollback_failed (path B): git revert failed: {}".format(e.stderr))
+        raise RollbackFailedError(
+            "rollback path B failed for deployment {}: git revert error: {}".format(deployment_id, e.stderr)
+        ) from e
+    except RollbackFailedError as e:
+        _block(conn, deployment_id, "rollback_failed (path B): {}".format(e))
+        raise
+
+    bug_record.record_core_engine_change(conn, task_id, description)
+    conn.execute("UPDATE deployment SET status = 'rolled_back' WHERE deployment_id = ?", (deployment_id,))
+    conn.commit()
+    return {"status": "rolled_back"}
+"""U-041 tests: c_deploy_supervisor_rollback. Real subprocesses, real git
+repos, real filesystem operations throughout -- this is the highest-rework-
+risk Unit on PH5's critical path (development-plan.md's own note: "回滚本身
+失败没有再下一层的自动兜底"), so its own "rollback itself fails" fallback is
+tested for real (a genuinely broken previous_instance_dir, a genuinely
+conflicting git revert), not assumed to work because the happy path does.
+
+Run: python3 -m unittest discover -s supervisor -p "test_*.py" -v
+"""
+
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+import time
+import unittest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from supervisor import deploy_cutover, deploy_health_check, deploy_rollback, deploy_stage, deploy_supervisor
+from src import paths, schema
+
+
+def _init_git_repo(path):
+    subprocess.run(["git", "init", "-q"], cwd=path, check=True)
+    subprocess.run(["git", "config", "user.email", "test@graphlemon.local"], cwd=path, check=True)
+    subprocess.run(["git", "config", "user.name", "test"], cwd=path, check=True)
+
+
+def _write(path, name, content):
+    with open(os.path.join(path, name), "w", encoding="utf-8") as f:
+        f.write(content)
+
+
+def _commit_all(path, message):
+    subprocess.run(["git", "add", "-A"], cwd=path, check=True)
+    subprocess.run(["git", "commit", "-q", "-m", message], cwd=path, check=True)
+    return subprocess.run(
+        ["git", "rev-parse", "HEAD"], cwd=path, capture_output=True, text=True, check=True
+    ).stdout.strip()
+
+
+class DeployRollbackTestBase(unittest.TestCase):
+    def setUp(self):
+        self.home_tmp = tempfile.mkdtemp(prefix="deploy-rollback-home-")
+        self.addCleanup(shutil.rmtree, self.home_tmp, ignore_errors=True)
+        self.old_home = os.environ.get("GRAPHLEMON_HOME")
+        os.environ["GRAPHLEMON_HOME"] = self.home_tmp
+
+        self.prod_db_path = os.path.join(self.home_tmp, "prod.db")
+        prod_conn = schema.connect(self.prod_db_path)
+        schema.create_schema(prod_conn)
+        prod_conn.close()
+
+        self.conn = schema.connect(":memory:")
+        schema.create_schema(self.conn)
+        self.addCleanup(self.conn.close)
+
+        self._cleanup_pids = []
+
+    def tearDown(self):
+        if self.old_home is None:
+            os.environ.pop("GRAPHLEMON_HOME", None)
+        else:
+            os.environ["GRAPHLEMON_HOME"] = self.old_home
+        for pid in self._cleanup_pids:
+            if deploy_supervisor.is_alive(pid):
+                os.kill(pid, 9)
+
+    def _seed_deployment(self, deployment_id, status, **extra):
+        cols = ["deployment_id", "git_commit_hash", "previous_commit_hash", "status", "previous_instance_dir"]
+        vals = [deployment_id, extra.get("git_commit_hash", "c1"), extra.get("previous_commit_hash", "c0"), status,
+                extra.get("previous_instance_dir", "/prev")]
+        self.conn.execute(
+            "INSERT INTO deployment ({}) VALUES ({})".format(", ".join(cols), ", ".join("?" * len(cols))), vals
+        )
+        self.conn.commit()
+
+
+class RollbackPathAHappyPathTest(DeployRollbackTestBase):
+    def setUp(self):
+        super().setUp()
+        self.repo_dir = tempfile.mkdtemp(prefix="rollback-a-repo-")
+        self.addCleanup(shutil.rmtree, self.repo_dir, ignore_errors=True)
+        _init_git_repo(self.repo_dir)
+        _write(self.repo_dir, "f.txt", "v1\n")
+        self.commit_v1 = _commit_all(self.repo_dir, "v1")
+        _write(self.repo_dir, "f.txt", "v2 (self-modification)\n")
+        self.commit_v2 = _commit_all(self.repo_dir, "v2")
+
+    def test_stops_candidate_deletes_db_reverts_commit_records_bug(self):
+        candidate_db = os.path.join(self.home_tmp, "candidate.db")
+        shutil.copy2(self.prod_db_path, candidate_db)
+        proc = subprocess.Popen(["sleep", "300"])
+        self._cleanup_pids.append(proc.pid)
+
+        self._seed_deployment("dep-a1", "health_check_failed", git_commit_hash=self.commit_v2)
+
+        result = deploy_rollback.rollback_path_a(
+            self.conn, "dep-a1", proc.pid, candidate_db, self.commit_v2, repo_root=self.repo_dir,
+            description="canary_task failed: connection refused",
+        )
+
+        self.assertEqual(result["status"], "rolled_back")
+        self.assertFalse(deploy_supervisor.is_alive(proc.pid))
+        self.assertFalse(os.path.exists(candidate_db))
+
+        with open(os.path.join(self.repo_dir, "f.txt"), encoding="utf-8") as f:
+            self.assertEqual(f.read(), "v1\n")  # revert landed
+
+        row = self.conn.execute("SELECT status FROM deployment WHERE deployment_id = 'dep-a1'").fetchone()
+        self.assertEqual(row[0], "rolled_back")
+
+        bug = self.conn.execute(
+            "SELECT root_cause_category, status, description FROM bug_record"
+        ).fetchone()
+        self.assertEqual(bug[0], "core_engine")
+        self.assertEqual(bug[1], "resolved")
+        self.assertIn("canary_task failed", bug[2])
+
+
+class RollbackPathAFailureTest(DeployRollbackTestBase):
+    def test_git_revert_conflict_marks_blocked_pending_human_and_raises(self):
+        repo_dir = tempfile.mkdtemp(prefix="rollback-a-conflict-repo-")
+        self.addCleanup(shutil.rmtree, repo_dir, ignore_errors=True)
+        _init_git_repo(repo_dir)
+        _write(repo_dir, "f.txt", "v1\n")
+        _commit_all(repo_dir, "v1")
+        _write(repo_dir, "f.txt", "v2 (self-modification)\n")
+        commit_v2 = _commit_all(repo_dir, "v2")
+        # A later commit that touches the SAME line v2 changed -- reverting
+        # v2 now conflicts, the exact scenario design doc §10 open question
+        # 3 describes ("主分支又有新的提交，revert 可能冲突").
+        _write(repo_dir, "f.txt", "v3 (unrelated later change, same line)\n")
+        _commit_all(repo_dir, "v3")
+
+        self._seed_deployment("dep-a2", "health_check_failed", git_commit_hash=commit_v2)
+
+        with self.assertRaises(deploy_rollback.RollbackFailedError):
+            deploy_rollback.rollback_path_a(
+                self.conn, "dep-a2", candidate_pid=None, candidate_db_path=None,
+                git_commit_hash=commit_v2, repo_root=repo_dir,
+            )
+
+        row = self.conn.execute(
+            "SELECT status, rollback_reason FROM deployment WHERE deployment_id = 'dep-a2'"
+        ).fetchone()
+        self.assertEqual(row[0], "blocked_pending_human")
+        self.assertIn("git revert failed", row[1])
+
+        # repo left clean, not mid-conflict, for whoever investigates
+        status_output = subprocess.run(
+            ["git", "status", "--porcelain"], cwd=repo_dir, capture_output=True, text=True, check=True
+        ).stdout
+        self.assertEqual(status_output.strip(), "")
+
+
+class RollbackPathBHappyPathTest(DeployRollbackTestBase):
+    def setUp(self):
+        super().setUp()
+        self.repo_dir = tempfile.mkdtemp(prefix="rollback-b-repo-")
+        self.addCleanup(shutil.rmtree, self.repo_dir, ignore_errors=True)
+        _init_git_repo(self.repo_dir)
+        _write(self.repo_dir, "f.txt", "v1\n")
+        self.commit_v1 = _commit_all(self.repo_dir, "v1")
+        _write(self.repo_dir, "f.txt", "v2 (self-modification, buggy)\n")
+        self.commit_v2 = _commit_all(self.repo_dir, "v2")
+
+        # The "previously known-good" checkout -- a REAL worktree of THIS
+        # actual graphlemon repo (needs real src/backend_server.py to relaunch),
+        # not repo_dir (which has no real graphlemon source in it).
+        self._prev_stash = subprocess.run(
+            ["git", "stash", "create"], cwd=paths.repo_root(), capture_output=True, text=True, check=True
+        ).stdout.strip() or deploy_stage._current_commit(paths.repo_root())
+        self.prev_dir = os.path.join(self.home_tmp, "previous-instance")
+        subprocess.run(
+            ["git", "worktree", "add", "--detach", self.prev_dir, self._prev_stash],
+            cwd=paths.repo_root(), check=True, capture_output=True, text=True,
+        )
+        self.addCleanup(
+            lambda: subprocess.run(
+                ["git", "worktree", "remove", "--force", self.prev_dir], cwd=paths.repo_root(), capture_output=True
+            )
+        )
+
+    def test_stops_bad_live_revives_previous_reverts_commit_records_bug(self):
+        bad_live_port = deploy_stage._find_free_port()
+        bad_live = deploy_stage._launch_candidate(paths.repo_root(), bad_live_port, self.prod_db_path)
+        self._cleanup_pids.append(bad_live.pid)
+        self._wait_for_port(bad_live_port)
+
+        self._seed_deployment(
+            "dep-b1", "anomaly_detected", git_commit_hash=self.commit_v2,
+            previous_instance_dir=self.prev_dir,
+        )
+
+        rollback_port = deploy_stage._find_free_port()
+        result = deploy_rollback.rollback_path_b(
+            self.conn, "dep-b1", bad_live.pid, self.prev_dir, self.prod_db_path, self.commit_v2,
+            repo_root=self.repo_dir, live_port=rollback_port,
+        )
+        self._cleanup_pids.append(  # find the revived process's pid via port not tracked -- best-effort cleanup
+            bad_live.pid
+        )
+
+        self.assertEqual(result["status"], "rolled_back")
+        self.assertFalse(deploy_supervisor.is_alive(bad_live.pid))
+
+        with open(os.path.join(self.repo_dir, "f.txt"), encoding="utf-8") as f:
+            self.assertEqual(f.read(), "v1\n")
+
+        row = self.conn.execute("SELECT status FROM deployment WHERE deployment_id = 'dep-b1'").fetchone()
+        self.assertEqual(row[0], "rolled_back")
+
+        bug = self.conn.execute("SELECT root_cause_category FROM bug_record").fetchone()
+        self.assertEqual(bug[0], "core_engine")
+
+        # revived instance genuinely serving on rollback_port
+        health = deploy_health_check.check_http_health(rollback_port)
+        self.assertEqual(health["status"], "ok")
+
+    def _wait_for_port(self, port, timeout=10):
+        deadline = time.time() + timeout
+        while time.time() < deadline:
+            if deploy_health_check.check_http_health(port, timeout=1)["status"] == "ok":
+                return
+            time.sleep(0.2)
+        self.fail("port {} never came up".format(port))
+
+
+class RollbackPathBFailureTest(DeployRollbackTestBase):
+    """The Unit's own highest-risk scenario, tested for real: the old
+    version ALSO can't start."""
+
+    def test_previous_instance_also_fails_to_start_marks_blocked_pending_human(self):
+        repo_dir = tempfile.mkdtemp(prefix="rollback-b-fail-repo-")
+        self.addCleanup(shutil.rmtree, repo_dir, ignore_errors=True)
+        _init_git_repo(repo_dir)
+        _write(repo_dir, "f.txt", "v1\n")
+        _commit_all(repo_dir, "v1")
+        _write(repo_dir, "f.txt", "v2\n")
+        commit_v2 = _commit_all(repo_dir, "v2")
+
+        broken_previous_dir = os.path.join(self.home_tmp, "broken-previous")
+        os.makedirs(broken_previous_dir)  # a real directory, but with NO python package in it at all
+
+        self._seed_deployment(
+            "dep-b2", "anomaly_detected", git_commit_hash=commit_v2, previous_instance_dir=broken_previous_dir,
+        )
+
+        with self.assertRaises(deploy_rollback.RollbackFailedError):
+            deploy_rollback.rollback_path_b(
+                self.conn, "dep-b2", live_pid=None, previous_instance_dir=broken_previous_dir,
+                prod_db_path=self.prod_db_path, git_commit_hash=commit_v2, repo_root=repo_dir,
+                live_port=deploy_stage._find_free_port(), bind_timeout=3,
+            )
+
+        row = self.conn.execute(
+            "SELECT status, rollback_reason FROM deployment WHERE deployment_id = 'dep-b2'"
+        ).fetchone()
+        self.assertEqual(row[0], "blocked_pending_human")
+        self.assertIn("rollback_failed", row[1])
+
+        # commit was NOT reverted -- rollback failed before it ever got there
+        with open(os.path.join(repo_dir, "f.txt"), encoding="utf-8") as f:
+            self.assertEqual(f.read(), "v2\n")
+
+
+if __name__ == "__main__":
+    unittest.main()