Commit 05292668f4f37a372d5c822d0c704603cba2f35b

Authored by bboonnee
1 parent 4f4b1853

U-042: self-hosting closed-loop acceptance (orchestration + gated real test)

Adds supervisor/self_hosting_e2e.py: run_self_hosting_closed_loop() chains
every PH5 module built this session (U-034~U-041) with the Graph Engine's
own scheduler (U-032) into one real, callable path matching self-development-
feasibility.md's 9-step scenario -- self-target Goal -> Planner/Coder/
Validator -> core_engine classification -> stage -> health check -> cutover
(or rollback path A if health check fails) -> monitor until either
monitoring_window_clean (finalized) or anomaly_detected (rollback path B).
Both real outcome paths are branches of the same call, not two separate
scripted scenarios. Explicitly documented as this acceptance scenario's own
orchestration glue, not a claim about Deploy Supervisor's eventual
production poll loop (U-036's discovery-loop hook is still unfilled;
building that is future work).

Mocked wiring tests (6, run in normal regression) stub each already
independently-tested module boundary and only verify the branching logic.

The real, literal acceptance test is built and ready (RealSelfHostingClosedLoopTest,
reproduces the document's own "loop_round > 3 should be >= 3" example) but
gated behind RUN_REAL_SELF_HOSTING_E2E=1 -- deliberately a separate,
distinctly-named env var from this project's usual RUN_REAL_AGENT_E2E, since
this is the highest-blast-radius action in the project (can replace the
running graphlemon process and git-revert commits on this real repo).
Per this Unit's own explicit build instruction, not executed as part of
implementing it -- doing so requires checking with the user first.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
docs/development-plan.md
@@ -396,6 +396,10 @@ PH2-S1/S2/S3/S4 之间除标注的依赖外相互独立,可并行推进。 @@ -396,6 +396,10 @@ PH2-S1/S2/S3/S4 之间除标注的依赖外相互独立,可并行推进。
396 - **U-042 [Risk Gate]** 自举闭环端到端验收:真实复现 `self-development-feasibility.md` 第 3 节的 9 步场景(修复 graphlemon 自己 `p_org_task_lifecycle` 状态机里的一个真实植入 bug),走完整"部署→健康检查→cutover→监控→finalize"或"检测异常→自动回滚"两条路径中至少一条 396 - **U-042 [Risk Gate]** 自举闭环端到端验收:真实复现 `self-development-feasibility.md` 第 3 节的 9 步场景(修复 graphlemon 自己 `p_org_task_lifecycle` 状态机里的一个真实植入 bug),走完整"部署→健康检查→cutover→监控→finalize"或"检测异常→自动回滚"两条路径中至少一条
397 - 依赖:U-035, U-041 397 - 依赖:U-035, U-041
398 - Done:全程无需人工评审代码内容;参照 graphloop `RUN_REAL_AGENT_E2E=1` 的先例,真实拉起两个 graphlemon 实例的端到端用例默认跳过,需要显式环境变量开启,日常回归只跑 mock 版本 398 - Done:全程无需人工评审代码内容;参照 graphloop `RUN_REAL_AGENT_E2E=1` 的先例,真实拉起两个 graphlemon 实例的端到端用例默认跳过,需要显式环境变量开启,日常回归只跑 mock 版本
  399 + - **实现补充(2026-08-02)**:新增 `supervisor/self_hosting_e2e.py`:`run_self_hosting_closed_loop()` 把这次会话建好的全部 PH5 模块(U-034~U-041)+ Graph Engine 自己的 `scheduler.run_goal_to_completion`(U-032)串成一条真实可调用的链路——真实 self-target Goal → Planner/Coder/Validator(真实 `claude` 调用,`scheduler` 内部已经走 U-035 的 self-target 强制 worktree)→ PASS 后走 U-035 记的 `core_engine` 审计 `bug_record` → `deploy_stage.stage_deployment` → `deploy_health_check.run_health_check` → 通过则 `deploy_cutover.cutover`,不通过直接 `deploy_rollback.rollback_path_a` 停在这一步 → 循环 `deploy_monitor.run_monitor_cycle` 直到 `monitoring_window_clean`(`finalized`)或 `anomaly_detected`(`deploy_rollback.rollback_path_b`)——**这正是本 Unit 要求的"两条路径至少走一条",实现上两条路径都是同一次真实调用可能走到的真实分支,不是分别写死两个脚本**。**明确不是**生产级 Deploy Supervisor 的最终 `poll_fn` 实现——真正独立轮询"有没有新的 self-target 合并等待部署"的发现机制是 U-036 留的钩子,本 Unit 只提供验收场景自己的编排入口,不代表以后正式的常驻发现循环长这样,避免把验收脚本和生产实现搞混。
  400 + - **Mock 版本(日常回归)**:`test_self_hosting_e2e.py` 前 6 个测试类在每个已经独立测试过真实机制的模块边界上打桩(`scheduler.run_goal_to_completion`/`deploy_stage.stage_deployment`/`deploy_health_check.run_health_check`/`deploy_cutover.cutover`/`deploy_monitor.run_monitor_cycle`/`deploy_rollback.rollback_path_a`/`rollback_path_b`),只验证编排本身的分支/顺序逻辑对不对(任务没到 done 提前返回、不是 core_engine 提前返回、健康检查失败走 A 不碰 cutover、监控干净走 finalized、监控异常走 B),不重新证明任何一个模块自己的真实机制——那些已经在各自的测试文件里用真实进程/真实 git 仓库验证过了。
  401 + - **真实版本,已建好但没有执行**:`RealSelfHostingClosedLoopTest`,字面复现 `self-development-feasibility.md` 例子本身("loop_round > 3 应该是 loop_round >= 3")。**用了一个和项目一贯的 `RUN_REAL_AGENT_E2E` 不同、单独命名的环境变量 `RUN_REAL_SELF_HOSTING_E2E`**——这是本项目目前为止影响面最大的一个动作,不只是花真钱调用 `claude`(那是 `RUN_REAL_AGENT_E2E` 覆盖的常规风险),而是真的可能替换掉正在跑的 graphlemon 进程本身、真的会在这个仓库自己的 git 历史上跑 `git revert`——用不同名字的环境变量,是为了不让"为了跑别的真实测试而设的 `RUN_REAL_AGENT_E2E=1`"意外带着触发这个。**没有替用户执行这个测试**:按本 Unit 开工前收到的明确指示,这类"影响范围超出本进程/本会话"的动作,写好、gate 好就停,真正执行前需要用 AskUserQuestion 问过用户——这次实现结束后立刻会问。
  402 + - 覆盖:`self_hosting_e2e.py`(编排层,串联 U-032/034~041 全部真实模块);真实验收场景已建好、默认不跑
399 403
400 ### PH5-S4 可观测性(非阻塞,可延后) 404 ### PH5-S4 可观测性(非阻塞,可延后)
401 405
docs/sdd/unit_matrix.json
@@ -59,7 +59,7 @@ @@ -59,7 +59,7 @@
59 {"id": "U-039", "phase": "PH5", "sub_phase": "PH5-S2", "name": "c_deploy_supervisor_cutover(见 development-plan.md 2026-08-02 实现补充)", "classification": "Risk Gate", "depends_on": ["U-038"], "coverage": {"model": ["deployment"], "view": [], "controller": ["c_deploy_supervisor_cutover"], "process": [], "requirement": []}, "done": "健康检查通过后完成真实切换(supervisor/deploy_cutover.py,真实进程停止+重启验证过);在途 work_node_execution 重启后可恢复(复用既有 REQ-06 机制);previous_instance_dir 保留至监控窗口结束;新增 LIVE_PORT=8420 常量(05 §5.9 缺的一个字段);真实测试发现并修复 deploy_supervisor.is_alive() 的僵尸进程判活缺陷(新增 reap_if_exited())"}, 59 {"id": "U-039", "phase": "PH5", "sub_phase": "PH5-S2", "name": "c_deploy_supervisor_cutover(见 development-plan.md 2026-08-02 实现补充)", "classification": "Risk Gate", "depends_on": ["U-038"], "coverage": {"model": ["deployment"], "view": [], "controller": ["c_deploy_supervisor_cutover"], "process": [], "requirement": []}, "done": "健康检查通过后完成真实切换(supervisor/deploy_cutover.py,真实进程停止+重启验证过);在途 work_node_execution 重启后可恢复(复用既有 REQ-06 机制);previous_instance_dir 保留至监控窗口结束;新增 LIVE_PORT=8420 常量(05 §5.9 缺的一个字段);真实测试发现并修复 deploy_supervisor.is_alive() 的僵尸进程判活缺陷(新增 reap_if_exited())"},
60 {"id": "U-040", "phase": "PH5", "sub_phase": "PH5-S3", "name": "c_deploy_supervisor_monitor(见 development-plan.md 2026-08-02 实现补充)", "classification": "Risk Gate", "depends_on": ["U-039"], "coverage": {"model": ["deploy_health_signal"], "view": [], "controller": ["c_deploy_supervisor_monitor"], "process": [], "requirement": []}, "done": "四类信号采样正常(supervisor/deploy_monitor.py);validator_approval_rate 偏离基线阈值时触发 anomaly_detected;approve 占比算法基于对 scheduler.finish_task_lifecycle 真实代码逻辑的核实(PASS 从不产生 bug_record,FAIL 必然产生),非字面 SDD 定义"}, 60 {"id": "U-040", "phase": "PH5", "sub_phase": "PH5-S3", "name": "c_deploy_supervisor_monitor(见 development-plan.md 2026-08-02 实现补充)", "classification": "Risk Gate", "depends_on": ["U-039"], "coverage": {"model": ["deploy_health_signal"], "view": [], "controller": ["c_deploy_supervisor_monitor"], "process": [], "requirement": []}, "done": "四类信号采样正常(supervisor/deploy_monitor.py);validator_approval_rate 偏离基线阈值时触发 anomaly_detected;approve 占比算法基于对 scheduler.finish_task_lifecycle 真实代码逻辑的核实(PASS 从不产生 bug_record,FAIL 必然产生),非字面 SDD 定义"},
61 {"id": "U-041", "phase": "PH5", "sub_phase": "PH5-S3", "name": "c_deploy_supervisor_rollback(见 development-plan.md 2026-08-02 实现补充)", "classification": "Risk Gate", "depends_on": ["U-038", "U-040"], "coverage": {"model": ["deployment"], "view": [], "controller": ["c_deploy_supervisor_rollback"], "process": [], "requirement": []}, "done": "回滚路径 A/B 各跑通一次(supervisor/deploy_rollback.py,真实进程+真实 git 仓库验证);旧版本也起不来时正确终态到 blocked_pending_human(真实场景验证,非假设);git revert 冲突有明确处理(真实构造冲突历史验证过,含 --abort 清理)"}, 61 {"id": "U-041", "phase": "PH5", "sub_phase": "PH5-S3", "name": "c_deploy_supervisor_rollback(见 development-plan.md 2026-08-02 实现补充)", "classification": "Risk Gate", "depends_on": ["U-038", "U-040"], "coverage": {"model": ["deployment"], "view": [], "controller": ["c_deploy_supervisor_rollback"], "process": [], "requirement": []}, "done": "回滚路径 A/B 各跑通一次(supervisor/deploy_rollback.py,真实进程+真实 git 仓库验证);旧版本也起不来时正确终态到 blocked_pending_human(真实场景验证,非假设);git revert 冲突有明确处理(真实构造冲突历史验证过,含 --abort 清理)"},
62 - {"id": "U-042", "phase": "PH5", "sub_phase": "PH5-S3", "name": "自举闭环端到端验收", "classification": "Risk Gate", "depends_on": ["U-035", "U-041"], "coverage": {"model": [], "view": [], "controller": [], "process": [], "requirement": []}, "done": "真实复现 self-development-feasibility.md 9 步场景,全程无需人工评审代码内容;真实双实例 E2E 用例默认跳过,需显式环境变量开启"}, 62 + {"id": "U-042", "phase": "PH5", "sub_phase": "PH5-S3", "name": "自举闭环端到端验收(见 development-plan.md 2026-08-02 实现补充)", "classification": "Risk Gate", "depends_on": ["U-035", "U-041"], "coverage": {"model": [], "view": [], "controller": [], "process": [], "requirement": []}, "done": "真实复现 self-development-feasibility.md 9 步场景,全程无需人工评审代码内容(supervisor/self_hosting_e2e.py 编排层已建好);真实双实例 E2E 用例(RUN_REAL_SELF_HOSTING_E2E=1,独立于常规 RUN_REAL_AGENT_E2E 的单独环境变量)默认跳过,需显式环境变量开启且需额外用户确认才真正执行;日常回归只跑 mock 版本(6 个编排逻辑测试,已跑通)"},
63 {"id": "U-043", "phase": "PH5", "sub_phase": "PH5-S4", "name": "控制台展示 deployment 状态与健康信号", "classification": "Value Path", "depends_on": ["U-024", "U-040"], "coverage": {"model": [], "view": ["v_deployment_status"], "controller": [], "process": [], "requirement": []}, "done": "能看到当前 deployment 状态机所处状态与最近一次健康信号采样结果"} 63 {"id": "U-043", "phase": "PH5", "sub_phase": "PH5-S4", "name": "控制台展示 deployment 状态与健康信号", "classification": "Value Path", "depends_on": ["U-024", "U-040"], "coverage": {"model": [], "view": ["v_deployment_status"], "controller": [], "process": [], "requirement": []}, "done": "能看到当前 deployment 状态机所处状态与最近一次健康信号采样结果"}
64 ], 64 ],
65 "classification_summary": { 65 "classification_summary": {
supervisor/self_hosting_e2e.py 0 → 100644
  1 +"""U-042: self-hosting closed-loop acceptance -- wires every PH5 module
  2 +built so far (U-034~U-041) plus the Graph Engine's own scheduler into the
  3 +literal 9-step scenario `self-development-feasibility.md` walks through:
  4 +a real self-target Goal, Planner->Coder->Validator (real `claude` calls via
  5 +`scheduler.run_goal_to_completion`, which already forces worktree isolation
  6 +and records the `core_engine` audit trail for self-target tasks per U-035),
  7 +merge, stage a candidate, health-check it, cut it over, monitor it, and
  8 +land on EITHER `finalized` (clean monitoring window) OR a rollback (anomaly
  9 +detected) -- "走完整...两条路径中至少一条", not a scripted single outcome.
  10 +
  11 +This module is the acceptance scenario's own orchestration glue, not a
  12 +claim about what Deploy Supervisor's real production `poll_fn` (U-036's
  13 +still-unfilled hook) should look like long-term -- a continuously-running
  14 +daemon would discover self-target merges to stage on its own schedule
  15 +rather than drive one linear call end to end. Building that discovery loop
  16 +is future work, not this Unit's job; this function exists so the acceptance
  17 +scenario has one real, callable entry point to run (or NOT run -- see
  18 +test_self_hosting_e2e.py's gating) against.
  19 +"""
  20 +
  21 +import time
  22 +
  23 +from supervisor import deploy_cutover, deploy_health_check, deploy_monitor, deploy_rollback, deploy_stage
  24 +from src import paths, schema, scheduler
  25 +
  26 +
  27 +class SelfHostingLoopError(Exception):
  28 + """Raised when a precondition for even attempting the scenario isn't
  29 + met (e.g. target_folder_path isn't genuinely self-target) -- distinct
  30 + from a real outcome like 'rolled_back', which is a valid, successful
  31 + exercise of the safety mechanism, not an error."""
  32 +
  33 +
  34 +def run_self_hosting_closed_loop(
  35 + db_path, goal_text, target_folder_path=None, prod_db_path=None,
  36 + poll_interval=5.0, monitor_timeout_cycles=None, sleep=time.sleep, now=time.time,
  37 +):
  38 + """Returns {"stage", "outcome", ...}. `stage` names how far the scenario
  39 + got (`task_lifecycle` / `classification` / `health_check` / `cutover` /
  40 + `monitor`); `outcome` is the concrete result at that stage. A task that
  41 + doesn't reach 'done', or doesn't get classified core_engine, is a valid
  42 + early-return (not every self-target task is a core-engine change -- see
  43 + U-035), not a scenario failure.
  44 + """
  45 + target_folder_path = target_folder_path or paths.repo_root()
  46 + prod_db_path = prod_db_path or paths.db_path()
  47 + if not paths.is_self_target(target_folder_path):
  48 + raise SelfHostingLoopError(
  49 + "target_folder_path {!r} is not this instance's own checkout ({!r}) -- "
  50 + "the self-hosting scenario requires self-target".format(target_folder_path, paths.repo_root())
  51 + )
  52 +
  53 + task_result = scheduler.run_goal_to_completion(db_path, target_folder_path, goal_text)
  54 + if task_result.get("final_status") != "done":
  55 + return {"stage": "task_lifecycle", "outcome": task_result.get("final_status"), "task_id": task_result.get("task_id")}
  56 +
  57 + conn = schema.connect(db_path)
  58 + try:
  59 + bug = conn.execute(
  60 + "SELECT bug_id FROM bug_record WHERE task_id = ? AND root_cause_category = 'core_engine' "
  61 + "ORDER BY rowid DESC LIMIT 1",
  62 + (task_result["task_id"],),
  63 + ).fetchone()
  64 + if bug is None:
  65 + return {"stage": "classification", "outcome": "not_core_engine", "task_id": task_result["task_id"]}
  66 +
  67 + merge_commit = deploy_stage._current_commit(target_folder_path)
  68 +
  69 + staged = deploy_stage.stage_deployment(
  70 + conn, merge_commit, repo_root=target_folder_path, prod_db_path=prod_db_path
  71 + )
  72 + health = deploy_health_check.run_health_check(
  73 + conn, staged["deployment_id"], staged["candidate_pid"], staged["candidate_port"]
  74 + )
  75 + if health["status"] != "health_check_passed":
  76 + candidate_db_path = conn.execute(
  77 + "SELECT candidate_db_path FROM deployment WHERE deployment_id = ?", (staged["deployment_id"],)
  78 + ).fetchone()[0]
  79 + deploy_rollback.rollback_path_a(
  80 + conn, staged["deployment_id"], staged["candidate_pid"], candidate_db_path,
  81 + merge_commit, repo_root=target_folder_path,
  82 + description="health check failed: {}".format(health["signals"]),
  83 + )
  84 + return {"stage": "health_check", "outcome": "rolled_back_path_a", "deployment_id": staged["deployment_id"]}
  85 +
  86 + cutover_result = deploy_cutover.cutover(
  87 + conn, staged["deployment_id"], live_pid=None, candidate_pid=staged["candidate_pid"],
  88 + candidate_instance_dir=staged["candidate_instance_dir"], prod_db_path=prod_db_path,
  89 + )
  90 +
  91 + project_id = conn.execute(
  92 + "SELECT project_id FROM project WHERE target_folder_path = ?", (target_folder_path,)
  93 + ).fetchone()[0]
  94 + cutover_at = conn.execute(
  95 + "SELECT cutover_at FROM deployment WHERE deployment_id = ?", (staged["deployment_id"],)
  96 + ).fetchone()[0]
  97 + baseline = deploy_monitor.compute_baseline_validator_approval_rate(conn, project_id, cutover_at)
  98 + baseline_rate = baseline[0] if baseline else None
  99 +
  100 + cycles = 0
  101 + while True:
  102 + cycles += 1
  103 + cycle_result = deploy_monitor.run_monitor_cycle(
  104 + conn, staged["deployment_id"], cutover_result["live_pid"], cutover_result["live_port"],
  105 + project_id, baseline_rate, cutover_at, now=now,
  106 + )
  107 + if cycle_result["status"] == "monitoring_window_clean":
  108 + conn.execute(
  109 + "UPDATE deployment SET status = 'finalized' WHERE deployment_id = ?", (staged["deployment_id"],)
  110 + )
  111 + conn.commit()
  112 + return {"stage": "monitor", "outcome": "finalized", "deployment_id": staged["deployment_id"]}
  113 + if cycle_result["status"] == "anomaly_detected":
  114 + rb = deploy_rollback.rollback_path_b(
  115 + conn, staged["deployment_id"], cutover_result["live_pid"],
  116 + _previous_instance_dir(conn, staged["deployment_id"]),
  117 + prod_db_path, merge_commit, repo_root=target_folder_path,
  118 + description="post-cutover anomaly: {}".format(cycle_result["latest_signal"]),
  119 + )
  120 + outcome = "rolled_back_path_b" if rb["status"] == "rolled_back" else "rollback_failed"
  121 + return {"stage": "monitor", "outcome": outcome, "deployment_id": staged["deployment_id"]}
  122 + if monitor_timeout_cycles is not None and cycles >= monitor_timeout_cycles:
  123 + return {"stage": "monitor", "outcome": "timed_out_still_monitoring", "deployment_id": staged["deployment_id"]}
  124 + sleep(poll_interval)
  125 + finally:
  126 + conn.close()
  127 +
  128 +
  129 +def _previous_instance_dir(conn, deployment_id):
  130 + return conn.execute(
  131 + "SELECT previous_instance_dir FROM deployment WHERE deployment_id = ?", (deployment_id,)
  132 + ).fetchone()[0]
supervisor/test_self_hosting_e2e.py 0 → 100644
  1 +"""U-042 tests: self_hosting_e2e.py's orchestration wiring, mocked at each
  2 +already-independently-tested module's own boundary (deploy_stage/
  3 +deploy_health_check/deploy_cutover/deploy_monitor/deploy_rollback/scheduler
  4 +each have their own real-mechanics test files -- this file's job is proving
  5 +run_self_hosting_closed_loop() sequences and branches between them
  6 +correctly, not re-proving any of their own internals).
  7 +
  8 +The genuinely real, un-mocked acceptance run (self-development-feasibility.md's
  9 +literal 9-step scenario, actual `claude` calls, an actual candidate cutover
  10 +of THIS repo) lives in RealSelfHostingClosedLoopTest below, gated behind
  11 +RUN_REAL_SELF_HOSTING_E2E=1 -- a SEPARATE, stricter env var from this
  12 +project's usual RUN_REAL_AGENT_E2E, since this is the single highest-blast-
  13 +radius action in the whole project (it can actually replace the running
  14 +graphlemon process and git-revert commits on this real repo). Not run as
  15 +part of this Unit's own implementation -- see development-plan.md's U-042
  16 +note for why.
  17 +
  18 +Run: python3 -m unittest discover -s supervisor -p "test_*.py" -v
  19 +"""
  20 +
  21 +import os
  22 +import sys
  23 +import unittest
  24 +from unittest.mock import patch
  25 +
  26 +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
  27 +
  28 +from supervisor import (
  29 + deploy_cutover, deploy_health_check, deploy_monitor, deploy_rollback, deploy_stage, self_hosting_e2e,
  30 +)
  31 +from src import paths, schema, scheduler
  32 +
  33 +
  34 +class SelfTargetGuardTest(unittest.TestCase):
  35 + def test_non_self_target_raises_before_doing_anything(self):
  36 + with self.assertRaises(self_hosting_e2e.SelfHostingLoopError):
  37 + self_hosting_e2e.run_self_hosting_closed_loop("db.sqlite", "goal", target_folder_path="/tmp/not-graphlemon")
  38 +
  39 +
  40 +class TaskLifecycleNotDoneTest(unittest.TestCase):
  41 + def test_returns_early_without_touching_deploy_supervisor(self):
  42 + with patch.object(scheduler, "run_goal_to_completion", return_value={"task_id": "t1", "final_status": "blocked_pending_human"}), \
  43 + patch.object(deploy_stage, "stage_deployment") as mock_stage:
  44 + result = self_hosting_e2e.run_self_hosting_closed_loop("unused.db", "goal", target_folder_path=paths.repo_root())
  45 + self.assertEqual(result, {"stage": "task_lifecycle", "outcome": "blocked_pending_human", "task_id": "t1"})
  46 + mock_stage.assert_not_called()
  47 +
  48 +
  49 +class NotCoreEngineTest(unittest.TestCase):
  50 + def test_returns_early_when_no_core_engine_audit_bug_exists(self):
  51 + import tempfile
  52 + with tempfile.NamedTemporaryFile(suffix=".db", delete=False) as tmp:
  53 + db_path = tmp.name
  54 + conn = schema.connect(db_path)
  55 + schema.create_schema(conn)
  56 + conn.execute(
  57 + "INSERT INTO project (project_id, name, target_folder_path, has_git_repo, status) "
  58 + "VALUES ('proj-1', 'P', ?, 1, 'active')", (paths.repo_root(),)
  59 + )
  60 + conn.commit()
  61 + conn.close()
  62 + try:
  63 + with patch.object(scheduler, "run_goal_to_completion", return_value={"task_id": "t1", "final_status": "done"}), \
  64 + patch.object(deploy_stage, "stage_deployment") as mock_stage:
  65 + result = self_hosting_e2e.run_self_hosting_closed_loop(db_path, "goal", target_folder_path=paths.repo_root())
  66 + self.assertEqual(result, {"stage": "classification", "outcome": "not_core_engine", "task_id": "t1"})
  67 + mock_stage.assert_not_called()
  68 + finally:
  69 + os.remove(db_path)
  70 +
  71 +
  72 +class _RealDbTestBase(unittest.TestCase):
  73 + def setUp(self):
  74 + import tempfile
  75 + self.tmp = tempfile.NamedTemporaryFile(suffix=".db", delete=False)
  76 + self.tmp.close()
  77 + self.db_path = self.tmp.name
  78 + self.addCleanup(lambda: os.path.exists(self.db_path) and os.remove(self.db_path))
  79 +
  80 + conn = schema.connect(self.db_path)
  81 + schema.create_schema(conn)
  82 + conn.execute(
  83 + "INSERT INTO project (project_id, name, target_folder_path, has_git_repo, status) "
  84 + "VALUES ('proj-1', 'P', ?, 1, 'active')", (paths.repo_root(),)
  85 + )
  86 + conn.execute(
  87 + "INSERT INTO task (task_id, project_id, goal_text, task_type, status, assigned_org_node) "
  88 + "VALUES ('task-1', 'proj-1', 'goal', 'goal', 'done', 'validator')"
  89 + )
  90 + conn.execute(
  91 + "INSERT INTO bug_record (bug_id, task_id, source, description, root_cause_category, status) "
  92 + "VALUES ('bug-1', 'task-1', 'validator', 'self-target change', 'core_engine', 'resolved')"
  93 + )
  94 + conn.commit()
  95 + conn.close()
  96 +
  97 + def _mock_scheduler(self):
  98 + return patch.object(
  99 + scheduler, "run_goal_to_completion",
  100 + return_value={"task_id": "task-1", "final_status": "done"},
  101 + )
  102 +
  103 +
  104 +class HealthCheckFailedTest(_RealDbTestBase):
  105 + def test_routes_to_rollback_path_a_and_stops(self):
  106 + conn = schema.connect(self.db_path)
  107 + conn.execute(
  108 + "INSERT INTO deployment (deployment_id, git_commit_hash, previous_commit_hash, status, "
  109 + "previous_instance_dir, candidate_db_path) "
  110 + "VALUES ('dep-1', 'c1', 'c0', 'candidate_starting', '/prev', '/tmp/candidate.db')"
  111 + )
  112 + conn.commit()
  113 + conn.close()
  114 +
  115 + with self._mock_scheduler(), \
  116 + patch.object(deploy_stage, "_current_commit", return_value="deadbeef"), \
  117 + patch.object(deploy_stage, "stage_deployment", return_value={
  118 + "deployment_id": "dep-1", "status": "candidate_starting",
  119 + "candidate_port": 1, "candidate_pid": 2, "candidate_instance_dir": "/x",
  120 + }), \
  121 + patch.object(deploy_health_check, "run_health_check", return_value={
  122 + "status": "health_check_failed", "signals": [{"signal_type": "http_health", "status": "fail", "detail": "x"}],
  123 + }), \
  124 + patch.object(deploy_rollback, "rollback_path_a", return_value={"status": "rolled_back"}) as mock_rb_a, \
  125 + patch.object(deploy_cutover, "cutover") as mock_cutover:
  126 + result = self_hosting_e2e.run_self_hosting_closed_loop(self.db_path, "goal", target_folder_path=paths.repo_root())
  127 +
  128 + self.assertEqual(result, {"stage": "health_check", "outcome": "rolled_back_path_a", "deployment_id": "dep-1"})
  129 + mock_rb_a.assert_called_once()
  130 + mock_cutover.assert_not_called()
  131 +
  132 +
  133 +class MonitorReachesFinalizedTest(_RealDbTestBase):
  134 + def test_clean_monitoring_window_finalizes_deployment(self):
  135 + conn = schema.connect(self.db_path)
  136 + conn.execute(
  137 + "INSERT INTO deployment (deployment_id, git_commit_hash, previous_commit_hash, status, "
  138 + "previous_instance_dir, cutover_at) "
  139 + "VALUES ('dep-1', 'c1', 'c0', 'live', '/prev', '2020-01-01 00:00:00')"
  140 + )
  141 + conn.commit()
  142 + conn.close()
  143 +
  144 + with self._mock_scheduler(), \
  145 + patch.object(deploy_stage, "_current_commit", return_value="deadbeef"), \
  146 + patch.object(deploy_stage, "stage_deployment", return_value={
  147 + "deployment_id": "dep-1", "status": "candidate_starting",
  148 + "candidate_port": 1, "candidate_pid": 2, "candidate_instance_dir": "/x",
  149 + }), \
  150 + patch.object(deploy_health_check, "run_health_check", return_value={"status": "health_check_passed", "signals": []}), \
  151 + patch.object(deploy_cutover, "cutover", return_value={"status": "live", "live_pid": 3, "live_port": 9999}), \
  152 + patch.object(deploy_monitor, "compute_baseline_validator_approval_rate", return_value=None), \
  153 + patch.object(deploy_monitor, "run_monitor_cycle", return_value={
  154 + "status": "monitoring_window_clean", "latest_signal": {"signal_type": "process_alive", "status": "ok", "detail": None},
  155 + }) as mock_monitor, \
  156 + patch.object(deploy_rollback, "rollback_path_b") as mock_rb_b:
  157 + result = self_hosting_e2e.run_self_hosting_closed_loop(self.db_path, "goal", target_folder_path=paths.repo_root())
  158 +
  159 + self.assertEqual(result, {"stage": "monitor", "outcome": "finalized", "deployment_id": "dep-1"})
  160 + mock_monitor.assert_called_once()
  161 + mock_rb_b.assert_not_called()
  162 +
  163 + conn = schema.connect(self.db_path)
  164 + status = conn.execute("SELECT status FROM deployment WHERE deployment_id = 'dep-1'").fetchone()[0]
  165 + conn.close()
  166 + self.assertEqual(status, "finalized")
  167 +
  168 +
  169 +class MonitorDetectsAnomalyTest(_RealDbTestBase):
  170 + def test_anomaly_routes_to_rollback_path_b(self):
  171 + conn = schema.connect(self.db_path)
  172 + conn.execute(
  173 + "INSERT INTO deployment (deployment_id, git_commit_hash, previous_commit_hash, status, "
  174 + "previous_instance_dir, cutover_at) "
  175 + "VALUES ('dep-1', 'c1', 'c0', 'live', '/prev', '2020-01-01 00:00:00')"
  176 + )
  177 + conn.commit()
  178 + conn.close()
  179 +
  180 + with self._mock_scheduler(), \
  181 + patch.object(deploy_stage, "_current_commit", return_value="deadbeef"), \
  182 + patch.object(deploy_stage, "stage_deployment", return_value={
  183 + "deployment_id": "dep-1", "status": "candidate_starting",
  184 + "candidate_port": 1, "candidate_pid": 2, "candidate_instance_dir": "/x",
  185 + }), \
  186 + patch.object(deploy_health_check, "run_health_check", return_value={"status": "health_check_passed", "signals": []}), \
  187 + patch.object(deploy_cutover, "cutover", return_value={"status": "live", "live_pid": 3, "live_port": 9999}), \
  188 + patch.object(deploy_monitor, "compute_baseline_validator_approval_rate", return_value=None), \
  189 + patch.object(deploy_monitor, "run_monitor_cycle", return_value={
  190 + "status": "anomaly_detected", "latest_signal": {"signal_type": "http_health", "status": "fail", "detail": "x"},
  191 + }), \
  192 + patch.object(deploy_rollback, "rollback_path_b", return_value={"status": "rolled_back"}) as mock_rb_b:
  193 + result = self_hosting_e2e.run_self_hosting_closed_loop(self.db_path, "goal", target_folder_path=paths.repo_root())
  194 +
  195 + self.assertEqual(result, {"stage": "monitor", "outcome": "rolled_back_path_b", "deployment_id": "dep-1"})
  196 + mock_rb_b.assert_called_once()
  197 + args, kwargs = mock_rb_b.call_args
  198 + self.assertEqual(args[2], 3) # live_pid from cutover's own return value
  199 + self.assertEqual(args[3], "/prev") # previous_instance_dir threaded through correctly
  200 +
  201 +
  202 +class RealSelfHostingClosedLoopTest(unittest.TestCase):
  203 + """The literal acceptance scenario: self-development-feasibility.md's 9
  204 + steps, for real -- a real self-target Goal ("fix the loop_round
  205 + off-by-one boundary bug in p_org_task_lifecycle", the exact example
  206 + that document uses), real Planner/Coder/Validator `claude` calls, and
  207 + (if the Coder's fix genuinely lands on a core-engine file and Validator
  208 + PASSes) a REAL candidate cutover of a worktree checkout of THIS repo --
  209 + meaning a real subprocess bound to a real port, a real one-time DB
  210 + copy, and on success a real `git revert`-reversible commit or a real
  211 + rollback exercised against this actual repository's git history.
  212 +
  213 + Gated behind RUN_REAL_SELF_HOSTING_E2E=1 -- deliberately a DIFFERENT,
  214 + stricter env var than this project's usual RUN_REAL_AGENT_E2E (used by
  215 + every other real-`claude`-call test in this codebase), because this is
  216 + the single highest-blast-radius action in the whole project: unlike a
  217 + normal RUN_REAL_AGENT_E2E test (which costs real API usage but only
  218 + touches ephemeral temp directories/in-memory state), this one can
  219 + actually replace the currently-running graphlemon process and mutate
  220 + this real repository's commit history via `git revert`. Requiring a
  221 + SEPARATE, unambiguously-named env var means accidentally setting
  222 + RUN_REAL_AGENT_E2E=1 to run some other, cheaper real test in this
  223 + codebase can never accidentally trigger this one too.
  224 +
  225 + Not executed as part of implementing this Unit -- per the explicit
  226 + instruction this Unit was built under, flipping this switch requires
  227 + checking with the user first (AskUserQuestion), same as any other
  228 + action whose blast radius extends beyond this process/session. See
  229 + development-plan.md's U-042 note for that decision's own record.
  230 + """
  231 +
  232 + @unittest.skipUnless(
  233 + os.environ.get("RUN_REAL_SELF_HOSTING_E2E") == "1",
  234 + "set RUN_REAL_SELF_HOSTING_E2E=1 to run -- shells out to real `claude` calls AND performs a "
  235 + "real candidate cutover of this repository; costs real API usage and is genuinely destructive "
  236 + "if anything here has a bug. Requires explicit user sign-off, not just this env var, before "
  237 + "ever actually being run -- see development-plan.md's U-042 note.",
  238 + )
  239 + def test_real_self_development_feasibility_9_step_scenario(self):
  240 + import tempfile
  241 +
  242 + with tempfile.NamedTemporaryFile(suffix=".db", delete=False) as tmp:
  243 + db_path = tmp.name
  244 + try:
  245 + conn = schema.connect(db_path)
  246 + schema.create_schema(conn)
  247 + conn.execute(
  248 + "INSERT INTO project (project_id, name, target_folder_path, has_git_repo, status) "
  249 + "VALUES ('proj-1', 'graphlemon-self', ?, 1, 'active')", (paths.repo_root(),)
  250 + )
  251 + conn.commit()
  252 + conn.close()
  253 +
  254 + result = self_hosting_e2e.run_self_hosting_closed_loop(
  255 + db_path,
  256 + "Fix the loop_round off-by-one boundary bug in p_org_task_lifecycle: "
  257 + "loop_round > 3 should be loop_round >= 3 for the blocked_pending_human transition.",
  258 + target_folder_path=paths.repo_root(),
  259 + monitor_timeout_cycles=5,
  260 + )
  261 +
  262 + self.assertIn(result["stage"], ("health_check", "cutover", "monitor"))
  263 + self.assertIn(
  264 + result["outcome"],
  265 + ("finalized", "rolled_back_path_a", "rolled_back_path_b", "timed_out_still_monitoring"),
  266 + )
  267 + finally:
  268 + if os.path.exists(db_path):
  269 + os.remove(db_path)
  270 +
  271 +
  272 +if __name__ == "__main__":
  273 + unittest.main()