Commit 05292668f4f37a372d5c822d0c704603cba2f35b
1 parent
4f4b1853
U-042: self-hosting closed-loop acceptance (orchestration + gated real test)
Adds supervisor/self_hosting_e2e.py: run_self_hosting_closed_loop() chains every PH5 module built this session (U-034~U-041) with the Graph Engine's own scheduler (U-032) into one real, callable path matching self-development- feasibility.md's 9-step scenario -- self-target Goal -> Planner/Coder/ Validator -> core_engine classification -> stage -> health check -> cutover (or rollback path A if health check fails) -> monitor until either monitoring_window_clean (finalized) or anomaly_detected (rollback path B). Both real outcome paths are branches of the same call, not two separate scripted scenarios. Explicitly documented as this acceptance scenario's own orchestration glue, not a claim about Deploy Supervisor's eventual production poll loop (U-036's discovery-loop hook is still unfilled; building that is future work). Mocked wiring tests (6, run in normal regression) stub each already independently-tested module boundary and only verify the branching logic. The real, literal acceptance test is built and ready (RealSelfHostingClosedLoopTest, reproduces the document's own "loop_round > 3 should be >= 3" example) but gated behind RUN_REAL_SELF_HOSTING_E2E=1 -- deliberately a separate, distinctly-named env var from this project's usual RUN_REAL_AGENT_E2E, since this is the highest-blast-radius action in the project (can replace the running graphlemon process and git-revert commits on this real repo). Per this Unit's own explicit build instruction, not executed as part of implementing it -- doing so requires checking with the user first. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Showing
4 changed files
with
410 additions
and
1 deletions
docs/development-plan.md
| @@ -396,6 +396,10 @@ PH2-S1/S2/S3/S4 之间除标注的依赖外相互独立,可并行推进。 | @@ -396,6 +396,10 @@ PH2-S1/S2/S3/S4 之间除标注的依赖外相互独立,可并行推进。 | ||
| 396 | - **U-042 [Risk Gate]** 自举闭环端到端验收:真实复现 `self-development-feasibility.md` 第 3 节的 9 步场景(修复 graphlemon 自己 `p_org_task_lifecycle` 状态机里的一个真实植入 bug),走完整"部署→健康检查→cutover→监控→finalize"或"检测异常→自动回滚"两条路径中至少一条 | 396 | - **U-042 [Risk Gate]** 自举闭环端到端验收:真实复现 `self-development-feasibility.md` 第 3 节的 9 步场景(修复 graphlemon 自己 `p_org_task_lifecycle` 状态机里的一个真实植入 bug),走完整"部署→健康检查→cutover→监控→finalize"或"检测异常→自动回滚"两条路径中至少一条 |
| 397 | - 依赖:U-035, U-041 | 397 | - 依赖:U-035, U-041 |
| 398 | - Done:全程无需人工评审代码内容;参照 graphloop `RUN_REAL_AGENT_E2E=1` 的先例,真实拉起两个 graphlemon 实例的端到端用例默认跳过,需要显式环境变量开启,日常回归只跑 mock 版本 | 398 | - Done:全程无需人工评审代码内容;参照 graphloop `RUN_REAL_AGENT_E2E=1` 的先例,真实拉起两个 graphlemon 实例的端到端用例默认跳过,需要显式环境变量开启,日常回归只跑 mock 版本 |
| 399 | + - **实现补充(2026-08-02)**:新增 `supervisor/self_hosting_e2e.py`:`run_self_hosting_closed_loop()` 把这次会话建好的全部 PH5 模块(U-034~U-041)+ Graph Engine 自己的 `scheduler.run_goal_to_completion`(U-032)串成一条真实可调用的链路——真实 self-target Goal → Planner/Coder/Validator(真实 `claude` 调用,`scheduler` 内部已经走 U-035 的 self-target 强制 worktree)→ PASS 后走 U-035 记的 `core_engine` 审计 `bug_record` → `deploy_stage.stage_deployment` → `deploy_health_check.run_health_check` → 通过则 `deploy_cutover.cutover`,不通过直接 `deploy_rollback.rollback_path_a` 停在这一步 → 循环 `deploy_monitor.run_monitor_cycle` 直到 `monitoring_window_clean`(`finalized`)或 `anomaly_detected`(`deploy_rollback.rollback_path_b`)——**这正是本 Unit 要求的"两条路径至少走一条",实现上两条路径都是同一次真实调用可能走到的真实分支,不是分别写死两个脚本**。**明确不是**生产级 Deploy Supervisor 的最终 `poll_fn` 实现——真正独立轮询"有没有新的 self-target 合并等待部署"的发现机制是 U-036 留的钩子,本 Unit 只提供验收场景自己的编排入口,不代表以后正式的常驻发现循环长这样,避免把验收脚本和生产实现搞混。 | ||
| 400 | + - **Mock 版本(日常回归)**:`test_self_hosting_e2e.py` 前 6 个测试类在每个已经独立测试过真实机制的模块边界上打桩(`scheduler.run_goal_to_completion`/`deploy_stage.stage_deployment`/`deploy_health_check.run_health_check`/`deploy_cutover.cutover`/`deploy_monitor.run_monitor_cycle`/`deploy_rollback.rollback_path_a`/`rollback_path_b`),只验证编排本身的分支/顺序逻辑对不对(任务没到 done 提前返回、不是 core_engine 提前返回、健康检查失败走 A 不碰 cutover、监控干净走 finalized、监控异常走 B),不重新证明任何一个模块自己的真实机制——那些已经在各自的测试文件里用真实进程/真实 git 仓库验证过了。 | ||
| 401 | + - **真实版本,已建好但没有执行**:`RealSelfHostingClosedLoopTest`,字面复现 `self-development-feasibility.md` 例子本身("loop_round > 3 应该是 loop_round >= 3")。**用了一个和项目一贯的 `RUN_REAL_AGENT_E2E` 不同、单独命名的环境变量 `RUN_REAL_SELF_HOSTING_E2E`**——这是本项目目前为止影响面最大的一个动作,不只是花真钱调用 `claude`(那是 `RUN_REAL_AGENT_E2E` 覆盖的常规风险),而是真的可能替换掉正在跑的 graphlemon 进程本身、真的会在这个仓库自己的 git 历史上跑 `git revert`——用不同名字的环境变量,是为了不让"为了跑别的真实测试而设的 `RUN_REAL_AGENT_E2E=1`"意外带着触发这个。**没有替用户执行这个测试**:按本 Unit 开工前收到的明确指示,这类"影响范围超出本进程/本会话"的动作,写好、gate 好就停,真正执行前需要用 AskUserQuestion 问过用户——这次实现结束后立刻会问。 | ||
| 402 | + - 覆盖:`self_hosting_e2e.py`(编排层,串联 U-032/034~041 全部真实模块);真实验收场景已建好、默认不跑 | ||
| 399 | 403 | ||
| 400 | ### PH5-S4 可观测性(非阻塞,可延后) | 404 | ### PH5-S4 可观测性(非阻塞,可延后) |
| 401 | 405 |
docs/sdd/unit_matrix.json
| @@ -59,7 +59,7 @@ | @@ -59,7 +59,7 @@ | ||
| 59 | {"id": "U-039", "phase": "PH5", "sub_phase": "PH5-S2", "name": "c_deploy_supervisor_cutover(见 development-plan.md 2026-08-02 实现补充)", "classification": "Risk Gate", "depends_on": ["U-038"], "coverage": {"model": ["deployment"], "view": [], "controller": ["c_deploy_supervisor_cutover"], "process": [], "requirement": []}, "done": "健康检查通过后完成真实切换(supervisor/deploy_cutover.py,真实进程停止+重启验证过);在途 work_node_execution 重启后可恢复(复用既有 REQ-06 机制);previous_instance_dir 保留至监控窗口结束;新增 LIVE_PORT=8420 常量(05 §5.9 缺的一个字段);真实测试发现并修复 deploy_supervisor.is_alive() 的僵尸进程判活缺陷(新增 reap_if_exited())"}, | 59 | {"id": "U-039", "phase": "PH5", "sub_phase": "PH5-S2", "name": "c_deploy_supervisor_cutover(见 development-plan.md 2026-08-02 实现补充)", "classification": "Risk Gate", "depends_on": ["U-038"], "coverage": {"model": ["deployment"], "view": [], "controller": ["c_deploy_supervisor_cutover"], "process": [], "requirement": []}, "done": "健康检查通过后完成真实切换(supervisor/deploy_cutover.py,真实进程停止+重启验证过);在途 work_node_execution 重启后可恢复(复用既有 REQ-06 机制);previous_instance_dir 保留至监控窗口结束;新增 LIVE_PORT=8420 常量(05 §5.9 缺的一个字段);真实测试发现并修复 deploy_supervisor.is_alive() 的僵尸进程判活缺陷(新增 reap_if_exited())"}, |
| 60 | {"id": "U-040", "phase": "PH5", "sub_phase": "PH5-S3", "name": "c_deploy_supervisor_monitor(见 development-plan.md 2026-08-02 实现补充)", "classification": "Risk Gate", "depends_on": ["U-039"], "coverage": {"model": ["deploy_health_signal"], "view": [], "controller": ["c_deploy_supervisor_monitor"], "process": [], "requirement": []}, "done": "四类信号采样正常(supervisor/deploy_monitor.py);validator_approval_rate 偏离基线阈值时触发 anomaly_detected;approve 占比算法基于对 scheduler.finish_task_lifecycle 真实代码逻辑的核实(PASS 从不产生 bug_record,FAIL 必然产生),非字面 SDD 定义"}, | 60 | {"id": "U-040", "phase": "PH5", "sub_phase": "PH5-S3", "name": "c_deploy_supervisor_monitor(见 development-plan.md 2026-08-02 实现补充)", "classification": "Risk Gate", "depends_on": ["U-039"], "coverage": {"model": ["deploy_health_signal"], "view": [], "controller": ["c_deploy_supervisor_monitor"], "process": [], "requirement": []}, "done": "四类信号采样正常(supervisor/deploy_monitor.py);validator_approval_rate 偏离基线阈值时触发 anomaly_detected;approve 占比算法基于对 scheduler.finish_task_lifecycle 真实代码逻辑的核实(PASS 从不产生 bug_record,FAIL 必然产生),非字面 SDD 定义"}, |
| 61 | {"id": "U-041", "phase": "PH5", "sub_phase": "PH5-S3", "name": "c_deploy_supervisor_rollback(见 development-plan.md 2026-08-02 实现补充)", "classification": "Risk Gate", "depends_on": ["U-038", "U-040"], "coverage": {"model": ["deployment"], "view": [], "controller": ["c_deploy_supervisor_rollback"], "process": [], "requirement": []}, "done": "回滚路径 A/B 各跑通一次(supervisor/deploy_rollback.py,真实进程+真实 git 仓库验证);旧版本也起不来时正确终态到 blocked_pending_human(真实场景验证,非假设);git revert 冲突有明确处理(真实构造冲突历史验证过,含 --abort 清理)"}, | 61 | {"id": "U-041", "phase": "PH5", "sub_phase": "PH5-S3", "name": "c_deploy_supervisor_rollback(见 development-plan.md 2026-08-02 实现补充)", "classification": "Risk Gate", "depends_on": ["U-038", "U-040"], "coverage": {"model": ["deployment"], "view": [], "controller": ["c_deploy_supervisor_rollback"], "process": [], "requirement": []}, "done": "回滚路径 A/B 各跑通一次(supervisor/deploy_rollback.py,真实进程+真实 git 仓库验证);旧版本也起不来时正确终态到 blocked_pending_human(真实场景验证,非假设);git revert 冲突有明确处理(真实构造冲突历史验证过,含 --abort 清理)"}, |
| 62 | - {"id": "U-042", "phase": "PH5", "sub_phase": "PH5-S3", "name": "自举闭环端到端验收", "classification": "Risk Gate", "depends_on": ["U-035", "U-041"], "coverage": {"model": [], "view": [], "controller": [], "process": [], "requirement": []}, "done": "真实复现 self-development-feasibility.md 9 步场景,全程无需人工评审代码内容;真实双实例 E2E 用例默认跳过,需显式环境变量开启"}, | 62 | + {"id": "U-042", "phase": "PH5", "sub_phase": "PH5-S3", "name": "自举闭环端到端验收(见 development-plan.md 2026-08-02 实现补充)", "classification": "Risk Gate", "depends_on": ["U-035", "U-041"], "coverage": {"model": [], "view": [], "controller": [], "process": [], "requirement": []}, "done": "真实复现 self-development-feasibility.md 9 步场景,全程无需人工评审代码内容(supervisor/self_hosting_e2e.py 编排层已建好);真实双实例 E2E 用例(RUN_REAL_SELF_HOSTING_E2E=1,独立于常规 RUN_REAL_AGENT_E2E 的单独环境变量)默认跳过,需显式环境变量开启且需额外用户确认才真正执行;日常回归只跑 mock 版本(6 个编排逻辑测试,已跑通)"}, |
| 63 | {"id": "U-043", "phase": "PH5", "sub_phase": "PH5-S4", "name": "控制台展示 deployment 状态与健康信号", "classification": "Value Path", "depends_on": ["U-024", "U-040"], "coverage": {"model": [], "view": ["v_deployment_status"], "controller": [], "process": [], "requirement": []}, "done": "能看到当前 deployment 状态机所处状态与最近一次健康信号采样结果"} | 63 | {"id": "U-043", "phase": "PH5", "sub_phase": "PH5-S4", "name": "控制台展示 deployment 状态与健康信号", "classification": "Value Path", "depends_on": ["U-024", "U-040"], "coverage": {"model": [], "view": ["v_deployment_status"], "controller": [], "process": [], "requirement": []}, "done": "能看到当前 deployment 状态机所处状态与最近一次健康信号采样结果"} |
| 64 | ], | 64 | ], |
| 65 | "classification_summary": { | 65 | "classification_summary": { |
supervisor/self_hosting_e2e.py
0 → 100644
| 1 | +"""U-042: self-hosting closed-loop acceptance -- wires every PH5 module | ||
| 2 | +built so far (U-034~U-041) plus the Graph Engine's own scheduler into the | ||
| 3 | +literal 9-step scenario `self-development-feasibility.md` walks through: | ||
| 4 | +a real self-target Goal, Planner->Coder->Validator (real `claude` calls via | ||
| 5 | +`scheduler.run_goal_to_completion`, which already forces worktree isolation | ||
| 6 | +and records the `core_engine` audit trail for self-target tasks per U-035), | ||
| 7 | +merge, stage a candidate, health-check it, cut it over, monitor it, and | ||
| 8 | +land on EITHER `finalized` (clean monitoring window) OR a rollback (anomaly | ||
| 9 | +detected) -- "走完整...两条路径中至少一条", not a scripted single outcome. | ||
| 10 | + | ||
| 11 | +This module is the acceptance scenario's own orchestration glue, not a | ||
| 12 | +claim about what Deploy Supervisor's real production `poll_fn` (U-036's | ||
| 13 | +still-unfilled hook) should look like long-term -- a continuously-running | ||
| 14 | +daemon would discover self-target merges to stage on its own schedule | ||
| 15 | +rather than drive one linear call end to end. Building that discovery loop | ||
| 16 | +is future work, not this Unit's job; this function exists so the acceptance | ||
| 17 | +scenario has one real, callable entry point to run (or NOT run -- see | ||
| 18 | +test_self_hosting_e2e.py's gating) against. | ||
| 19 | +""" | ||
| 20 | + | ||
| 21 | +import time | ||
| 22 | + | ||
| 23 | +from supervisor import deploy_cutover, deploy_health_check, deploy_monitor, deploy_rollback, deploy_stage | ||
| 24 | +from src import paths, schema, scheduler | ||
| 25 | + | ||
| 26 | + | ||
| 27 | +class SelfHostingLoopError(Exception): | ||
| 28 | + """Raised when a precondition for even attempting the scenario isn't | ||
| 29 | + met (e.g. target_folder_path isn't genuinely self-target) -- distinct | ||
| 30 | + from a real outcome like 'rolled_back', which is a valid, successful | ||
| 31 | + exercise of the safety mechanism, not an error.""" | ||
| 32 | + | ||
| 33 | + | ||
| 34 | +def run_self_hosting_closed_loop( | ||
| 35 | + db_path, goal_text, target_folder_path=None, prod_db_path=None, | ||
| 36 | + poll_interval=5.0, monitor_timeout_cycles=None, sleep=time.sleep, now=time.time, | ||
| 37 | +): | ||
| 38 | + """Returns {"stage", "outcome", ...}. `stage` names how far the scenario | ||
| 39 | + got (`task_lifecycle` / `classification` / `health_check` / `cutover` / | ||
| 40 | + `monitor`); `outcome` is the concrete result at that stage. A task that | ||
| 41 | + doesn't reach 'done', or doesn't get classified core_engine, is a valid | ||
| 42 | + early-return (not every self-target task is a core-engine change -- see | ||
| 43 | + U-035), not a scenario failure. | ||
| 44 | + """ | ||
| 45 | + target_folder_path = target_folder_path or paths.repo_root() | ||
| 46 | + prod_db_path = prod_db_path or paths.db_path() | ||
| 47 | + if not paths.is_self_target(target_folder_path): | ||
| 48 | + raise SelfHostingLoopError( | ||
| 49 | + "target_folder_path {!r} is not this instance's own checkout ({!r}) -- " | ||
| 50 | + "the self-hosting scenario requires self-target".format(target_folder_path, paths.repo_root()) | ||
| 51 | + ) | ||
| 52 | + | ||
| 53 | + task_result = scheduler.run_goal_to_completion(db_path, target_folder_path, goal_text) | ||
| 54 | + if task_result.get("final_status") != "done": | ||
| 55 | + return {"stage": "task_lifecycle", "outcome": task_result.get("final_status"), "task_id": task_result.get("task_id")} | ||
| 56 | + | ||
| 57 | + conn = schema.connect(db_path) | ||
| 58 | + try: | ||
| 59 | + bug = conn.execute( | ||
| 60 | + "SELECT bug_id FROM bug_record WHERE task_id = ? AND root_cause_category = 'core_engine' " | ||
| 61 | + "ORDER BY rowid DESC LIMIT 1", | ||
| 62 | + (task_result["task_id"],), | ||
| 63 | + ).fetchone() | ||
| 64 | + if bug is None: | ||
| 65 | + return {"stage": "classification", "outcome": "not_core_engine", "task_id": task_result["task_id"]} | ||
| 66 | + | ||
| 67 | + merge_commit = deploy_stage._current_commit(target_folder_path) | ||
| 68 | + | ||
| 69 | + staged = deploy_stage.stage_deployment( | ||
| 70 | + conn, merge_commit, repo_root=target_folder_path, prod_db_path=prod_db_path | ||
| 71 | + ) | ||
| 72 | + health = deploy_health_check.run_health_check( | ||
| 73 | + conn, staged["deployment_id"], staged["candidate_pid"], staged["candidate_port"] | ||
| 74 | + ) | ||
| 75 | + if health["status"] != "health_check_passed": | ||
| 76 | + candidate_db_path = conn.execute( | ||
| 77 | + "SELECT candidate_db_path FROM deployment WHERE deployment_id = ?", (staged["deployment_id"],) | ||
| 78 | + ).fetchone()[0] | ||
| 79 | + deploy_rollback.rollback_path_a( | ||
| 80 | + conn, staged["deployment_id"], staged["candidate_pid"], candidate_db_path, | ||
| 81 | + merge_commit, repo_root=target_folder_path, | ||
| 82 | + description="health check failed: {}".format(health["signals"]), | ||
| 83 | + ) | ||
| 84 | + return {"stage": "health_check", "outcome": "rolled_back_path_a", "deployment_id": staged["deployment_id"]} | ||
| 85 | + | ||
| 86 | + cutover_result = deploy_cutover.cutover( | ||
| 87 | + conn, staged["deployment_id"], live_pid=None, candidate_pid=staged["candidate_pid"], | ||
| 88 | + candidate_instance_dir=staged["candidate_instance_dir"], prod_db_path=prod_db_path, | ||
| 89 | + ) | ||
| 90 | + | ||
| 91 | + project_id = conn.execute( | ||
| 92 | + "SELECT project_id FROM project WHERE target_folder_path = ?", (target_folder_path,) | ||
| 93 | + ).fetchone()[0] | ||
| 94 | + cutover_at = conn.execute( | ||
| 95 | + "SELECT cutover_at FROM deployment WHERE deployment_id = ?", (staged["deployment_id"],) | ||
| 96 | + ).fetchone()[0] | ||
| 97 | + baseline = deploy_monitor.compute_baseline_validator_approval_rate(conn, project_id, cutover_at) | ||
| 98 | + baseline_rate = baseline[0] if baseline else None | ||
| 99 | + | ||
| 100 | + cycles = 0 | ||
| 101 | + while True: | ||
| 102 | + cycles += 1 | ||
| 103 | + cycle_result = deploy_monitor.run_monitor_cycle( | ||
| 104 | + conn, staged["deployment_id"], cutover_result["live_pid"], cutover_result["live_port"], | ||
| 105 | + project_id, baseline_rate, cutover_at, now=now, | ||
| 106 | + ) | ||
| 107 | + if cycle_result["status"] == "monitoring_window_clean": | ||
| 108 | + conn.execute( | ||
| 109 | + "UPDATE deployment SET status = 'finalized' WHERE deployment_id = ?", (staged["deployment_id"],) | ||
| 110 | + ) | ||
| 111 | + conn.commit() | ||
| 112 | + return {"stage": "monitor", "outcome": "finalized", "deployment_id": staged["deployment_id"]} | ||
| 113 | + if cycle_result["status"] == "anomaly_detected": | ||
| 114 | + rb = deploy_rollback.rollback_path_b( | ||
| 115 | + conn, staged["deployment_id"], cutover_result["live_pid"], | ||
| 116 | + _previous_instance_dir(conn, staged["deployment_id"]), | ||
| 117 | + prod_db_path, merge_commit, repo_root=target_folder_path, | ||
| 118 | + description="post-cutover anomaly: {}".format(cycle_result["latest_signal"]), | ||
| 119 | + ) | ||
| 120 | + outcome = "rolled_back_path_b" if rb["status"] == "rolled_back" else "rollback_failed" | ||
| 121 | + return {"stage": "monitor", "outcome": outcome, "deployment_id": staged["deployment_id"]} | ||
| 122 | + if monitor_timeout_cycles is not None and cycles >= monitor_timeout_cycles: | ||
| 123 | + return {"stage": "monitor", "outcome": "timed_out_still_monitoring", "deployment_id": staged["deployment_id"]} | ||
| 124 | + sleep(poll_interval) | ||
| 125 | + finally: | ||
| 126 | + conn.close() | ||
| 127 | + | ||
| 128 | + | ||
| 129 | +def _previous_instance_dir(conn, deployment_id): | ||
| 130 | + return conn.execute( | ||
| 131 | + "SELECT previous_instance_dir FROM deployment WHERE deployment_id = ?", (deployment_id,) | ||
| 132 | + ).fetchone()[0] |
supervisor/test_self_hosting_e2e.py
0 → 100644
| 1 | +"""U-042 tests: self_hosting_e2e.py's orchestration wiring, mocked at each | ||
| 2 | +already-independently-tested module's own boundary (deploy_stage/ | ||
| 3 | +deploy_health_check/deploy_cutover/deploy_monitor/deploy_rollback/scheduler | ||
| 4 | +each have their own real-mechanics test files -- this file's job is proving | ||
| 5 | +run_self_hosting_closed_loop() sequences and branches between them | ||
| 6 | +correctly, not re-proving any of their own internals). | ||
| 7 | + | ||
| 8 | +The genuinely real, un-mocked acceptance run (self-development-feasibility.md's | ||
| 9 | +literal 9-step scenario, actual `claude` calls, an actual candidate cutover | ||
| 10 | +of THIS repo) lives in RealSelfHostingClosedLoopTest below, gated behind | ||
| 11 | +RUN_REAL_SELF_HOSTING_E2E=1 -- a SEPARATE, stricter env var from this | ||
| 12 | +project's usual RUN_REAL_AGENT_E2E, since this is the single highest-blast- | ||
| 13 | +radius action in the whole project (it can actually replace the running | ||
| 14 | +graphlemon process and git-revert commits on this real repo). Not run as | ||
| 15 | +part of this Unit's own implementation -- see development-plan.md's U-042 | ||
| 16 | +note for why. | ||
| 17 | + | ||
| 18 | +Run: python3 -m unittest discover -s supervisor -p "test_*.py" -v | ||
| 19 | +""" | ||
| 20 | + | ||
| 21 | +import os | ||
| 22 | +import sys | ||
| 23 | +import unittest | ||
| 24 | +from unittest.mock import patch | ||
| 25 | + | ||
| 26 | +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) | ||
| 27 | + | ||
| 28 | +from supervisor import ( | ||
| 29 | + deploy_cutover, deploy_health_check, deploy_monitor, deploy_rollback, deploy_stage, self_hosting_e2e, | ||
| 30 | +) | ||
| 31 | +from src import paths, schema, scheduler | ||
| 32 | + | ||
| 33 | + | ||
| 34 | +class SelfTargetGuardTest(unittest.TestCase): | ||
| 35 | + def test_non_self_target_raises_before_doing_anything(self): | ||
| 36 | + with self.assertRaises(self_hosting_e2e.SelfHostingLoopError): | ||
| 37 | + self_hosting_e2e.run_self_hosting_closed_loop("db.sqlite", "goal", target_folder_path="/tmp/not-graphlemon") | ||
| 38 | + | ||
| 39 | + | ||
| 40 | +class TaskLifecycleNotDoneTest(unittest.TestCase): | ||
| 41 | + def test_returns_early_without_touching_deploy_supervisor(self): | ||
| 42 | + with patch.object(scheduler, "run_goal_to_completion", return_value={"task_id": "t1", "final_status": "blocked_pending_human"}), \ | ||
| 43 | + patch.object(deploy_stage, "stage_deployment") as mock_stage: | ||
| 44 | + result = self_hosting_e2e.run_self_hosting_closed_loop("unused.db", "goal", target_folder_path=paths.repo_root()) | ||
| 45 | + self.assertEqual(result, {"stage": "task_lifecycle", "outcome": "blocked_pending_human", "task_id": "t1"}) | ||
| 46 | + mock_stage.assert_not_called() | ||
| 47 | + | ||
| 48 | + | ||
| 49 | +class NotCoreEngineTest(unittest.TestCase): | ||
| 50 | + def test_returns_early_when_no_core_engine_audit_bug_exists(self): | ||
| 51 | + import tempfile | ||
| 52 | + with tempfile.NamedTemporaryFile(suffix=".db", delete=False) as tmp: | ||
| 53 | + db_path = tmp.name | ||
| 54 | + conn = schema.connect(db_path) | ||
| 55 | + schema.create_schema(conn) | ||
| 56 | + conn.execute( | ||
| 57 | + "INSERT INTO project (project_id, name, target_folder_path, has_git_repo, status) " | ||
| 58 | + "VALUES ('proj-1', 'P', ?, 1, 'active')", (paths.repo_root(),) | ||
| 59 | + ) | ||
| 60 | + conn.commit() | ||
| 61 | + conn.close() | ||
| 62 | + try: | ||
| 63 | + with patch.object(scheduler, "run_goal_to_completion", return_value={"task_id": "t1", "final_status": "done"}), \ | ||
| 64 | + patch.object(deploy_stage, "stage_deployment") as mock_stage: | ||
| 65 | + result = self_hosting_e2e.run_self_hosting_closed_loop(db_path, "goal", target_folder_path=paths.repo_root()) | ||
| 66 | + self.assertEqual(result, {"stage": "classification", "outcome": "not_core_engine", "task_id": "t1"}) | ||
| 67 | + mock_stage.assert_not_called() | ||
| 68 | + finally: | ||
| 69 | + os.remove(db_path) | ||
| 70 | + | ||
| 71 | + | ||
| 72 | +class _RealDbTestBase(unittest.TestCase): | ||
| 73 | + def setUp(self): | ||
| 74 | + import tempfile | ||
| 75 | + self.tmp = tempfile.NamedTemporaryFile(suffix=".db", delete=False) | ||
| 76 | + self.tmp.close() | ||
| 77 | + self.db_path = self.tmp.name | ||
| 78 | + self.addCleanup(lambda: os.path.exists(self.db_path) and os.remove(self.db_path)) | ||
| 79 | + | ||
| 80 | + conn = schema.connect(self.db_path) | ||
| 81 | + schema.create_schema(conn) | ||
| 82 | + conn.execute( | ||
| 83 | + "INSERT INTO project (project_id, name, target_folder_path, has_git_repo, status) " | ||
| 84 | + "VALUES ('proj-1', 'P', ?, 1, 'active')", (paths.repo_root(),) | ||
| 85 | + ) | ||
| 86 | + conn.execute( | ||
| 87 | + "INSERT INTO task (task_id, project_id, goal_text, task_type, status, assigned_org_node) " | ||
| 88 | + "VALUES ('task-1', 'proj-1', 'goal', 'goal', 'done', 'validator')" | ||
| 89 | + ) | ||
| 90 | + conn.execute( | ||
| 91 | + "INSERT INTO bug_record (bug_id, task_id, source, description, root_cause_category, status) " | ||
| 92 | + "VALUES ('bug-1', 'task-1', 'validator', 'self-target change', 'core_engine', 'resolved')" | ||
| 93 | + ) | ||
| 94 | + conn.commit() | ||
| 95 | + conn.close() | ||
| 96 | + | ||
| 97 | + def _mock_scheduler(self): | ||
| 98 | + return patch.object( | ||
| 99 | + scheduler, "run_goal_to_completion", | ||
| 100 | + return_value={"task_id": "task-1", "final_status": "done"}, | ||
| 101 | + ) | ||
| 102 | + | ||
| 103 | + | ||
| 104 | +class HealthCheckFailedTest(_RealDbTestBase): | ||
| 105 | + def test_routes_to_rollback_path_a_and_stops(self): | ||
| 106 | + conn = schema.connect(self.db_path) | ||
| 107 | + conn.execute( | ||
| 108 | + "INSERT INTO deployment (deployment_id, git_commit_hash, previous_commit_hash, status, " | ||
| 109 | + "previous_instance_dir, candidate_db_path) " | ||
| 110 | + "VALUES ('dep-1', 'c1', 'c0', 'candidate_starting', '/prev', '/tmp/candidate.db')" | ||
| 111 | + ) | ||
| 112 | + conn.commit() | ||
| 113 | + conn.close() | ||
| 114 | + | ||
| 115 | + with self._mock_scheduler(), \ | ||
| 116 | + patch.object(deploy_stage, "_current_commit", return_value="deadbeef"), \ | ||
| 117 | + patch.object(deploy_stage, "stage_deployment", return_value={ | ||
| 118 | + "deployment_id": "dep-1", "status": "candidate_starting", | ||
| 119 | + "candidate_port": 1, "candidate_pid": 2, "candidate_instance_dir": "/x", | ||
| 120 | + }), \ | ||
| 121 | + patch.object(deploy_health_check, "run_health_check", return_value={ | ||
| 122 | + "status": "health_check_failed", "signals": [{"signal_type": "http_health", "status": "fail", "detail": "x"}], | ||
| 123 | + }), \ | ||
| 124 | + patch.object(deploy_rollback, "rollback_path_a", return_value={"status": "rolled_back"}) as mock_rb_a, \ | ||
| 125 | + patch.object(deploy_cutover, "cutover") as mock_cutover: | ||
| 126 | + result = self_hosting_e2e.run_self_hosting_closed_loop(self.db_path, "goal", target_folder_path=paths.repo_root()) | ||
| 127 | + | ||
| 128 | + self.assertEqual(result, {"stage": "health_check", "outcome": "rolled_back_path_a", "deployment_id": "dep-1"}) | ||
| 129 | + mock_rb_a.assert_called_once() | ||
| 130 | + mock_cutover.assert_not_called() | ||
| 131 | + | ||
| 132 | + | ||
| 133 | +class MonitorReachesFinalizedTest(_RealDbTestBase): | ||
| 134 | + def test_clean_monitoring_window_finalizes_deployment(self): | ||
| 135 | + conn = schema.connect(self.db_path) | ||
| 136 | + conn.execute( | ||
| 137 | + "INSERT INTO deployment (deployment_id, git_commit_hash, previous_commit_hash, status, " | ||
| 138 | + "previous_instance_dir, cutover_at) " | ||
| 139 | + "VALUES ('dep-1', 'c1', 'c0', 'live', '/prev', '2020-01-01 00:00:00')" | ||
| 140 | + ) | ||
| 141 | + conn.commit() | ||
| 142 | + conn.close() | ||
| 143 | + | ||
| 144 | + with self._mock_scheduler(), \ | ||
| 145 | + patch.object(deploy_stage, "_current_commit", return_value="deadbeef"), \ | ||
| 146 | + patch.object(deploy_stage, "stage_deployment", return_value={ | ||
| 147 | + "deployment_id": "dep-1", "status": "candidate_starting", | ||
| 148 | + "candidate_port": 1, "candidate_pid": 2, "candidate_instance_dir": "/x", | ||
| 149 | + }), \ | ||
| 150 | + patch.object(deploy_health_check, "run_health_check", return_value={"status": "health_check_passed", "signals": []}), \ | ||
| 151 | + patch.object(deploy_cutover, "cutover", return_value={"status": "live", "live_pid": 3, "live_port": 9999}), \ | ||
| 152 | + patch.object(deploy_monitor, "compute_baseline_validator_approval_rate", return_value=None), \ | ||
| 153 | + patch.object(deploy_monitor, "run_monitor_cycle", return_value={ | ||
| 154 | + "status": "monitoring_window_clean", "latest_signal": {"signal_type": "process_alive", "status": "ok", "detail": None}, | ||
| 155 | + }) as mock_monitor, \ | ||
| 156 | + patch.object(deploy_rollback, "rollback_path_b") as mock_rb_b: | ||
| 157 | + result = self_hosting_e2e.run_self_hosting_closed_loop(self.db_path, "goal", target_folder_path=paths.repo_root()) | ||
| 158 | + | ||
| 159 | + self.assertEqual(result, {"stage": "monitor", "outcome": "finalized", "deployment_id": "dep-1"}) | ||
| 160 | + mock_monitor.assert_called_once() | ||
| 161 | + mock_rb_b.assert_not_called() | ||
| 162 | + | ||
| 163 | + conn = schema.connect(self.db_path) | ||
| 164 | + status = conn.execute("SELECT status FROM deployment WHERE deployment_id = 'dep-1'").fetchone()[0] | ||
| 165 | + conn.close() | ||
| 166 | + self.assertEqual(status, "finalized") | ||
| 167 | + | ||
| 168 | + | ||
| 169 | +class MonitorDetectsAnomalyTest(_RealDbTestBase): | ||
| 170 | + def test_anomaly_routes_to_rollback_path_b(self): | ||
| 171 | + conn = schema.connect(self.db_path) | ||
| 172 | + conn.execute( | ||
| 173 | + "INSERT INTO deployment (deployment_id, git_commit_hash, previous_commit_hash, status, " | ||
| 174 | + "previous_instance_dir, cutover_at) " | ||
| 175 | + "VALUES ('dep-1', 'c1', 'c0', 'live', '/prev', '2020-01-01 00:00:00')" | ||
| 176 | + ) | ||
| 177 | + conn.commit() | ||
| 178 | + conn.close() | ||
| 179 | + | ||
| 180 | + with self._mock_scheduler(), \ | ||
| 181 | + patch.object(deploy_stage, "_current_commit", return_value="deadbeef"), \ | ||
| 182 | + patch.object(deploy_stage, "stage_deployment", return_value={ | ||
| 183 | + "deployment_id": "dep-1", "status": "candidate_starting", | ||
| 184 | + "candidate_port": 1, "candidate_pid": 2, "candidate_instance_dir": "/x", | ||
| 185 | + }), \ | ||
| 186 | + patch.object(deploy_health_check, "run_health_check", return_value={"status": "health_check_passed", "signals": []}), \ | ||
| 187 | + patch.object(deploy_cutover, "cutover", return_value={"status": "live", "live_pid": 3, "live_port": 9999}), \ | ||
| 188 | + patch.object(deploy_monitor, "compute_baseline_validator_approval_rate", return_value=None), \ | ||
| 189 | + patch.object(deploy_monitor, "run_monitor_cycle", return_value={ | ||
| 190 | + "status": "anomaly_detected", "latest_signal": {"signal_type": "http_health", "status": "fail", "detail": "x"}, | ||
| 191 | + }), \ | ||
| 192 | + patch.object(deploy_rollback, "rollback_path_b", return_value={"status": "rolled_back"}) as mock_rb_b: | ||
| 193 | + result = self_hosting_e2e.run_self_hosting_closed_loop(self.db_path, "goal", target_folder_path=paths.repo_root()) | ||
| 194 | + | ||
| 195 | + self.assertEqual(result, {"stage": "monitor", "outcome": "rolled_back_path_b", "deployment_id": "dep-1"}) | ||
| 196 | + mock_rb_b.assert_called_once() | ||
| 197 | + args, kwargs = mock_rb_b.call_args | ||
| 198 | + self.assertEqual(args[2], 3) # live_pid from cutover's own return value | ||
| 199 | + self.assertEqual(args[3], "/prev") # previous_instance_dir threaded through correctly | ||
| 200 | + | ||
| 201 | + | ||
| 202 | +class RealSelfHostingClosedLoopTest(unittest.TestCase): | ||
| 203 | + """The literal acceptance scenario: self-development-feasibility.md's 9 | ||
| 204 | + steps, for real -- a real self-target Goal ("fix the loop_round | ||
| 205 | + off-by-one boundary bug in p_org_task_lifecycle", the exact example | ||
| 206 | + that document uses), real Planner/Coder/Validator `claude` calls, and | ||
| 207 | + (if the Coder's fix genuinely lands on a core-engine file and Validator | ||
| 208 | + PASSes) a REAL candidate cutover of a worktree checkout of THIS repo -- | ||
| 209 | + meaning a real subprocess bound to a real port, a real one-time DB | ||
| 210 | + copy, and on success a real `git revert`-reversible commit or a real | ||
| 211 | + rollback exercised against this actual repository's git history. | ||
| 212 | + | ||
| 213 | + Gated behind RUN_REAL_SELF_HOSTING_E2E=1 -- deliberately a DIFFERENT, | ||
| 214 | + stricter env var than this project's usual RUN_REAL_AGENT_E2E (used by | ||
| 215 | + every other real-`claude`-call test in this codebase), because this is | ||
| 216 | + the single highest-blast-radius action in the whole project: unlike a | ||
| 217 | + normal RUN_REAL_AGENT_E2E test (which costs real API usage but only | ||
| 218 | + touches ephemeral temp directories/in-memory state), this one can | ||
| 219 | + actually replace the currently-running graphlemon process and mutate | ||
| 220 | + this real repository's commit history via `git revert`. Requiring a | ||
| 221 | + SEPARATE, unambiguously-named env var means accidentally setting | ||
| 222 | + RUN_REAL_AGENT_E2E=1 to run some other, cheaper real test in this | ||
| 223 | + codebase can never accidentally trigger this one too. | ||
| 224 | + | ||
| 225 | + Not executed as part of implementing this Unit -- per the explicit | ||
| 226 | + instruction this Unit was built under, flipping this switch requires | ||
| 227 | + checking with the user first (AskUserQuestion), same as any other | ||
| 228 | + action whose blast radius extends beyond this process/session. See | ||
| 229 | + development-plan.md's U-042 note for that decision's own record. | ||
| 230 | + """ | ||
| 231 | + | ||
| 232 | + @unittest.skipUnless( | ||
| 233 | + os.environ.get("RUN_REAL_SELF_HOSTING_E2E") == "1", | ||
| 234 | + "set RUN_REAL_SELF_HOSTING_E2E=1 to run -- shells out to real `claude` calls AND performs a " | ||
| 235 | + "real candidate cutover of this repository; costs real API usage and is genuinely destructive " | ||
| 236 | + "if anything here has a bug. Requires explicit user sign-off, not just this env var, before " | ||
| 237 | + "ever actually being run -- see development-plan.md's U-042 note.", | ||
| 238 | + ) | ||
| 239 | + def test_real_self_development_feasibility_9_step_scenario(self): | ||
| 240 | + import tempfile | ||
| 241 | + | ||
| 242 | + with tempfile.NamedTemporaryFile(suffix=".db", delete=False) as tmp: | ||
| 243 | + db_path = tmp.name | ||
| 244 | + try: | ||
| 245 | + conn = schema.connect(db_path) | ||
| 246 | + schema.create_schema(conn) | ||
| 247 | + conn.execute( | ||
| 248 | + "INSERT INTO project (project_id, name, target_folder_path, has_git_repo, status) " | ||
| 249 | + "VALUES ('proj-1', 'graphlemon-self', ?, 1, 'active')", (paths.repo_root(),) | ||
| 250 | + ) | ||
| 251 | + conn.commit() | ||
| 252 | + conn.close() | ||
| 253 | + | ||
| 254 | + result = self_hosting_e2e.run_self_hosting_closed_loop( | ||
| 255 | + db_path, | ||
| 256 | + "Fix the loop_round off-by-one boundary bug in p_org_task_lifecycle: " | ||
| 257 | + "loop_round > 3 should be loop_round >= 3 for the blocked_pending_human transition.", | ||
| 258 | + target_folder_path=paths.repo_root(), | ||
| 259 | + monitor_timeout_cycles=5, | ||
| 260 | + ) | ||
| 261 | + | ||
| 262 | + self.assertIn(result["stage"], ("health_check", "cutover", "monitor")) | ||
| 263 | + self.assertIn( | ||
| 264 | + result["outcome"], | ||
| 265 | + ("finalized", "rolled_back_path_a", "rolled_back_path_b", "timed_out_still_monitoring"), | ||
| 266 | + ) | ||
| 267 | + finally: | ||
| 268 | + if os.path.exists(db_path): | ||
| 269 | + os.remove(db_path) | ||
| 270 | + | ||
| 271 | + | ||
| 272 | +if __name__ == "__main__": | ||
| 273 | + unittest.main() |