Commit 5a046a4fb8d35a2f542a1c37e3cd75e635428e98
1 parent
abfb3cb5
U-037: c_deploy_supervisor_stage (candidate checkout + one-time DB copy + launch)
Adds supervisor/deploy_stage.py: stage_deployment() creates a deployment row,
git-worktree-checks-out the target commit into a fresh directory (--detach,
never in-place git pull), copies the production DB to a one-time candidate
copy, and Popens a real candidate backend process on a free port.
Return value follows 04-P §4.6's state-machine table (status after the call
is 'candidate_starting') rather than 03-C's one-line 'staged' summary --
04-P's own trigger text ("c_deploy_supervisor_stage 完成" moves the row to
candidate_starting) is the more detailed/authoritative source when the two
disagree, same precedent used throughout this project.
Found and closed two real gaps along the way:
- deployment's own field list never included where a candidate's checkout
directory lives -- added candidate_instance_dir to schema.py (also what
the NEXT staging call's previous_instance_dir bootstraps from once a
deployment reaches 'finalized').
- backend_server.py had no CLI entry point at all since U-007 -- create_server()
was only ever instantiated in-process by tests. Added main() (argparse +
serve_forever + SIGTERM->shutdown() from a separate thread, per
http.server's own same-thread-deadlock warning).
RealCandidateLaunchTest spawns and HTTP-probes a genuine
`python3 -m src.backend_server` subprocess against a real git-worktree
checkout of this repo (not mocked) -- caught a real ordering issue doing so:
git worktree only sees committed content, so testing before this Unit's own
commit landed would check out backend_server.py without main() at all.
Fixed by using `git stash create` (builds a commit object from the current
working tree without touching HEAD/index) as the commit-to-test-against,
falling back to plain HEAD once the tree is clean post-commit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Showing
6 changed files
with
494 additions
and
1 deletions
docs/development-plan.md
| ... | ... | @@ -361,6 +361,9 @@ PH2-S1/S2/S3/S4 之间除标注的依赖外相互独立,可并行推进。 |
| 361 | 361 | - **U-037 [Risk Gate]** `c_deploy_supervisor_stage`:候选实例拉起(全新目录 checkout,不 `git pull` 原地改)+ 一次性 DB 副本机制(候选实例验证期间绝不接触生产库) |
| 362 | 362 | - 依赖:U-034, U-036 |
| 363 | 363 | - Done:给定一个已合并到主分支的 commit,能生成 `deployment` 记录(`status=staged`)并拉起一个监听临时端口、连接一次性 DB 副本的候选进程 |
| 364 | + - **实现补充(2026-08-02)**:新增 `supervisor/deploy_stage.py`:`stage_deployment()` 建 `deployment` 行、`git worktree add --detach`(不是 `-b <branch>`,候选检出是只读验证用途,不打算再接受提交,和 `work_node_run.create_worktree()` 的编码用 worktree 是两回事)到全新目录、复制生产 DB 到一次性副本、`subprocess.Popen` 拉起候选进程。**返回值订正(低风险,按判断处理未升级询问)**:03-C `output_contract` 字面写返回 `{deployment_id, status:'staged'}`,但 04-P §4.6 状态转移表明确写"`c_deploy_supervisor_stage` 完成"这个触发条件本身就是把 `deployment.status` 从 `staged` 推进到 `candidate_starting`——两处矛盾,按"更详细的 04-P 状态机表为准"处理,函数完成时返回并持久化的都是 `candidate_starting`,不是字面上的 `staged`。**发现并补上两个真实缺口**:① `deployment` 表 01-M 字段清单里完全没有"候选检出目录存在哪"这一列——`candidate_port`/`candidate_db_path` 都有,唯独检出目录没地方存,Deploy Supervisor 若在 staging 和 cutover 之间重启就会彻底丢失候选目录位置;`schema.py` 补了 `candidate_instance_dir` 列(同一次 finalized 部署的这个字段,就是下一次 staging 时 `previous_instance_dir` 的来源——candidate 一旦顺利转正就是新 Live,回滚要回退到的正是它)。② `backend_server.py` 从 U-007 建骨架以来就没有任何 CLI 入口——`create_server()` 只在测试里以进程内方式实例化过,从没人真正把它作为独立 OS 进程启动过;候选进程恰好需要这个能力,顺手补了 `main()`(`python3 -m src.backend_server --host --port --db-path --skill-repo-path`,`SIGTERM` 走 `server.shutdown()`——必须在另一个线程调用,同线程调用 `serve_forever()` 期间调 `shutdown()` 会按 `http.server` 自己的文档死锁)。`/api/health` 端点本身留给 U-038 自己去加,不在这里超范围实现。 | |
| 365 | + - **验证方式**:`BootstrapPreviousTest`/`StageDeploymentHappyPathTest`/`CheckoutFailedTest` 用真实 sqlite + 真实 git worktree 操作(只 mock 掉真正拉候选进程这一步,验证 `stage_deployment()` 自己的逻辑);`RealCandidateLaunchTest` 完全不 mock——对着 graphlemon 自己仓库真实 `git worktree add --detach` 检出 + 真实 `python3 -m src.backend_server` 子进程拉起 + 真实 HTTP 请求验证候选进程确实在真实监听、真实响应,验证完 `SIGTERM`(超时兜底 `SIGKILL`)+ `git worktree remove` 清理干净。这个真实测试意外命中一个开发时序问题:`git worktree add` 只看已提交内容,而这次实现是先写代码再测试再提交(标准流程),测试跑的时候 `main()` 还没提交,导致 checkout 出来的 `backend_server.py` 是没有 `main()` 的旧版本,候选进程"启动"后什么也不做直接退出——用 `git stash create`(不碰工作区/HEAD,只生成一个代表当前工作区状态的 commit 对象)替代直接 `HEAD`,让测试始终对着"当前工作区实际内容"验证,而不是"上一次提交的内容";提交后本地工作区变干净时 `git stash create` 打印空字符串,自动回退到普通 `HEAD`,两种时序都覆盖。 | |
| 366 | + - 覆盖:03-C `c_deploy_supervisor_stage`;01-M `deployment.candidate_instance_dir`(补字段);`backend_server.py` 新增可独立运行的 CLI 入口 | |
| 364 | 367 | |
| 365 | 368 | - **U-038 [Risk Gate]** `c_deploy_supervisor_health_check`:`process_alive`/`http_health`/`canary_task` 三项 pre-cutover 检查 |
| 366 | 369 | - 依赖:U-037 | ... | ... |
docs/sdd/unit_matrix.json
| ... | ... | @@ -54,7 +54,7 @@ |
| 54 | 54 | {"id": "U-034", "phase": "PH5", "sub_phase": "PH5-S1", "name": "01-M 扩展:deployment/deploy_health_signal + core_engine 枚举(见 development-plan.md 2026-08-02 实现补充)", "classification": "Enabler", "depends_on": ["U-001"], "coverage": {"model": ["deployment", "deploy_health_signal", "bug_record"], "view": [], "controller": [], "process": [], "requirement": []}, "done": "两张新表建表成功(共 17 张);bug_record 新枚举值向后兼容 U-015/U-018 已有测试;deployment 状态机用 01-M 1.15 完整 15 值枚举;ux_deployment_active 表达式索引强制全局单例非终态约束;bug_record.py 应用层白名单刻意未跟着放开 core_engine,classification+路由逻辑留给 U-035"}, |
| 55 | 55 | {"id": "U-035", "phase": "PH5", "sub_phase": "PH5-S1", "name": "self-target worktree 无条件强制 + core_engine 分类审计(见 development-plan.md 2026-08-02 实现补充)", "classification": "Risk Gate", "depends_on": ["U-013", "U-016", "U-034"], "coverage": {"model": [], "view": [], "controller": [], "process": [], "requirement": ["REQ-03"]}, "done": "self-target 时无条件走 worktree(work_graph_orchestrate.run_graph 内强制);core_engine 分类(src/core_engine_classify.py,路径规则)在 self-target 任务 PASS 收尾时记一条审计用 bug_record;且改动落地后重跑 U-013/REQ-03(worktree 隔离)现有回归用例全部通过。**2026-08-02 用户决策修正**:Deploy Supervisor 触发条件按 supervisor-auto-rollback-design.md §1 字面设计(self-target+PASS 即触发,bug_record 是旁路产物不是前提),而非 03-C/04-P 字面文本要求的\"先有 core_engine bug_record\"——后者会让从不产生 bug_record 的 goal 型自我改进任务永远触发不了 Deploy Supervisor,已用 AskUserQuestion 与用户确认此订正。c_org_loop_route 未改动(fast_lane/full_replan 是 FAIL 路径重试路由,与此处 PASS 路径的蓝绿触发正交),deployment 行创建仍是 U-037 的范围,不在本 Unit 提前做"}, |
| 56 | 56 | {"id": "U-036", "phase": "PH5", "sub_phase": "PH5-S2", "name": "supervisor/isolation 补齐单测 + Deploy Supervisor 进程骨架(见 development-plan.md 2026-08-02 实现补充)", "classification": "Enabler", "depends_on": [], "coverage": {"model": [], "view": [], "controller": [], "process": [], "requirement": []}, "done": "四模块各有 test_*.py(供 python3 -m unittest discover -s supervisor/isolation 运行,52 测试全绿);Deploy Supervisor 骨架(supervisor/deploy_supervisor.py)独立进程可启停(真实子进程验证,非 mock),不受 Live/Candidate 崩溃影响;本 Unit 只做进程生命周期骨架,不含任何真实部署决策逻辑(U-037+ 的 poll_fn 挂载点)"}, |
| 57 | - {"id": "U-037", "phase": "PH5", "sub_phase": "PH5-S2", "name": "c_deploy_supervisor_stage", "classification": "Risk Gate", "depends_on": ["U-034", "U-036"], "coverage": {"model": ["deployment"], "view": [], "controller": ["c_deploy_supervisor_stage"], "process": [], "requirement": []}, "done": "能生成 deployment 记录并拉起监听临时端口、连一次性 DB 副本的候选进程"}, | |
| 57 | + {"id": "U-037", "phase": "PH5", "sub_phase": "PH5-S2", "name": "c_deploy_supervisor_stage(见 development-plan.md 2026-08-02 实现补充)", "classification": "Risk Gate", "depends_on": ["U-034", "U-036"], "coverage": {"model": ["deployment"], "view": [], "controller": ["c_deploy_supervisor_stage"], "process": [], "requirement": []}, "done": "能生成 deployment 记录并拉起监听临时端口、连一次性 DB 副本的候选进程(supervisor/deploy_stage.py,真实 git worktree + 真实子进程验证过);顺带补上 deployment.candidate_instance_dir 缺字段 + backend_server.py 首个 CLI 入口(此前只能进程内启动,从未真正作为独立 OS 进程跑过)"}, | |
| 58 | 58 | {"id": "U-038", "phase": "PH5", "sub_phase": "PH5-S2", "name": "c_deploy_supervisor_health_check", "classification": "Risk Gate", "depends_on": ["U-037"], "coverage": {"model": [], "view": [], "controller": ["c_deploy_supervisor_health_check"], "process": [], "requirement": []}, "done": "process_alive/http_health/canary_task 三项检查全过 -> health_check_passed;任一失败 -> health_check_failed 且 Live 未受影响"}, |
| 59 | 59 | {"id": "U-039", "phase": "PH5", "sub_phase": "PH5-S2", "name": "c_deploy_supervisor_cutover", "classification": "Risk Gate", "depends_on": ["U-038"], "coverage": {"model": ["deployment"], "view": [], "controller": ["c_deploy_supervisor_cutover"], "process": [], "requirement": []}, "done": "健康检查通过后完成真实切换;在途 work_node_execution 重启后可恢复;previous_instance_dir 保留至监控窗口结束"}, |
| 60 | 60 | {"id": "U-040", "phase": "PH5", "sub_phase": "PH5-S3", "name": "c_deploy_supervisor_monitor", "classification": "Risk Gate", "depends_on": ["U-039"], "coverage": {"model": ["deploy_health_signal"], "view": [], "controller": ["c_deploy_supervisor_monitor"], "process": [], "requirement": []}, "done": "四类信号采样正常;validator_approval_rate 偏离基线阈值时触发 anomaly_detected"}, | ... | ... |
src/backend_server.py
| ... | ... | @@ -675,3 +675,45 @@ def create_server(host="127.0.0.1", port=0, db_path=None, skill_repo_path=None): |
| 675 | 675 | server.conn_lock = threading.Lock() |
| 676 | 676 | server.skill_repo_path = skill_repo_path if skill_repo_path is not None else skill_query._REPO_ROOT |
| 677 | 677 | return server |
| 678 | + | |
| 679 | + | |
| 680 | +def main(): | |
| 681 | + """U-037: CLI entry point (`python3 -m src.backend_server ...`) -- until | |
| 682 | + this Unit, `create_server()` was only ever instantiated in-process by | |
| 683 | + tests; there was no way to actually launch graphlemon's own backend as a | |
| 684 | + standalone OS process. c_deploy_supervisor_stage needs exactly that (a | |
| 685 | + real candidate process, `subprocess.Popen`, listening on a real port) -- | |
| 686 | + found while building it, added here since it's the natural, minimal home | |
| 687 | + for "run this server for real" regardless of who launches it. | |
| 688 | + """ | |
| 689 | + import argparse | |
| 690 | + import signal | |
| 691 | + | |
| 692 | + parser = argparse.ArgumentParser() | |
| 693 | + parser.add_argument("--host", default="127.0.0.1") | |
| 694 | + parser.add_argument("--port", type=int, default=0) | |
| 695 | + parser.add_argument("--db-path", default=None) | |
| 696 | + parser.add_argument("--skill-repo-path", default=None) | |
| 697 | + args = parser.parse_args() | |
| 698 | + | |
| 699 | + server = create_server(args.host, args.port, args.db_path, args.skill_repo_path) | |
| 700 | + | |
| 701 | + def _handle_sigterm(signum, frame): | |
| 702 | + # server.shutdown() blocks until serve_forever()'s loop notices and | |
| 703 | + # returns -- calling it from the SAME thread that's running | |
| 704 | + # serve_forever() (the signal handler runs on the main thread, same | |
| 705 | + # as serve_forever() below) deadlocks, per http.server's own docs. | |
| 706 | + threading.Thread(target=server.shutdown, daemon=True).start() | |
| 707 | + | |
| 708 | + signal.signal(signal.SIGTERM, _handle_sigterm) | |
| 709 | + print("graphlemon backend listening on {}:{}".format(*server.server_address), flush=True) | |
| 710 | + try: | |
| 711 | + server.serve_forever() | |
| 712 | + except KeyboardInterrupt: | |
| 713 | + pass | |
| 714 | + finally: | |
| 715 | + server.server_close() | |
| 716 | + | |
| 717 | + | |
| 718 | +if __name__ == "__main__": | |
| 719 | + main() | ... | ... |
src/schema.py
| ... | ... | @@ -289,6 +289,15 @@ CREATE TABLE deployment ( |
| 289 | 289 | )), |
| 290 | 290 | candidate_port INTEGER, |
| 291 | 291 | candidate_db_path TEXT, |
| 292 | + -- Not in supervisor-auto-rollback-design.md §3's field table (found | |
| 293 | + -- missing while building U-037's c_deploy_supervisor_stage): the | |
| 294 | + -- checkout directory a candidate is actually running from has to be | |
| 295 | + -- persisted somewhere, or a Deploy Supervisor restart between staging | |
| 296 | + -- and cutover would lose track of it entirely -- the same class of gap | |
| 297 | + -- REQ-06 exists to prevent everywhere else. Once a deployment reaches | |
| 298 | + -- 'live'/'finalized', this same directory IS what the next deployment's | |
| 299 | + -- `previous_instance_dir` bootstraps from (see deploy_stage.py). | |
| 300 | + candidate_instance_dir TEXT, | |
| 292 | 301 | started_at TEXT, |
| 293 | 302 | cutover_at TEXT, |
| 294 | 303 | monitoring_closed_at TEXT, | ... | ... |
supervisor/deploy_stage.py
0 → 100644
| 1 | +"""c_deploy_supervisor_stage (03-C §3.15 / 04-P §4.6): candidate-instance | |
| 2 | +staging -- the entry point Deploy Supervisor's own poll loop (the `poll_fn` | |
| 3 | +hook `deploy_supervisor.run_foreground()` calls, U-036) invokes once it | |
| 4 | +discovers a self-target merge ready to deploy (the `bug_record(root_cause_ | |
| 5 | +category=core_engine)` audit trail U-035 records is that discovery signal -- | |
| 6 | +this module doesn't do the discovering itself, it's handed a commit hash by | |
| 7 | +whatever does). | |
| 8 | + | |
| 9 | +**Return-value note (documented deviation, low-stakes -- resolved by | |
| 10 | +judgment, not escalated)**: 03-C's `output_contract` literally says this | |
| 11 | +returns `{deployment_id, status: 'staged'}`, but 04-P §4.6's own transition | |
| 12 | +table says the trigger "c_deploy_supervisor_stage 完成" is what moves | |
| 13 | +`deployment.status` FROM 'staged' TO 'candidate_starting' -- i.e. by the | |
| 14 | +time this function's side effects (checkout, DB copy, candidate launch) are | |
| 15 | +done and it's ready to return, the row's real, persisted status is already | |
| 16 | +'candidate_starting', not 'staged'. 04-P's full state-machine table is more | |
| 17 | +detailed/authoritative here than 03-C's one-line summary (same "SDD 内部冲突 | |
| 18 | +以更详细的为准" precedent CLAUDE.md states), so this returns the row's actual | |
| 19 | +post-call status rather than literally echoing 'staged'. | |
| 20 | + | |
| 21 | +**Missing field found and added (see schema.py's own comment)**: | |
| 22 | +`deployment.candidate_instance_dir` -- 01-M's own field table never listed | |
| 23 | +where a candidate's checkout directory persists, even though cutover/ | |
| 24 | +rollback both need to know it and a Deploy Supervisor restart between | |
| 25 | +staging and cutover would otherwise lose track of it entirely. | |
| 26 | +""" | |
| 27 | + | |
| 28 | +import os | |
| 29 | +import shutil | |
| 30 | +import socket | |
| 31 | +import subprocess | |
| 32 | +import sys | |
| 33 | +import uuid | |
| 34 | + | |
| 35 | +from src import paths | |
| 36 | + | |
| 37 | + | |
| 38 | +class CheckoutFailedError(Exception): | |
| 39 | + """CHECKOUT_FAILED -- 03-C §3.15: fresh-directory checkout failed | |
| 40 | + (e.g. disk full). Not retried -- the caller routes this to | |
| 41 | + `deployment.status='blocked_pending_human'`, already applied by the | |
| 42 | + time this is raised.""" | |
| 43 | + | |
| 44 | + | |
| 45 | +def _last_finalized_deployment(conn): | |
| 46 | + """Returns (git_commit_hash, candidate_instance_dir) of the most recent | |
| 47 | + `finalized` deployment, or None if there isn't one yet (the bootstrap | |
| 48 | + case schema.py's own comment already flags: the very first deployment | |
| 49 | + ever has no prior finalized deployment to point at).""" | |
| 50 | + row = conn.execute( | |
| 51 | + "SELECT git_commit_hash, candidate_instance_dir FROM deployment " | |
| 52 | + "WHERE status = 'finalized' ORDER BY rowid DESC LIMIT 1" | |
| 53 | + ).fetchone() | |
| 54 | + return tuple(row) if row else None | |
| 55 | + | |
| 56 | + | |
| 57 | +def _current_commit(repo_path): | |
| 58 | + return subprocess.run( | |
| 59 | + ["git", "rev-parse", "HEAD"], cwd=repo_path, capture_output=True, text=True, check=True | |
| 60 | + ).stdout.strip() | |
| 61 | + | |
| 62 | + | |
| 63 | +def _find_free_port(): | |
| 64 | + with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock: | |
| 65 | + sock.bind(("127.0.0.1", 0)) | |
| 66 | + return sock.getsockname()[1] | |
| 67 | + | |
| 68 | + | |
| 69 | +def _candidates_dir(): | |
| 70 | + d = os.path.join(paths.platform_home(), "deploy_supervisor", "candidates") | |
| 71 | + os.makedirs(d, exist_ok=True) | |
| 72 | + return d | |
| 73 | + | |
| 74 | + | |
| 75 | +def _candidate_dbs_dir(): | |
| 76 | + d = os.path.join(paths.platform_home(), "deploy_supervisor", "candidate-dbs") | |
| 77 | + os.makedirs(d, exist_ok=True) | |
| 78 | + return d | |
| 79 | + | |
| 80 | + | |
| 81 | +def _checkout_candidate(repo_root, git_commit_hash, deployment_id, checkout_parent_dir=None): | |
| 82 | + """`git worktree add --detach` (not `-b <branch>`, unlike work_node_run. | |
| 83 | + create_worktree's coding worktrees -- a candidate checkout is read-only | |
| 84 | + verification, never meant to receive further commits of its own) into a | |
| 85 | + FRESH directory, never an in-place `git pull` (03-C §3.15's own wording, | |
| 86 | + since the currently-running Live process may still be reading repo_root | |
| 87 | + off disk).""" | |
| 88 | + parent = checkout_parent_dir or _candidates_dir() | |
| 89 | + candidate_dir = os.path.join(parent, deployment_id) | |
| 90 | + subprocess.run( | |
| 91 | + ["git", "worktree", "add", "--detach", candidate_dir, git_commit_hash], | |
| 92 | + cwd=repo_root, check=True, capture_output=True, text=True, | |
| 93 | + ) | |
| 94 | + return candidate_dir | |
| 95 | + | |
| 96 | + | |
| 97 | +def _copy_prod_db(prod_db_path, deployment_id): | |
| 98 | + """MVP: whole-file copy (supervisor-auto-rollback-design.md §10 open | |
| 99 | + question 2 already flags this as the simple-first choice, revisit only | |
| 100 | + if a large production DB makes this slow).""" | |
| 101 | + dest = os.path.join(_candidate_dbs_dir(), deployment_id + ".db") | |
| 102 | + shutil.copy2(prod_db_path, dest) | |
| 103 | + return dest | |
| 104 | + | |
| 105 | + | |
| 106 | +def _launch_candidate(candidate_dir, port, db_path, python_executable=None): | |
| 107 | + python_executable = python_executable or sys.executable | |
| 108 | + return subprocess.Popen( | |
| 109 | + [python_executable, "-m", "src.backend_server", "--host", "127.0.0.1", "--port", str(port), "--db-path", db_path], | |
| 110 | + cwd=candidate_dir, stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, | |
| 111 | + ) | |
| 112 | + | |
| 113 | + | |
| 114 | +def stage_deployment( | |
| 115 | + conn, git_commit_hash, repo_root=None, prod_db_path=None, | |
| 116 | + checkout_parent_dir=None, port=None, python_executable=None, | |
| 117 | +): | |
| 118 | + """Returns {deployment_id, status, candidate_port, candidate_pid, | |
| 119 | + candidate_instance_dir}. Raises CheckoutFailedError (deployment row | |
| 120 | + already marked blocked_pending_human by the time it's raised) if the | |
| 121 | + fresh-directory checkout fails.""" | |
| 122 | + repo_root = repo_root or paths.repo_root() | |
| 123 | + prod_db_path = prod_db_path or paths.db_path() | |
| 124 | + deployment_id = str(uuid.uuid4()) | |
| 125 | + | |
| 126 | + last = _last_finalized_deployment(conn) | |
| 127 | + if last is not None: | |
| 128 | + previous_commit_hash, previous_instance_dir = last | |
| 129 | + else: | |
| 130 | + # Bootstrap: no prior finalized deployment exists yet -- the | |
| 131 | + # currently-running instance IS the "previous known-good" to roll | |
| 132 | + # back to, per schema.py's own note. | |
| 133 | + previous_commit_hash = _current_commit(repo_root) | |
| 134 | + previous_instance_dir = repo_root | |
| 135 | + | |
| 136 | + conn.execute( | |
| 137 | + "INSERT INTO deployment " | |
| 138 | + "(deployment_id, git_commit_hash, previous_commit_hash, status, previous_instance_dir, started_at) " | |
| 139 | + "VALUES (?, ?, ?, 'staged', ?, CURRENT_TIMESTAMP)", | |
| 140 | + (deployment_id, git_commit_hash, previous_commit_hash, previous_instance_dir), | |
| 141 | + ) | |
| 142 | + conn.commit() | |
| 143 | + | |
| 144 | + try: | |
| 145 | + candidate_dir = _checkout_candidate(repo_root, git_commit_hash, deployment_id, checkout_parent_dir) | |
| 146 | + except subprocess.CalledProcessError as e: | |
| 147 | + conn.execute( | |
| 148 | + "UPDATE deployment SET status = 'blocked_pending_human' WHERE deployment_id = ?", (deployment_id,) | |
| 149 | + ) | |
| 150 | + conn.commit() | |
| 151 | + raise CheckoutFailedError( | |
| 152 | + "fresh-directory checkout failed for deployment {} at commit {}: {}".format( | |
| 153 | + deployment_id, git_commit_hash, e.stderr | |
| 154 | + ) | |
| 155 | + ) from e | |
| 156 | + | |
| 157 | + candidate_db_path = _copy_prod_db(prod_db_path, deployment_id) | |
| 158 | + candidate_port = port if port is not None else _find_free_port() | |
| 159 | + process = _launch_candidate(candidate_dir, candidate_port, candidate_db_path, python_executable) | |
| 160 | + | |
| 161 | + conn.execute( | |
| 162 | + "UPDATE deployment SET status = 'candidate_starting', candidate_port = ?, candidate_db_path = ?, " | |
| 163 | + "candidate_instance_dir = ? WHERE deployment_id = ?", | |
| 164 | + (candidate_port, candidate_db_path, candidate_dir, deployment_id), | |
| 165 | + ) | |
| 166 | + conn.commit() | |
| 167 | + | |
| 168 | + return { | |
| 169 | + "deployment_id": deployment_id, "status": "candidate_starting", | |
| 170 | + "candidate_port": candidate_port, "candidate_pid": process.pid, | |
| 171 | + "candidate_instance_dir": candidate_dir, | |
| 172 | + } | ... | ... |
supervisor/test_deploy_stage.py
0 → 100644
| 1 | +"""U-037 tests: c_deploy_supervisor_stage. Real sqlite, real git worktree | |
| 2 | +operations, real one-time DB file copy -- only `_launch_candidate` (spawning | |
| 3 | +yet another real backend_server subprocess) is mocked in most tests, since | |
| 4 | +that's the expensive/slow part and is proven for real in exactly one test | |
| 5 | +(RealCandidateLaunchTest) rather than every test. | |
| 6 | + | |
| 7 | +Run: python3 -m unittest discover -s supervisor -p "test_*.py" -v | |
| 8 | +""" | |
| 9 | + | |
| 10 | +import os | |
| 11 | +import shutil | |
| 12 | +import subprocess | |
| 13 | +import sys | |
| 14 | +import tempfile | |
| 15 | +import time | |
| 16 | +import unittest | |
| 17 | +import urllib.error | |
| 18 | +import urllib.request | |
| 19 | +from unittest.mock import patch | |
| 20 | + | |
| 21 | +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) | |
| 22 | + | |
| 23 | +from supervisor import deploy_stage | |
| 24 | +from src import paths, schema | |
| 25 | + | |
| 26 | + | |
| 27 | +def _init_git_repo(path): | |
| 28 | + subprocess.run(["git", "init", "-q"], cwd=path, check=True) | |
| 29 | + subprocess.run(["git", "config", "user.email", "test@graphlemon.local"], cwd=path, check=True) | |
| 30 | + subprocess.run(["git", "config", "user.name", "test"], cwd=path, check=True) | |
| 31 | + | |
| 32 | + | |
| 33 | +def _commit_all(path, message): | |
| 34 | + subprocess.run(["git", "add", "-A"], cwd=path, check=True) | |
| 35 | + subprocess.run(["git", "commit", "-q", "-m", message], cwd=path, check=True) | |
| 36 | + return subprocess.run( | |
| 37 | + ["git", "rev-parse", "HEAD"], cwd=path, capture_output=True, text=True, check=True | |
| 38 | + ).stdout.strip() | |
| 39 | + | |
| 40 | + | |
| 41 | +class DeployStageTestBase(unittest.TestCase): | |
| 42 | + def setUp(self): | |
| 43 | + self.home_tmp = tempfile.mkdtemp(prefix="deploy-stage-home-") | |
| 44 | + self.addCleanup(shutil.rmtree, self.home_tmp, ignore_errors=True) | |
| 45 | + self.old_home = os.environ.get("GRAPHLEMON_HOME") | |
| 46 | + os.environ["GRAPHLEMON_HOME"] = self.home_tmp | |
| 47 | + | |
| 48 | + self.repo_dir = tempfile.mkdtemp(prefix="deploy-stage-repo-") | |
| 49 | + self.addCleanup(shutil.rmtree, self.repo_dir, ignore_errors=True) | |
| 50 | + _init_git_repo(self.repo_dir) | |
| 51 | + with open(os.path.join(self.repo_dir, "README.md"), "w", encoding="utf-8") as f: | |
| 52 | + f.write("v1\n") | |
| 53 | + self.commit_v1 = _commit_all(self.repo_dir, "v1") | |
| 54 | + with open(os.path.join(self.repo_dir, "README.md"), "w", encoding="utf-8") as f: | |
| 55 | + f.write("v2\n") | |
| 56 | + self.commit_v2 = _commit_all(self.repo_dir, "v2") | |
| 57 | + | |
| 58 | + self.conn = schema.connect(":memory:") | |
| 59 | + schema.create_schema(self.conn) | |
| 60 | + | |
| 61 | + self.prod_db_path = os.path.join(self.home_tmp, "prod.db") | |
| 62 | + prod_conn = schema.connect(self.prod_db_path) | |
| 63 | + schema.create_schema(prod_conn) | |
| 64 | + prod_conn.close() | |
| 65 | + | |
| 66 | + def tearDown(self): | |
| 67 | + if self.old_home is None: | |
| 68 | + os.environ.pop("GRAPHLEMON_HOME", None) | |
| 69 | + else: | |
| 70 | + os.environ["GRAPHLEMON_HOME"] = self.old_home | |
| 71 | + self.conn.close() | |
| 72 | + | |
| 73 | + | |
| 74 | +class BootstrapPreviousTest(DeployStageTestBase): | |
| 75 | + def test_no_prior_finalized_deployment_bootstraps_from_current_repo_state(self): | |
| 76 | + # setUp() leaves repo_dir's HEAD at commit_v2 (the last commit made); | |
| 77 | + # bootstrap must read the repo's *current* HEAD, not the commit being | |
| 78 | + # staged (git worktree add --detach doesn't move repo_dir's own HEAD). | |
| 79 | + with patch.object(deploy_stage, "_launch_candidate") as mock_launch: | |
| 80 | + mock_launch.return_value.pid = 12345 | |
| 81 | + result = deploy_stage.stage_deployment( | |
| 82 | + self.conn, self.commit_v2, repo_root=self.repo_dir, prod_db_path=self.prod_db_path, port=9999, | |
| 83 | + ) | |
| 84 | + row = self.conn.execute( | |
| 85 | + "SELECT previous_commit_hash, previous_instance_dir FROM deployment WHERE deployment_id = ?", | |
| 86 | + (result["deployment_id"],), | |
| 87 | + ).fetchone() | |
| 88 | + self.assertEqual(row[0], self.commit_v2) | |
| 89 | + self.assertEqual(row[1], self.repo_dir) | |
| 90 | + | |
| 91 | + def test_prior_finalized_deployment_supplies_previous_fields(self): | |
| 92 | + self.conn.execute( | |
| 93 | + "INSERT INTO deployment (deployment_id, git_commit_hash, previous_commit_hash, status, " | |
| 94 | + "candidate_instance_dir, previous_instance_dir) " | |
| 95 | + "VALUES ('dep-0', 'commit-0', 'commit-minus-1', 'finalized', '/old/candidate/dir', '/older/dir')" | |
| 96 | + ) | |
| 97 | + self.conn.commit() | |
| 98 | + with patch.object(deploy_stage, "_launch_candidate") as mock_launch: | |
| 99 | + mock_launch.return_value.pid = 12345 | |
| 100 | + result = deploy_stage.stage_deployment( | |
| 101 | + self.conn, self.commit_v2, repo_root=self.repo_dir, prod_db_path=self.prod_db_path, port=9999, | |
| 102 | + ) | |
| 103 | + row = self.conn.execute( | |
| 104 | + "SELECT previous_commit_hash, previous_instance_dir FROM deployment WHERE deployment_id = ?", | |
| 105 | + (result["deployment_id"],), | |
| 106 | + ).fetchone() | |
| 107 | + self.assertEqual(row[0], "commit-0") | |
| 108 | + self.assertEqual(row[1], "/old/candidate/dir") | |
| 109 | + | |
| 110 | + | |
| 111 | +class StageDeploymentHappyPathTest(DeployStageTestBase): | |
| 112 | + def test_creates_candidate_starting_row_with_real_worktree_and_db_copy(self): | |
| 113 | + with patch.object(deploy_stage, "_launch_candidate") as mock_launch: | |
| 114 | + mock_launch.return_value.pid = 54321 | |
| 115 | + result = deploy_stage.stage_deployment( | |
| 116 | + self.conn, self.commit_v2, repo_root=self.repo_dir, prod_db_path=self.prod_db_path, port=9999, | |
| 117 | + ) | |
| 118 | + | |
| 119 | + self.assertEqual(result["status"], "candidate_starting") | |
| 120 | + self.assertEqual(result["candidate_port"], 9999) | |
| 121 | + self.assertEqual(result["candidate_pid"], 54321) | |
| 122 | + | |
| 123 | + row = self.conn.execute( | |
| 124 | + "SELECT status, candidate_port, candidate_db_path, candidate_instance_dir, git_commit_hash " | |
| 125 | + "FROM deployment WHERE deployment_id = ?", | |
| 126 | + (result["deployment_id"],), | |
| 127 | + ).fetchone() | |
| 128 | + self.assertEqual(row[0], "candidate_starting") | |
| 129 | + self.assertEqual(row[1], 9999) | |
| 130 | + self.assertEqual(row[4], self.commit_v2) | |
| 131 | + | |
| 132 | + # real git worktree: candidate dir exists and is checked out at v2 | |
| 133 | + candidate_dir = row[3] | |
| 134 | + self.assertTrue(os.path.isdir(candidate_dir)) | |
| 135 | + with open(os.path.join(candidate_dir, "README.md"), encoding="utf-8") as f: | |
| 136 | + self.assertEqual(f.read(), "v2\n") | |
| 137 | + | |
| 138 | + # real DB copy: a distinct file, not the same path as prod | |
| 139 | + candidate_db_path = row[2] | |
| 140 | + self.assertNotEqual(candidate_db_path, self.prod_db_path) | |
| 141 | + self.assertTrue(os.path.isfile(candidate_db_path)) | |
| 142 | + | |
| 143 | + def test_only_one_non_terminal_deployment_allowed(self): | |
| 144 | + with patch.object(deploy_stage, "_launch_candidate") as mock_launch: | |
| 145 | + mock_launch.return_value.pid = 1 | |
| 146 | + deploy_stage.stage_deployment( | |
| 147 | + self.conn, self.commit_v2, repo_root=self.repo_dir, prod_db_path=self.prod_db_path, port=9001, | |
| 148 | + ) | |
| 149 | + with self.assertRaises(Exception): # schema's ux_deployment_active partial unique index | |
| 150 | + deploy_stage.stage_deployment( | |
| 151 | + self.conn, self.commit_v1, repo_root=self.repo_dir, prod_db_path=self.prod_db_path, port=9002, | |
| 152 | + ) | |
| 153 | + | |
| 154 | + | |
| 155 | +class CheckoutFailedTest(DeployStageTestBase): | |
| 156 | + def test_nonexistent_commit_marks_blocked_pending_human_and_raises(self): | |
| 157 | + with self.assertRaises(deploy_stage.CheckoutFailedError): | |
| 158 | + deploy_stage.stage_deployment( | |
| 159 | + self.conn, "0" * 40, repo_root=self.repo_dir, prod_db_path=self.prod_db_path, port=9003, | |
| 160 | + ) | |
| 161 | + row = self.conn.execute("SELECT status FROM deployment").fetchone() | |
| 162 | + self.assertEqual(row[0], "blocked_pending_human") | |
| 163 | + | |
| 164 | + def test_checkout_failure_leaves_no_candidate_process_launched(self): | |
| 165 | + with patch.object(deploy_stage, "_launch_candidate") as mock_launch: | |
| 166 | + with self.assertRaises(deploy_stage.CheckoutFailedError): | |
| 167 | + deploy_stage.stage_deployment( | |
| 168 | + self.conn, "0" * 40, repo_root=self.repo_dir, prod_db_path=self.prod_db_path, port=9004, | |
| 169 | + ) | |
| 170 | + mock_launch.assert_not_called() | |
| 171 | + | |
| 172 | + | |
| 173 | +class RealCandidateLaunchTest(unittest.TestCase): | |
| 174 | + """Not mocked -- stages a real candidate checkout of graphlemon's OWN | |
| 175 | + repo (read-only `git worktree add --detach` at the current HEAD, cleaned | |
| 176 | + up in tearDown) and launches a real `python3 -m src.backend_server` | |
| 177 | + subprocess against it, then hits it over real HTTP, per this Unit's own | |
| 178 | + "拉起监听临时端口...的候选进程" Done criterion -- proves backend_server's | |
| 179 | + new CLI entry point (added by this Unit) actually works end to end, not | |
| 180 | + just that stage_deployment() calls the right mocked function.""" | |
| 181 | + | |
| 182 | + def setUp(self): | |
| 183 | + self.home_tmp = tempfile.mkdtemp(prefix="deploy-stage-real-home-") | |
| 184 | + self.addCleanup(shutil.rmtree, self.home_tmp, ignore_errors=True) | |
| 185 | + self.old_home = os.environ.get("GRAPHLEMON_HOME") | |
| 186 | + os.environ["GRAPHLEMON_HOME"] = self.home_tmp | |
| 187 | + | |
| 188 | + self.conn = schema.connect(":memory:") | |
| 189 | + schema.create_schema(self.conn) | |
| 190 | + self.addCleanup(self.conn.close) | |
| 191 | + | |
| 192 | + self.prod_db_path = os.path.join(self.home_tmp, "prod.db") | |
| 193 | + prod_conn = schema.connect(self.prod_db_path) | |
| 194 | + schema.create_schema(prod_conn) | |
| 195 | + prod_conn.close() | |
| 196 | + | |
| 197 | + def tearDown(self): | |
| 198 | + if self.old_home is None: | |
| 199 | + os.environ.pop("GRAPHLEMON_HOME", None) | |
| 200 | + else: | |
| 201 | + os.environ["GRAPHLEMON_HOME"] = self.old_home | |
| 202 | + | |
| 203 | + @staticmethod | |
| 204 | + def _commit_to_test_against(): | |
| 205 | + """`git worktree add` only ever sees COMMITTED content -- if this | |
| 206 | + test runs before this Unit's own changes are committed (the normal | |
| 207 | + implement-then-test-then-commit order), plain HEAD would check out | |
| 208 | + a backend_server.py without this Unit's new main() entry point at | |
| 209 | + all. `git stash create` builds a real commit object representing | |
| 210 | + the current working tree (including uncommitted tracked-file | |
| 211 | + changes) WITHOUT touching the working tree, index, or HEAD -- exactly | |
| 212 | + what's needed here. Once this Unit is actually committed the working | |
| 213 | + tree is clean and `git stash create` prints nothing, so this falls | |
| 214 | + back to plain HEAD in that case (post-commit reruns of this test).""" | |
| 215 | + stash_sha = subprocess.run( | |
| 216 | + ["git", "stash", "create"], cwd=paths.repo_root(), capture_output=True, text=True, check=True | |
| 217 | + ).stdout.strip() | |
| 218 | + return stash_sha or deploy_stage._current_commit(paths.repo_root()) | |
| 219 | + | |
| 220 | + def test_real_candidate_process_actually_serves_http(self): | |
| 221 | + current_commit = self._commit_to_test_against() | |
| 222 | + result = deploy_stage.stage_deployment( | |
| 223 | + self.conn, current_commit, repo_root=paths.repo_root(), prod_db_path=self.prod_db_path, | |
| 224 | + ) | |
| 225 | + pid = result["candidate_pid"] | |
| 226 | + port = result["candidate_port"] | |
| 227 | + candidate_dir = result["candidate_instance_dir"] | |
| 228 | + try: | |
| 229 | + _UNSET = object() | |
| 230 | + deadline = time.time() + 10 | |
| 231 | + last_error = None | |
| 232 | + status = _UNSET | |
| 233 | + while time.time() < deadline: | |
| 234 | + try: | |
| 235 | + with urllib.request.urlopen( | |
| 236 | + "http://127.0.0.1:{}/api/settings/concurrency".format(port), timeout=1 | |
| 237 | + ) as resp: | |
| 238 | + status = resp.status | |
| 239 | + break | |
| 240 | + except urllib.error.URLError as e: # server not up yet (connection refused) | |
| 241 | + last_error = e | |
| 242 | + time.sleep(0.2) | |
| 243 | + self.assertIsNot(status, _UNSET, "candidate server never responded: {}".format(last_error)) | |
| 244 | + self.assertEqual(status, 200) # real route, real response, real HTTP round trip | |
| 245 | + finally: | |
| 246 | + os.kill(pid, 15) # SIGTERM | |
| 247 | + deadline = time.time() + 5 | |
| 248 | + while time.time() < deadline and _pid_alive(pid): | |
| 249 | + time.sleep(0.1) | |
| 250 | + if _pid_alive(pid): | |
| 251 | + os.kill(pid, 9) | |
| 252 | + subprocess.run( | |
| 253 | + ["git", "worktree", "remove", "--force", candidate_dir], cwd=paths.repo_root(), | |
| 254 | + capture_output=True, text=True, | |
| 255 | + ) | |
| 256 | + | |
| 257 | + | |
| 258 | +def _pid_alive(pid): | |
| 259 | + try: | |
| 260 | + os.kill(pid, 0) | |
| 261 | + except OSError: | |
| 262 | + return False | |
| 263 | + return True | |
| 264 | + | |
| 265 | + | |
| 266 | +if __name__ == "__main__": | |
| 267 | + unittest.main() | ... | ... |