-
_launch_candidate() spawns a real `python -m src.backend_server` process (`revived`) to re-verify the previous version during a post-cutover rollback. On the success path it's correctly left running -- it IS the new Live. But on every failure path (never bound live_port, failed re-verification, git revert conflict) nothing ever terminated it -- found via 31 accumulated orphaned processes on this machine, one per failed rollback_path_b call across many real test runs this session. This is also the likely cause of RollbackPathBFailureTest's own reported non-determinism (three independent real Validator sessions saw it fail, never reproduced manually): a leaked process from an earlier failed run can still be squatting on the port a later run's _find_free_port() picks, producing a false "alive" health-check reading. Fix: call deploy_cutover._stop_process(revived.pid) in both failure branches before raising, mirroring the pattern cutover() already uses for its own stop_timeout-bounded process teardown. Also fixes RollbackPathBHappyPathTest's own matching leak (it re-appended bad_live.pid a second time instead of tracking the actual revived process, a copy-paste bug, not real cleanup) by capturing the real pid via a thin _launch_candidate wrapper. 31 pre-existing orphaned processes killed by hand; verified 5 repeated test runs after the fix leak zero new ones. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
-
check_canary_task() creates a real project + task through the candidate's own HTTP API to verify the write path works, but only ever cleaned up its throwaway git directory -- the DB rows it created via the API were never removed. Every real health check permanently grew whatever DB the candidate was using by one dead project (pointing at an already-deleted temp dir) and one task stuck at 'queued' forever. Found by inspecting the real ~/.graphlemon/graphlemon.db, which had 5 such leaked 'deploy-canary' projects from earlier real health-check runs (now archived by hand). Fix: archive the created project in a finally block that covers every return path, including task-creation/verification failures where only the project (not the task) got created. Best-effort -- the archive call failing must not override the real check result. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
-
Attempt 13b was the first real run to get all the way through a real cutover -- task lifecycle, classification, staging, health check (fixed last commit), and a genuine blue-green cutover all succeeded for real. It correctly landed in 'monitor' and then correctly couldn't finish: the real production exit condition (deploy_monitor.MONITORING_WINDOW_HOURS=24h OR MONITORING_MIN_TASK_COUNT=10 real tasks in the monitored project, NOT the disposable per-cycle canary project) isn't practical to wait out in an interactive session. This isn't a bug -- those are sensible real defaults -- it's a testability gap: run_self_hosting_closed_loop already exposes monitor_timeout_cycles as an override knob for exactly this kind of situation, but never threaded the two constants that actually gate "monitoring_window_clean" through to run_monitor_cycle. Added monitor_window_hours/monitor_min_task_count params, defaulting to deploy_monitor's real constants (production behavior unchanged), threaded through to the actual run_monitor_cycle call. New assertion locks in that overrides actually reach it. Also documents the full run of 9 real bugs found and fixed across 13 attempts in development-plan.md's U-042 note. 564 tests green (441 src + 123 supervisor). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
-
Attempts 7 and 11 both failed at exactly the same spot: process_alive read 'ok' within milliseconds of Popen, then http_health hit ECONNREFUSED in the same second and gave up. Real cause: process_alive only confirms the process EXISTS, not that backend_server.py has finished importing modules and actually bound its listen socket -- a real, non-zero startup gap. The old check_http_health did one urlopen(timeout=X) call; that timeout only bounds how long to wait for a response once connected, not how long to wait for the port to start accepting connections at all -- ECONNREFUSED fires essentially instantly when nothing's listening yet, regardless of timeout. This touches an explicit 03-C design statement ("不重试——重试属于给坏版本 第二次机会"), so confirmed with the user before changing it rather than just reinterpreting the policy unilaterally. Resolution: tolerating normal startup latency within ONE health-check pass isn't "retrying a failed check" -- check_process_alive already does exactly this (polls up to 30s for the pid to exist). http_health now does the same, polling within its own timeout for connection-refused-class errors specifically. A genuine non-2xx HTTP response is still a definitive answer from a live server and still fails immediately, no polling -- that boundary is unchanged. Two new real regression tests: a real server that starts listening 0.3s late is correctly waited for and passes; a real 500 response still fails immediately without being mistaken for "not up yet". 564 tests green (441 src + 123 supervisor). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> -
The 7th real self-hosting attempt was the first to actually reach a real health check against a real candidate process -- it correctly failed and triggered rollback_path_a for real: candidate terminated, git revert landed cleanly. But the candidate's `git worktree add --detach` checkout (candidate_instance_dir, tracked since U-037 specifically to know where this lives) was never removed -- left permanently on disk and still registered in `git worktree list`. A confirmed-bad candidate is never needed again after path A, unlike a successful cutover's candidate (which correctly becomes previous_instance_dir, kept until the monitoring window closes) -- these are different lifecycles that shouldn't share the same "leave it" behavior. rollback_path_a gains an optional candidate_instance_dir parameter (default None, existing tests unaffected) and removes the worktree before git revert when provided. self_hosting_e2e.py now threads through the value stage_deployment() already returns. New regression test uses a real `git worktree add --detach` checkout and confirms both the directory and its git worktree registration are gone after rollback. Also cleaned up 3 orphaned processes left by earlier test_deploy_rollback.py runs (backing temp dirs already gone) and the real leftover candidate worktree from this run. 561 tests green (440 src + 121 supervisor). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
-
The third real self-hosting attempt (first to actually reach stage_deployment itself, past the earlier bootstrap/commit bugs) hit an uncaught FileNotFoundError: _copy_prod_db() assumed prod_db_path always exists and shutil.copy2()'d it unconditionally. graphlemon has never actually run as a persistent service, so ~/.graphlemon/graphlemon.db never existed. stage_deployment() already handles this exact class of "no prior state" bootstrap case for previous_commit_hash/previous_instance_dir -- prod_db_path never got the same treatment, and every existing test seeds a prod db in setUp so this was never exercised. Fix: when prod_db_path doesn't exist, create a fresh schema'd db instead of copying -- semantically correct, since a genuinely-first deployment has no production data to inherit. New regression test (BootstrapProdDbTest) covers it. 555 tests green (435 src + 120 supervisor). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
-
Adds supervisor/self_hosting_e2e.py: run_self_hosting_closed_loop() chains every PH5 module built this session (U-034~U-041) with the Graph Engine's own scheduler (U-032) into one real, callable path matching self-development- feasibility.md's 9-step scenario -- self-target Goal -> Planner/Coder/ Validator -> core_engine classification -> stage -> health check -> cutover (or rollback path A if health check fails) -> monitor until either monitoring_window_clean (finalized) or anomaly_detected (rollback path B). Both real outcome paths are branches of the same call, not two separate scripted scenarios. Explicitly documented as this acceptance scenario's own orchestration glue, not a claim about Deploy Supervisor's eventual production poll loop (U-036's discovery-loop hook is still unfilled; building that is future work). Mocked wiring tests (6, run in normal regression) stub each already independently-tested module boundary and only verify the branching logic. The real, literal acceptance test is built and ready (RealSelfHostingClosedLoopTest, reproduces the document's own "loop_round > 3 should be >= 3" example) but gated behind RUN_REAL_SELF_HOSTING_E2E=1 -- deliberately a separate, distinctly-named env var from this project's usual RUN_REAL_AGENT_E2E, since this is the highest-blast-radius action in the project (can replace the running graphlemon process and git-revert commits on this real repo). Per this Unit's own explicit build instruction, not executed as part of implementing it -- doing so requires checking with the user first. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
-
Adds supervisor/deploy_rollback.py: rollback_path_a (pre-cutover, health check failed -- terminate candidate, delete its one-time DB copy, git revert, record a bug_record(core_engine) audit trail) and rollback_path_b (post-cutover, anomaly detected -- stop the bad Live, revive previous_instance_dir, re-run the same three health-check signals directly against it, git revert only once re-verification passes). Design choices follow established conventions rather than inventing new ones: no unified rollback() entry point (deployment has no PID columns, same reason deploy_stage/deploy_cutover already require callers to pass pids explicitly); ROLLBACK_FAILED is the raised error name while the persisted deployment.status is 'blocked_pending_human' (same pattern CheckoutFailedError/CutoverFailedError already use). git revert conflicts fold into the same rollback-itself-failed fallback both paths share, per design doc §10 open question 3's own recommendation, with `git revert --abort` run afterward so a failed attempt doesn't leave the repo mid-conflict. This is PH5's highest-rework-risk Unit per development-plan.md's own flag, so its own "rollback itself fails" fallback is tested for real, not assumed: a genuinely broken previous_instance_dir (a real empty directory with no Python package in it) that really fails to bind its port, and a genuinely conflicting git history (a later commit touching the same line the self-modification changed) that really fails `git revert` -- both correctly land on blocked_pending_human without retrying forever or silently succeeding, and the conflict case leaves the repo clean afterward. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
-
Adds supervisor/deploy_monitor.py: run_monitor_cycle() is one sampling pass, called repeatedly by Deploy Supervisor's own poll loop (03-C's own text: not a one-shot call). process_alive/http_health/canary_task/ validator_approval_rate sampled in order, stopping at the first anomaly. validator_approval_rate's "approve rate" has no literal definition anywhere in 01-M/03-C/04-P (no such execution_event type exists) -- derived instead from a verified structural fact in scheduler.finish_task_lifecycle: PASS closes straight to done with no bug_record ever created, FAIL/INCONCLUSIVE always creates one via judge_root_cause. A task counts as approved iff no bug_record(source='validator') exists for it. Baseline is computed by the same function looking back MONITORING_WINDOW_HOURS from cutover_at rather than persisted as a new column; baseline=None (cold start) never flags an anomaly rather than defaulting to 0. Crash-count/consecutive-http-fail windows are approximated by sample count (N = CRASH_WINDOW / MONITOR_SAMPLE_INTERVAL) rather than a true rolling time window -- documented simplification, correct as long as sampling stays roughly interval-paced. Caught my own design misreading via the tests: process_alive's monitoring- phase anomaly condition is MAX_CRASH_COUNT crashes within CRASH_WINDOW (a crash-loop detector), not "currently not alive" -- unlike U-038's health check where any single failure fails closed. A single dead reading during monitoring is recorded but must not trigger an immediate rollback by itself. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
-
Adds supervisor/deploy_cutover.py: stops the old Live process, retires the health-check-mode candidate, and relaunches the same already-verified candidate checkout bound to LIVE_PORT/PROD_DB_PATH -- http.server has no live socket-rebind operation, so "switch to production port/db" is implemented as stop-then-relaunch-from-the-same-dir rather than an in-place rebind (functionally equivalent, the only mechanism stdlib actually offers). Adds LIVE_PORT=8420 to 05 §5.9's MVP-fixed parameter table -- the one parameter that table was missing (frontend/app.js never needed one, same- origin relative fetches don't care, but successive deployments need to keep landing on the same port for a human bookmark to keep working). "在途 work_node_execution 重启后可恢复" is satisfied by relying on the existing REQ-06 recovery machinery unchanged -- cutover doesn't add any special-case logic, a cutover-triggered restart is just another interruption REQ-06 already survives. A real (not mocked) test caught a real bug: deploy_supervisor.is_alive()'s os.kill(pid, 0) check can't distinguish "truly running" from "exited but never reaped by its own parent" -- a zombie's PID slot answers kill(pid, 0) successfully until waited on. U-036's own subprocess test never caught this because that skeleton deletes its own PID file before exiting, sidestepping is_alive() entirely; deploy_cutover's test is the first one to actually poll for a stopped process to disappear, and it hung until fixed. Added reap_if_exited() (real os.waitpid for processes we spawned ourselves, falls back to is_alive() for pids from a prior session where ChildProcessError means we're not the real parent) and switched deploy_cutover._stop_process, deploy_health_check.check_process_alive, and deploy_supervisor.stop() to use it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
-
Adds supervisor/deploy_health_check.py: run_health_check() runs the three pre-cutover checks in order, stopping at the first failure (03-C: "任一检查项 fail 立即整体判定 health_check_failed，不等待其余检查项跑完") and persisting a deploy_health_signal row for every check that actually ran. - process_alive: polls deploy_supervisor.is_alive() up to HEALTH_CHECK_STARTUP_TIMEOUT. - http_health: hits the candidate's /api/health -- added to backend_server.py (U-037 deliberately left this for U-038). Not just "process exists" -- runs a real SELECT 1 to catch a wedged sqlite connection a live-but-stuck process would otherwise pass. - canary_task: a real (not mocked) project+task creation round-trip against the candidate over HTTP -- no real Org Graph/claude calls, since c_task_create's synchronous 'queued' landing is already a deterministic, always-should-pass result, which is what a canary needs. Timeouts are the MVP-fixed drafts from 05 §5.9, explicitly flagged there as needing real calibration later, not hardcoded as authoritative. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
-
Adds supervisor/deploy_stage.py: stage_deployment() creates a deployment row, git-worktree-checks-out the target commit into a fresh directory (--detach, never in-place git pull), copies the production DB to a one-time candidate copy, and Popens a real candidate backend process on a free port. Return value follows 04-P §4.6's state-machine table (status after the call is 'candidate_starting') rather than 03-C's one-line 'staged' summary -- 04-P's own trigger text ("c_deploy_supervisor_stage 完成" moves the row to candidate_starting) is the more detailed/authoritative source when the two disagree, same precedent used throughout this project. Found and closed two real gaps along the way: - deployment's own field list never included where a candidate's checkout directory lives -- added candidate_instance_dir to schema.py (also what the NEXT staging call's previous_instance_dir bootstraps from once a deployment reaches 'finalized'). - backend_server.py had no CLI entry point at all since U-007 -- create_server() was only ever instantiated in-process by tests. Added main() (argparse + serve_forever + SIGTERM->shutdown() from a separate thread, per http.server's own same-thread-deadlock warning). RealCandidateLaunchTest spawns and HTTP-probes a genuine `python3 -m src.backend_server` subprocess against a real git-worktree checkout of this repo (not mocked) -- caught a real ordering issue doing so: git worktree only sees committed content, so testing before this Unit's own commit landed would check out backend_server.py without main() at all. Fixed by using `git stash create` (builds a commit object from the current working tree without touching HEAD/index) as the commit-to-test-against, falling back to plain HEAD once the tree is clean post-commit. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> -
Ports graphloop's test_isolated_coding_workspace.py/test_workspace_security.py/ test_change_manifest.py/test_selective_apply.py into supervisor/isolation/ (only the import path changed -- the graphlemon modules were already adapted verbatim from graphloop, confirmed API-identical) plus a few extra edge cases the originals didn't cover. 52 tests green, replacing what had been a single manual smoke test since these modules were staged. Adds supervisor/deploy_supervisor.py: process lifecycle only (PID file, atomic status file, start/stop/is_running), no deploy decision logic yet -- that's U-037+'s job, injected via the poll_fn hook run_foreground() calls on a schedule. PID/status conventions adapted (not imported) from graphloop's hermes_watchdog.py atomic-write pattern. Fail-closed: any exit path (normal, SIGTERM, or an exception from poll_fn) unconditionally writes status='stopped' and removes the PID file, so is_running() never lies. RealSubprocessLifecycleTest actually spawns and stops a real `python3 -m supervisor.deploy_supervisor` process rather than mocking it, per this Unit's own "能独立启停" Done criterion. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
-
Establishes the first commit for graphlemon before U-001 work begins: architecture/SDD docs (including the PH5 self-hosting design already folded into 01-M/03-C/04-P/mvcp-contract v1.7.0), the A-1 spike results, the graphloop-adapted supervisor/isolation utilities, PH3 wireframe prototypes (now cross-referenced from 02-V), and this session's development-plan review (U-024 dependency split, U-035 regression requirement, new U-004.5 e2e-harness unit, src/ directory convention). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>