-
Attempts 7 and 11 both failed at exactly the same spot: process_alive read 'ok' within milliseconds of Popen, then http_health hit ECONNREFUSED in the same second and gave up. Real cause: process_alive only confirms the process EXISTS, not that backend_server.py has finished importing modules and actually bound its listen socket -- a real, non-zero startup gap. The old check_http_health did one urlopen(timeout=X) call; that timeout only bounds how long to wait for a response once connected, not how long to wait for the port to start accepting connections at all -- ECONNREFUSED fires essentially instantly when nothing's listening yet, regardless of timeout. This touches an explicit 03-C design statement ("不重试——重试属于给坏版本 第二次机会"), so confirmed with the user before changing it rather than just reinterpreting the policy unilaterally. Resolution: tolerating normal startup latency within ONE health-check pass isn't "retrying a failed check" -- check_process_alive already does exactly this (polls up to 30s for the pid to exist). http_health now does the same, polling within its own timeout for connection-refused-class errors specifically. A genuine non-2xx HTTP response is still a definitive answer from a live server and still fails immediately, no polling -- that boundary is unchanged. Two new real regression tests: a real server that starts listening 0.3s late is correctly waited for and passes; a real 500 response still fails immediately without being mistaken for "not up yet". 564 tests green (441 src + 123 supervisor). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> -
The 7th real self-hosting attempt was the first to actually reach a real health check against a real candidate process -- it correctly failed and triggered rollback_path_a for real: candidate terminated, git revert landed cleanly. But the candidate's `git worktree add --detach` checkout (candidate_instance_dir, tracked since U-037 specifically to know where this lives) was never removed -- left permanently on disk and still registered in `git worktree list`. A confirmed-bad candidate is never needed again after path A, unlike a successful cutover's candidate (which correctly becomes previous_instance_dir, kept until the monitoring window closes) -- these are different lifecycles that shouldn't share the same "leave it" behavior. rollback_path_a gains an optional candidate_instance_dir parameter (default None, existing tests unaffected) and removes the worktree before git revert when provided. self_hosting_e2e.py now threads through the value stage_deployment() already returns. New regression test uses a real `git worktree add --detach` checkout and confirms both the directory and its git worktree registration are gone after rollback. Also cleaned up 3 orphaned processes left by earlier test_deploy_rollback.py runs (backing temp dirs already gone) and the real leftover candidate worktree from this run. 561 tests green (440 src + 121 supervisor). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
-
The third real self-hosting attempt (first to actually reach stage_deployment itself, past the earlier bootstrap/commit bugs) hit an uncaught FileNotFoundError: _copy_prod_db() assumed prod_db_path always exists and shutil.copy2()'d it unconditionally. graphlemon has never actually run as a persistent service, so ~/.graphlemon/graphlemon.db never existed. stage_deployment() already handles this exact class of "no prior state" bootstrap case for previous_commit_hash/previous_instance_dir -- prod_db_path never got the same treatment, and every existing test seeds a prod db in setUp so this was never exercised. Fix: when prod_db_path doesn't exist, create a fresh schema'd db instead of copying -- semantically correct, since a genuinely-first deployment has no production data to inherit. New regression test (BootstrapProdDbTest) covers it. 555 tests green (435 src + 120 supervisor). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
-
Adds supervisor/self_hosting_e2e.py: run_self_hosting_closed_loop() chains every PH5 module built this session (U-034~U-041) with the Graph Engine's own scheduler (U-032) into one real, callable path matching self-development- feasibility.md's 9-step scenario -- self-target Goal -> Planner/Coder/ Validator -> core_engine classification -> stage -> health check -> cutover (or rollback path A if health check fails) -> monitor until either monitoring_window_clean (finalized) or anomaly_detected (rollback path B). Both real outcome paths are branches of the same call, not two separate scripted scenarios. Explicitly documented as this acceptance scenario's own orchestration glue, not a claim about Deploy Supervisor's eventual production poll loop (U-036's discovery-loop hook is still unfilled; building that is future work). Mocked wiring tests (6, run in normal regression) stub each already independently-tested module boundary and only verify the branching logic. The real, literal acceptance test is built and ready (RealSelfHostingClosedLoopTest, reproduces the document's own "loop_round > 3 should be >= 3" example) but gated behind RUN_REAL_SELF_HOSTING_E2E=1 -- deliberately a separate, distinctly-named env var from this project's usual RUN_REAL_AGENT_E2E, since this is the highest-blast-radius action in the project (can replace the running graphlemon process and git-revert commits on this real repo). Per this Unit's own explicit build instruction, not executed as part of implementing it -- doing so requires checking with the user first. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
-
Adds supervisor/deploy_rollback.py: rollback_path_a (pre-cutover, health check failed -- terminate candidate, delete its one-time DB copy, git revert, record a bug_record(core_engine) audit trail) and rollback_path_b (post-cutover, anomaly detected -- stop the bad Live, revive previous_instance_dir, re-run the same three health-check signals directly against it, git revert only once re-verification passes). Design choices follow established conventions rather than inventing new ones: no unified rollback() entry point (deployment has no PID columns, same reason deploy_stage/deploy_cutover already require callers to pass pids explicitly); ROLLBACK_FAILED is the raised error name while the persisted deployment.status is 'blocked_pending_human' (same pattern CheckoutFailedError/CutoverFailedError already use). git revert conflicts fold into the same rollback-itself-failed fallback both paths share, per design doc §10 open question 3's own recommendation, with `git revert --abort` run afterward so a failed attempt doesn't leave the repo mid-conflict. This is PH5's highest-rework-risk Unit per development-plan.md's own flag, so its own "rollback itself fails" fallback is tested for real, not assumed: a genuinely broken previous_instance_dir (a real empty directory with no Python package in it) that really fails to bind its port, and a genuinely conflicting git history (a later commit touching the same line the self-modification changed) that really fails `git revert` -- both correctly land on blocked_pending_human without retrying forever or silently succeeding, and the conflict case leaves the repo clean afterward. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
-
Adds supervisor/deploy_monitor.py: run_monitor_cycle() is one sampling pass, called repeatedly by Deploy Supervisor's own poll loop (03-C's own text: not a one-shot call). process_alive/http_health/canary_task/ validator_approval_rate sampled in order, stopping at the first anomaly. validator_approval_rate's "approve rate" has no literal definition anywhere in 01-M/03-C/04-P (no such execution_event type exists) -- derived instead from a verified structural fact in scheduler.finish_task_lifecycle: PASS closes straight to done with no bug_record ever created, FAIL/INCONCLUSIVE always creates one via judge_root_cause. A task counts as approved iff no bug_record(source='validator') exists for it. Baseline is computed by the same function looking back MONITORING_WINDOW_HOURS from cutover_at rather than persisted as a new column; baseline=None (cold start) never flags an anomaly rather than defaulting to 0. Crash-count/consecutive-http-fail windows are approximated by sample count (N = CRASH_WINDOW / MONITOR_SAMPLE_INTERVAL) rather than a true rolling time window -- documented simplification, correct as long as sampling stays roughly interval-paced. Caught my own design misreading via the tests: process_alive's monitoring- phase anomaly condition is MAX_CRASH_COUNT crashes within CRASH_WINDOW (a crash-loop detector), not "currently not alive" -- unlike U-038's health check where any single failure fails closed. A single dead reading during monitoring is recorded but must not trigger an immediate rollback by itself. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
-
Adds supervisor/deploy_cutover.py: stops the old Live process, retires the health-check-mode candidate, and relaunches the same already-verified candidate checkout bound to LIVE_PORT/PROD_DB_PATH -- http.server has no live socket-rebind operation, so "switch to production port/db" is implemented as stop-then-relaunch-from-the-same-dir rather than an in-place rebind (functionally equivalent, the only mechanism stdlib actually offers). Adds LIVE_PORT=8420 to 05 §5.9's MVP-fixed parameter table -- the one parameter that table was missing (frontend/app.js never needed one, same- origin relative fetches don't care, but successive deployments need to keep landing on the same port for a human bookmark to keep working). "在途 work_node_execution 重启后可恢复" is satisfied by relying on the existing REQ-06 recovery machinery unchanged -- cutover doesn't add any special-case logic, a cutover-triggered restart is just another interruption REQ-06 already survives. A real (not mocked) test caught a real bug: deploy_supervisor.is_alive()'s os.kill(pid, 0) check can't distinguish "truly running" from "exited but never reaped by its own parent" -- a zombie's PID slot answers kill(pid, 0) successfully until waited on. U-036's own subprocess test never caught this because that skeleton deletes its own PID file before exiting, sidestepping is_alive() entirely; deploy_cutover's test is the first one to actually poll for a stopped process to disappear, and it hung until fixed. Added reap_if_exited() (real os.waitpid for processes we spawned ourselves, falls back to is_alive() for pids from a prior session where ChildProcessError means we're not the real parent) and switched deploy_cutover._stop_process, deploy_health_check.check_process_alive, and deploy_supervisor.stop() to use it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
-
Adds supervisor/deploy_health_check.py: run_health_check() runs the three pre-cutover checks in order, stopping at the first failure (03-C: "任一检查项 fail 立即整体判定 health_check_failed，不等待其余检查项跑完") and persisting a deploy_health_signal row for every check that actually ran. - process_alive: polls deploy_supervisor.is_alive() up to HEALTH_CHECK_STARTUP_TIMEOUT. - http_health: hits the candidate's /api/health -- added to backend_server.py (U-037 deliberately left this for U-038). Not just "process exists" -- runs a real SELECT 1 to catch a wedged sqlite connection a live-but-stuck process would otherwise pass. - canary_task: a real (not mocked) project+task creation round-trip against the candidate over HTTP -- no real Org Graph/claude calls, since c_task_create's synchronous 'queued' landing is already a deterministic, always-should-pass result, which is what a canary needs. Timeouts are the MVP-fixed drafts from 05 §5.9, explicitly flagged there as needing real calibration later, not hardcoded as authoritative. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
-
Adds supervisor/deploy_stage.py: stage_deployment() creates a deployment row, git-worktree-checks-out the target commit into a fresh directory (--detach, never in-place git pull), copies the production DB to a one-time candidate copy, and Popens a real candidate backend process on a free port. Return value follows 04-P §4.6's state-machine table (status after the call is 'candidate_starting') rather than 03-C's one-line 'staged' summary -- 04-P's own trigger text ("c_deploy_supervisor_stage 完成" moves the row to candidate_starting) is the more detailed/authoritative source when the two disagree, same precedent used throughout this project. Found and closed two real gaps along the way: - deployment's own field list never included where a candidate's checkout directory lives -- added candidate_instance_dir to schema.py (also what the NEXT staging call's previous_instance_dir bootstraps from once a deployment reaches 'finalized'). - backend_server.py had no CLI entry point at all since U-007 -- create_server() was only ever instantiated in-process by tests. Added main() (argparse + serve_forever + SIGTERM->shutdown() from a separate thread, per http.server's own same-thread-deadlock warning). RealCandidateLaunchTest spawns and HTTP-probes a genuine `python3 -m src.backend_server` subprocess against a real git-worktree checkout of this repo (not mocked) -- caught a real ordering issue doing so: git worktree only sees committed content, so testing before this Unit's own commit landed would check out backend_server.py without main() at all. Fixed by using `git stash create` (builds a commit object from the current working tree without touching HEAD/index) as the commit-to-test-against, falling back to plain HEAD once the tree is clean post-commit. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> -
Ports graphloop's test_isolated_coding_workspace.py/test_workspace_security.py/ test_change_manifest.py/test_selective_apply.py into supervisor/isolation/ (only the import path changed -- the graphlemon modules were already adapted verbatim from graphloop, confirmed API-identical) plus a few extra edge cases the originals didn't cover. 52 tests green, replacing what had been a single manual smoke test since these modules were staged. Adds supervisor/deploy_supervisor.py: process lifecycle only (PID file, atomic status file, start/stop/is_running), no deploy decision logic yet -- that's U-037+'s job, injected via the poll_fn hook run_foreground() calls on a schedule. PID/status conventions adapted (not imported) from graphloop's hermes_watchdog.py atomic-write pattern. Fail-closed: any exit path (normal, SIGTERM, or an exception from poll_fn) unconditionally writes status='stopped' and removes the PID file, so is_running() never lies. RealSubprocessLifecycleTest actually spawns and stops a real `python3 -m supervisor.deploy_supervisor` process rather than mocking it, per this Unit's own "能独立启停" Done criterion. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
-
Establishes the first commit for graphlemon before U-001 work begins: architecture/SDD docs (including the PH5 self-hosting design already folded into 01-M/03-C/04-P/mvcp-contract v1.7.0), the A-1 spike results, the graphloop-adapted supervisor/isolation utilities, PH3 wireframe prototypes (now cross-referenced from 02-V), and this session's development-plan review (U-024 dependency split, U-035 regression requirement, new U-004.5 e2e-harness unit, src/ directory convention). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>