宜博 / lemongraph

…c231-main-checkout-drift

_launch_candidate() spawns a real `python -m src.backend_server` process
(`revived`) to re-verify the previous version during a post-cutover
rollback. On the success path it's correctly left running -- it IS the new
Live. But on every failure path (never bound live_port, failed
re-verification, git revert conflict) nothing ever terminated it -- found
via 31 accumulated orphaned processes on this machine, one per failed
rollback_path_b call across many real test runs this session.

This is also the likely cause of RollbackPathBFailureTest's own reported
non-determinism (three independent real Validator sessions saw it fail,
never reproduced manually): a leaked process from an earlier failed run
can still be squatting on the port a later run's _find_free_port() picks,
producing a false "alive" health-check reading.

Fix: call deploy_cutover._stop_process(revived.pid) in both failure
branches before raising, mirroring the pattern cutover() already uses for
its own stop_timeout-bounded process teardown. Also fixes
RollbackPathBHappyPathTest's own matching leak (it re-appended bad_live.pid
a second time instead of tracking the actual revived process, a
copy-paste bug, not real cleanup) by capturing the real pid via a thin
_launch_candidate wrapper.

31 pre-existing orphaned processes killed by hand; verified 5 repeated
test runs after the fix leak zero new ones.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

check_canary_task() creates a real project + task through the candidate's
own HTTP API to verify the write path works, but only ever cleaned up its
throwaway git directory -- the DB rows it created via the API were never
removed. Every real health check permanently grew whatever DB the candidate
was using by one dead project (pointing at an already-deleted temp dir) and
one task stuck at 'queued' forever.

Found by inspecting the real ~/.graphlemon/graphlemon.db, which had 5 such
leaked 'deploy-canary' projects from earlier real health-check runs (now
archived by hand). Fix: archive the created project in a finally block that
covers every return path, including task-creation/verification failures
where only the project (not the task) got created. Best-effort -- the
archive call failing must not override the real check result.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Attempt 13b was the first real run to get all the way through a real
cutover -- task lifecycle, classification, staging, health check (fixed
last commit), and a genuine blue-green cutover all succeeded for real. It
correctly landed in 'monitor' and then correctly couldn't finish: the real
production exit condition (deploy_monitor.MONITORING_WINDOW_HOURS=24h OR
MONITORING_MIN_TASK_COUNT=10 real tasks in the monitored project, NOT the
disposable per-cycle canary project) isn't practical to wait out in an
interactive session. This isn't a bug -- those are sensible real defaults
-- it's a testability gap: run_self_hosting_closed_loop already exposes
monitor_timeout_cycles as an override knob for exactly this kind of
situation, but never threaded the two constants that actually gate
"monitoring_window_clean" through to run_monitor_cycle.

Added monitor_window_hours/monitor_min_task_count params, defaulting to
deploy_monitor's real constants (production behavior unchanged), threaded
through to the actual run_monitor_cycle call. New assertion locks in that
overrides actually reach it.

Also documents the full run of 9 real bugs found and fixed across 13
attempts in development-plan.md's U-042 note.

564 tests green (441 src + 123 supervisor).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Attempts 7 and 11 both failed at exactly the same spot: process_alive read
'ok' within milliseconds of Popen, then http_health hit ECONNREFUSED in the
same second and gave up. Real cause: process_alive only confirms the
process EXISTS, not that backend_server.py has finished importing modules
and actually bound its listen socket -- a real, non-zero startup gap. The
old check_http_health did one urlopen(timeout=X) call; that timeout only
bounds how long to wait for a response once connected, not how long to
wait for the port to start accepting connections at all -- ECONNREFUSED
fires essentially instantly when nothing's listening yet, regardless of
timeout.

This touches an explicit 03-C design statement ("不重试——重试属于给坏版本
第二次机会"), so confirmed with the user before changing it rather than
just reinterpreting the policy unilaterally. Resolution: tolerating normal
startup latency within ONE health-check pass isn't "retrying a failed
check" -- check_process_alive already does exactly this (polls up to 30s
for the pid to exist). http_health now does the same, polling within its
own timeout for connection-refused-class errors specifically. A genuine
non-2xx HTTP response is still a definitive answer from a live server and
still fails immediately, no polling -- that boundary is unchanged.

Two new real regression tests: a real server that starts listening 0.3s
late is correctly waited for and passes; a real 500 response still fails
immediately without being mistaken for "not up yet".

564 tests green (441 src + 123 supervisor).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

The 7th real self-hosting attempt was the first to actually reach a real
health check against a real candidate process -- it correctly failed and
triggered rollback_path_a for real: candidate terminated, git revert landed
cleanly. But the candidate's `git worktree add --detach` checkout
(candidate_instance_dir, tracked since U-037 specifically to know where
this lives) was never removed -- left permanently on disk and still
registered in `git worktree list`. A confirmed-bad candidate is never
needed again after path A, unlike a successful cutover's candidate (which
correctly becomes previous_instance_dir, kept until the monitoring window
closes) -- these are different lifecycles that shouldn't share the same
"leave it" behavior.

rollback_path_a gains an optional candidate_instance_dir parameter
(default None, existing tests unaffected) and removes the worktree before
git revert when provided. self_hosting_e2e.py now threads through the
value stage_deployment() already returns. New regression test uses a real
`git worktree add --detach` checkout and confirms both the directory and
its git worktree registration are gone after rollback.

Also cleaned up 3 orphaned processes left by earlier test_deploy_rollback.py
runs (backing temp dirs already gone) and the real leftover candidate
worktree from this run.

561 tests green (440 src + 121 supervisor).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

The third real self-hosting attempt (first to actually reach stage_deployment
itself, past the earlier bootstrap/commit bugs) hit an uncaught
FileNotFoundError: _copy_prod_db() assumed prod_db_path always exists and
shutil.copy2()'d it unconditionally. graphlemon has never actually run as a
persistent service, so ~/.graphlemon/graphlemon.db never existed.
stage_deployment() already handles this exact class of "no prior state"
bootstrap case for previous_commit_hash/previous_instance_dir -- prod_db_path
never got the same treatment, and every existing test seeds a prod db in
setUp so this was never exercised.

Fix: when prod_db_path doesn't exist, create a fresh schema'd db instead of
copying -- semantically correct, since a genuinely-first deployment has no
production data to inherit. New regression test (BootstrapProdDbTest) covers
it. 555 tests green (435 src + 120 supervisor).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Adds supervisor/self_hosting_e2e.py: run_self_hosting_closed_loop() chains
every PH5 module built this session (U-034~U-041) with the Graph Engine's
own scheduler (U-032) into one real, callable path matching self-development-
feasibility.md's 9-step scenario -- self-target Goal -> Planner/Coder/
Validator -> core_engine classification -> stage -> health check -> cutover
(or rollback path A if health check fails) -> monitor until either
monitoring_window_clean (finalized) or anomaly_detected (rollback path B).
Both real outcome paths are branches of the same call, not two separate
scripted scenarios. Explicitly documented as this acceptance scenario's own
orchestration glue, not a claim about Deploy Supervisor's eventual
production poll loop (U-036's discovery-loop hook is still unfilled;
building that is future work).

Mocked wiring tests (6, run in normal regression) stub each already
independently-tested module boundary and only verify the branching logic.

The real, literal acceptance test is built and ready (RealSelfHostingClosedLoopTest,
reproduces the document's own "loop_round > 3 should be >= 3" example) but
gated behind RUN_REAL_SELF_HOSTING_E2E=1 -- deliberately a separate,
distinctly-named env var from this project's usual RUN_REAL_AGENT_E2E, since
this is the highest-blast-radius action in the project (can replace the
running graphlemon process and git-revert commits on this real repo).
Per this Unit's own explicit build instruction, not executed as part of
implementing it -- doing so requires checking with the user first.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Adds supervisor/deploy_rollback.py: rollback_path_a (pre-cutover, health
check failed -- terminate candidate, delete its one-time DB copy, git
revert, record a bug_record(core_engine) audit trail) and rollback_path_b
(post-cutover, anomaly detected -- stop the bad Live, revive
previous_instance_dir, re-run the same three health-check signals directly
against it, git revert only once re-verification passes).

Design choices follow established conventions rather than inventing new
ones: no unified rollback() entry point (deployment has no PID columns,
same reason deploy_stage/deploy_cutover already require callers to pass
pids explicitly); ROLLBACK_FAILED is the raised error name while the
persisted deployment.status is 'blocked_pending_human' (same pattern
CheckoutFailedError/CutoverFailedError already use). git revert conflicts
fold into the same rollback-itself-failed fallback both paths share, per
design doc §10 open question 3's own recommendation, with `git revert
--abort` run afterward so a failed attempt doesn't leave the repo mid-conflict.

This is PH5's highest-rework-risk Unit per development-plan.md's own flag,
so its own "rollback itself fails" fallback is tested for real, not assumed:
a genuinely broken previous_instance_dir (a real empty directory with no
Python package in it) that really fails to bind its port, and a genuinely
conflicting git history (a later commit touching the same line the
self-modification changed) that really fails `git revert` -- both correctly
land on blocked_pending_human without retrying forever or silently
succeeding, and the conflict case leaves the repo clean afterward.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Adds supervisor/deploy_monitor.py: run_monitor_cycle() is one sampling
pass, called repeatedly by Deploy Supervisor's own poll loop (03-C's own
text: not a one-shot call). process_alive/http_health/canary_task/
validator_approval_rate sampled in order, stopping at the first anomaly.

validator_approval_rate's "approve rate" has no literal definition anywhere
in 01-M/03-C/04-P (no such execution_event type exists) -- derived instead
from a verified structural fact in scheduler.finish_task_lifecycle: PASS
closes straight to done with no bug_record ever created, FAIL/INCONCLUSIVE
always creates one via judge_root_cause. A task counts as approved iff no
bug_record(source='validator') exists for it. Baseline is computed by the
same function looking back MONITORING_WINDOW_HOURS from cutover_at rather
than persisted as a new column; baseline=None (cold start) never flags an
anomaly rather than defaulting to 0.

Crash-count/consecutive-http-fail windows are approximated by sample count
(N = CRASH_WINDOW / MONITOR_SAMPLE_INTERVAL) rather than a true rolling
time window -- documented simplification, correct as long as sampling stays
roughly interval-paced.

Caught my own design misreading via the tests: process_alive's monitoring-
phase anomaly condition is MAX_CRASH_COUNT crashes within CRASH_WINDOW (a
crash-loop detector), not "currently not alive" -- unlike U-038's health
check where any single failure fails closed. A single dead reading during
monitoring is recorded but must not trigger an immediate rollback by itself.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Adds supervisor/deploy_cutover.py: stops the old Live process, retires the
health-check-mode candidate, and relaunches the same already-verified
candidate checkout bound to LIVE_PORT/PROD_DB_PATH -- http.server has no
live socket-rebind operation, so "switch to production port/db" is
implemented as stop-then-relaunch-from-the-same-dir rather than an in-place
rebind (functionally equivalent, the only mechanism stdlib actually offers).

Adds LIVE_PORT=8420 to 05 §5.9's MVP-fixed parameter table -- the one
parameter that table was missing (frontend/app.js never needed one, same-
origin relative fetches don't care, but successive deployments need to keep
landing on the same port for a human bookmark to keep working).

"在途 work_node_execution 重启后可恢复" is satisfied by relying on the
existing REQ-06 recovery machinery unchanged -- cutover doesn't add any
special-case logic, a cutover-triggered restart is just another interruption
REQ-06 already survives.

A real (not mocked) test caught a real bug: deploy_supervisor.is_alive()'s
os.kill(pid, 0) check can't distinguish "truly running" from "exited but
never reaped by its own parent" -- a zombie's PID slot answers kill(pid, 0)
successfully until waited on. U-036's own subprocess test never caught this
because that skeleton deletes its own PID file before exiting, sidestepping
is_alive() entirely; deploy_cutover's test is the first one to actually poll
for a stopped process to disappear, and it hung until fixed. Added
reap_if_exited() (real os.waitpid for processes we spawned ourselves, falls
back to is_alive() for pids from a prior session where ChildProcessError
means we're not the real parent) and switched deploy_cutover._stop_process,
deploy_health_check.check_process_alive, and deploy_supervisor.stop() to use
it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Adds supervisor/deploy_health_check.py: run_health_check() runs the three
pre-cutover checks in order, stopping at the first failure (03-C: "任一检查项
fail 立即整体判定 health_check_failed，不等待其余检查项跑完") and persisting
a deploy_health_signal row for every check that actually ran.

- process_alive: polls deploy_supervisor.is_alive() up to HEALTH_CHECK_STARTUP_TIMEOUT.
- http_health: hits the candidate's /api/health -- added to backend_server.py
  (U-037 deliberately left this for U-038). Not just "process exists" --
  runs a real SELECT 1 to catch a wedged sqlite connection a live-but-stuck
  process would otherwise pass.
- canary_task: a real (not mocked) project+task creation round-trip against
  the candidate over HTTP -- no real Org Graph/claude calls, since
  c_task_create's synchronous 'queued' landing is already a deterministic,
  always-should-pass result, which is what a canary needs.

Timeouts are the MVP-fixed drafts from 05 §5.9, explicitly flagged there as
needing real calibration later, not hardcoded as authoritative.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Adds supervisor/deploy_stage.py: stage_deployment() creates a deployment row,
git-worktree-checks-out the target commit into a fresh directory (--detach,
never in-place git pull), copies the production DB to a one-time candidate
copy, and Popens a real candidate backend process on a free port.

Return value follows 04-P §4.6's state-machine table (status after the call
is 'candidate_starting') rather than 03-C's one-line 'staged' summary --
04-P's own trigger text ("c_deploy_supervisor_stage 完成" moves the row to
candidate_starting) is the more detailed/authoritative source when the two
disagree, same precedent used throughout this project.

Found and closed two real gaps along the way:
- deployment's own field list never included where a candidate's checkout
  directory lives -- added candidate_instance_dir to schema.py (also what
  the NEXT staging call's previous_instance_dir bootstraps from once a
  deployment reaches 'finalized').
- backend_server.py had no CLI entry point at all since U-007 -- create_server()
  was only ever instantiated in-process by tests. Added main() (argparse +
  serve_forever + SIGTERM->shutdown() from a separate thread, per
  http.server's own same-thread-deadlock warning).

RealCandidateLaunchTest spawns and HTTP-probes a genuine
`python3 -m src.backend_server` subprocess against a real git-worktree
checkout of this repo (not mocked) -- caught a real ordering issue doing so:
git worktree only sees committed content, so testing before this Unit's own
commit landed would check out backend_server.py without main() at all.
Fixed by using `git stash create` (builds a commit object from the current
working tree without touching HEAD/index) as the commit-to-test-against,
falling back to plain HEAD once the tree is clean post-commit.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Ports graphloop's test_isolated_coding_workspace.py/test_workspace_security.py/
test_change_manifest.py/test_selective_apply.py into supervisor/isolation/
(only the import path changed -- the graphlemon modules were already adapted
verbatim from graphloop, confirmed API-identical) plus a few extra edge
cases the originals didn't cover. 52 tests green, replacing what had been a
single manual smoke test since these modules were staged.

Adds supervisor/deploy_supervisor.py: process lifecycle only (PID file,
atomic status file, start/stop/is_running), no deploy decision logic yet --
that's U-037+'s job, injected via the poll_fn hook run_foreground() calls on
a schedule. PID/status conventions adapted (not imported) from graphloop's
hermes_watchdog.py atomic-write pattern. Fail-closed: any exit path
(normal, SIGTERM, or an exception from poll_fn) unconditionally writes
status='stopped' and removes the PID file, so is_running() never lies.
RealSubprocessLifecycleTest actually spawns and stops a real
`python3 -m supervisor.deploy_supervisor` process rather than mocking it,
per this Unit's own "能独立启停" Done criterion.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Establishes the first commit for graphlemon before U-001 work begins:
architecture/SDD docs (including the PH5 self-hosting design already
folded into 01-M/03-C/04-P/mvcp-contract v1.7.0), the A-1 spike results,
the graphloop-adapted supervisor/isolation utilities, PH3 wireframe
prototypes (now cross-referenced from 02-V), and this session's
development-plan review (U-024 dependency split, U-035 regression
requirement, new U-004.5 e2e-harness unit, src/ directory convention).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>