宜博 / lemongraph

check_canary_task() creates a real project + task through the candidate's
own HTTP API to verify the write path works, but only ever cleaned up its
throwaway git directory -- the DB rows it created via the API were never
removed. Every real health check permanently grew whatever DB the candidate
was using by one dead project (pointing at an already-deleted temp dir) and
one task stuck at 'queued' forever.

Found by inspecting the real ~/.graphlemon/graphlemon.db, which had 5 such
leaked 'deploy-canary' projects from earlier real health-check runs (now
archived by hand). Fix: archive the created project in a finally block that
covers every return path, including task-creation/verification failures
where only the project (not the task) got created. Best-effort -- the
archive call failing must not override the real check result.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…10c5-main-checkout-drift

New target projects never got .claude/agents/*.md deployed, so their first
Planner call always failed immediately (claude --agent planner finds nothing
in cwd) -- self-target usage never noticed since those files are already
committed as part of this repo. create_project() now deploys and commits
them alongside the existing .graphlemon/ init commit.

Also: unroutable execution failures were recorded as a bare
blocked_reason='execution_error' with no way to see what actually broke.
scheduler._block_task()/task_execution._mark_blocked() now persist the real
exception message into execution_event.payload (blocked_reason itself is
unchanged -- budget_guardrail matches its exact value elsewhere).

Found via a real console Start-button failure on a freshly created project;
see docs/implementation-gap-analysis.md for the full writeup.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Users had no way to actually start a queued goal from the console --
task.status just sits at 'queued' forever. Traced the root cause through
03-C: c_task_create's own contract ("若 max_concurrent_goals 已达上限，
任务落 queued 但不触发调度") implies task creation auto-triggers
scheduling otherwise, and 03-C/04-P repeatedly assume an always-on Org
Graph scheduler that watches the queue -- src/scheduler.py builds the
mechanism but nothing ever spins it up as a running process, so no
"start" button was ever designed because none was supposed to be needed.

Design (confirmed with user, both recommended options chosen):
- Pause/Stop are a soft stop, not a real subprocess kill -- the current
  in-flight `claude` call finishes naturally, only the next phase is
  withheld. Hard-killing would need Popen + PID tracking, out of scope.
- No new task.status enum value -- reuses blocked_pending_human with
  blocked_reason='user_paused'/'user_stopped', riding the existing
  manual_resume mechanism instead of a schema change.

New src/task_execution.py: run_or_resume_task_lifecycle() resumes from
whichever phase actually hasn't run yet (queued/planning_brd_prd_sdd/
coding), start_task_execution_in_background() runs it to completion on
its own daemon thread + sqlite connection, request_pause() blocks an
in-flight task. Extended /route's manual_resume handling so resuming
from a user-initiated pause/stop actually re-continues execution in the
background, not just flips the status field back.

Real gap found while building this: backend_server.create_server() never
stored db_path on the server object -- needed for the new background
thread to open its own connection to the same db.

22 new tests (14 src/test_task_execution.py + 8 TaskExecutionRouteTest in
test_backend_server.py, HTTP wiring mocked around the already-unit-tested
background thread mechanics). Verified against a real running server via
curl: /execute 409s a non-queued task, /pause 409s a queued task and a
double-pause, succeeds on a coding task. src/ suite: 496 tests, all green.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Two new console pages requested directly by the user, beyond 02-V's
original declared non-goal ("不做完整多项目管理 UI（项目列表/切换）"):

- v_project_list: full project list + rename + archive/reactivate.
  "Import an existing directory with real code" needs no new code --
  project_scaffold.create_project() already only ever adds .graphlemon/
  inside the target folder. New: src/project_query.py (list/get/update),
  GET/PUT/DELETE /api/projects[/:id]. DELETE archives (status='archived'),
  never a physical row delete -- project_id is FK-referenced from task/
  artifact/bug_record/etc., and 01-M's own active/archived enum already
  models this as soft-delete.

- v_goal_list: extends the existing v_task_board/v_task_detail rather than
  duplicating them. Added sort/order to task_query.list_tasks() (column
  whitelist, no injection surface) and edit/delete on v_task_detail, both
  restricted to status='queued' (task_query.TaskNotEditableError, HTTP 409)
  -- editing or deleting a goal after real work has referenced it would
  leave that history pointing at something that no longer matches reality.

Implemented directly rather than through graphlemon's own self-hosting
loop -- confirmed with the user first that implement_code() always uses a
hardcoded single-node manifest (no plan_work_graph() call for Coder), so a
multi-page/multi-endpoint feature like this would've been one unreviewable
Coder session with no way to decompose it.

Real bug found on first verification pass (not caught by the existing
_insert_task test helper, which never writes execution_event): a real
task_create.create_task() always writes an execution_event row in the same
transaction (01-M's FK, no ON DELETE CASCADE) -- even a never-touched
queued task already has one, so DELETE FROM task raised
sqlite3.IntegrityError. Fixed by clearing the task's own execution_event
rows first; new regression test goes through the real create_task() path.

474 tests green. Verified end-to-end against a real running server (every
new endpoint exercised via curl) since the Chrome extension is still
unavailable this session.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

The original CLAUDE.md described a pre-scaffolding repo with no commits and
open questions about A-2/critical path -- stale since PH0. Updates it to
point at git log as the actual status record, document the two independent
unittest discover roots (src/ and supervisor/), the real backend_server.py
CLI, the .graphlemon-worktrees/ isolation mechanism and its three real
incident notes, and a full repo layout reflecting what's actually there.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Attempt 13b was the first real run to get all the way through a real
cutover -- task lifecycle, classification, staging, health check (fixed
last commit), and a genuine blue-green cutover all succeeded for real. It
correctly landed in 'monitor' and then correctly couldn't finish: the real
production exit condition (deploy_monitor.MONITORING_WINDOW_HOURS=24h OR
MONITORING_MIN_TASK_COUNT=10 real tasks in the monitored project, NOT the
disposable per-cycle canary project) isn't practical to wait out in an
interactive session. This isn't a bug -- those are sensible real defaults
-- it's a testability gap: run_self_hosting_closed_loop already exposes
monitor_timeout_cycles as an override knob for exactly this kind of
situation, but never threaded the two constants that actually gate
"monitoring_window_clean" through to run_monitor_cycle.

Added monitor_window_hours/monitor_min_task_count params, defaulting to
deploy_monitor's real constants (production behavior unchanged), threaded
through to the actual run_monitor_cycle call. New assertion locks in that
overrides actually reach it.

Also documents the full run of 9 real bugs found and fixed across 13
attempts in development-plan.md's U-042 note.

564 tests green (441 src + 123 supervisor).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…3203-main-checkout-drift

…2750-main-checkout-drift

Attempts 7 and 11 both failed at exactly the same spot: process_alive read
'ok' within milliseconds of Popen, then http_health hit ECONNREFUSED in the
same second and gave up. Real cause: process_alive only confirms the
process EXISTS, not that backend_server.py has finished importing modules
and actually bound its listen socket -- a real, non-zero startup gap. The
old check_http_health did one urlopen(timeout=X) call; that timeout only
bounds how long to wait for a response once connected, not how long to
wait for the port to start accepting connections at all -- ECONNREFUSED
fires essentially instantly when nothing's listening yet, regardless of
timeout.

This touches an explicit 03-C design statement ("不重试——重试属于给坏版本
第二次机会"), so confirmed with the user before changing it rather than
just reinterpreting the policy unilaterally. Resolution: tolerating normal
startup latency within ONE health-check pass isn't "retrying a failed
check" -- check_process_alive already does exactly this (polls up to 30s
for the pid to exist). http_health now does the same, polling within its
own timeout for connection-refused-class errors specifically. A genuine
non-2xx HTTP response is still a definitive answer from a live server and
still fails immediately, no polling -- that boundary is unchanged.

Two new real regression tests: a real server that starts listening 0.3s
late is correctly waited for and passes; a real 500 response still fails
immediately without being mistaken for "not up yet".

564 tests green (441 src + 123 supervisor).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

This reverts commit b14c41371f803541c1f01debb0658ea13e816e0d.

…be04-main-checkout-drift

Attempt 10 crashed immediately in the fix committed last: git add -A --
. ':!.graphlemon-worktrees' throws "paths are ignored by one of your
.gitignore files" (exit 1, not just advisory text -- confirmed even with
advice.addIgnoredFile=false suppressed) the moment that path is ALSO
gitignored, which it now always is. Reproduced locally.

Plain `git add -A` (no explicit pathspec) already skips gitignored paths
correctly on its own -- the explicit exclude was redundant now that
.gitignore covers it, and actively conflicting. Simplified
ensure_worktree_committed() back down. Also added the same .gitignore
entry to test_work_node_run.py's _init_git_repo() fixture so real-worktree
tests match production config instead of coincidentally not hitting this.

562 tests green (441 src + 121 supervisor).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Attempts 8 and 9 both hit "local changes would be overwritten by merge"
again, despite the earlier _check_prd_consistency worktree-isolation fix.
Root cause is deeper: git worktrees deliberately SHARE project identity
with their main checkout (same .git, just another checkout) -- a real
`claude -p` call's memory:project write doesn't reliably respect the cwd it
was invoked with, and can land directly on the main checkout regardless of
which specific call triggered it. Worktree-wrapping one call (as done for
_check_prd_consistency) only moves around which call happens to trigger
this; it doesn't fix the underlying issue.

ensure_worktree_committed() now also runs against repo_path itself (not
just the worktree) right before every merge_worktree() call, in both
work_node_run.run_work_node() and e2e_pipeline._check_prd_consistency().
Same already-tested mechanism, now applied generically regardless of which
real call caused the drift or why.

Fixing this surfaced a second real bug in the fix itself: running `git add
-A` against repo_path while a node's own .graphlemon-worktrees/<id>/
checkout is still present on disk picks it up as a gitlink-like entry,
which then shows as "deleted" the instant remove_worktree() runs right
after. Fixed by excluding .graphlemon-worktrees/ from both the dirty-check
and the add via pathspec (not relying on the caller repo's own .gitignore),
plus added it to graphlemon's own .gitignore for correctness generally.

New regression test: test_drift_on_main_checkout_is_auto_committed_before_merge
reproduces a run_fn writing directly to repo_path instead of its worktree,
and confirms a second node's own merge isn't affected by it.

562 tests green (441 src + 121 supervisor).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

The 7th real self-hosting attempt was the first to actually reach a real
health check against a real candidate process -- it correctly failed and
triggered rollback_path_a for real: candidate terminated, git revert landed
cleanly. But the candidate's `git worktree add --detach` checkout
(candidate_instance_dir, tracked since U-037 specifically to know where
this lives) was never removed -- left permanently on disk and still
registered in `git worktree list`. A confirmed-bad candidate is never
needed again after path A, unlike a successful cutover's candidate (which
correctly becomes previous_instance_dir, kept until the monitoring window
closes) -- these are different lifecycles that shouldn't share the same
"leave it" behavior.

rollback_path_a gains an optional candidate_instance_dir parameter
(default None, existing tests unaffected) and removes the worktree before
git revert when provided. self_hosting_e2e.py now threads through the
value stage_deployment() already returns. New regression test uses a real
`git worktree add --detach` checkout and confirms both the directory and
its git worktree registration are gone after rollback.

Also cleaned up 3 orphaned processes left by earlier test_deploy_rollback.py
runs (backing temp dirs already gone) and the real leftover candidate
worktree from this run.

561 tests green (440 src + 121 supervisor).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

This reverts commit d77320f8d5b0600c99dcb6969e868f6d348ce9fb.