宜博 / lemongraph

…437e-main-checkout-drift

_launch_candidate() starts backend_server against a COPY of the prod db for
health-checking only; without --disable-reconciliation-loop it would find
resumable tasks in that copy and dispatch real claude -p calls before Deploy
Supervisor decides whether the candidate is even good. Adds the flag plus a
mocked-Popen regression test (LaunchCandidateArgsTest) so a future regression
here fails fast instead of surfacing as a hang or a real incident.

Also folds this into docs/todo/scheduler-production-polling-loop-design.md
(§6.1) as part of the FR-2 design record.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Recovers tasks abandoned mid-flight (status in queued/planning_brd_prd_sdd/
coding/validating with no recent execution_event) by re-driving them through
task_execution.run_or_resume_task_lifecycle on a background poll, guarded by
an in-process in-flight lock to avoid double dispatch. Responds to a real
2026-08-06 incident where a task sat stuck in 'validating' for 15+ hours with
nothing polling it.

Scoped to FR-2 only from docs/todo/scheduler-production-polling-loop-
requirements.md; FR-1 (new-goal auto-dispatch) stays out per that doc's own
non-goals -- it's a separate, larger autonomy-boundary decision. Design
recorded in docs/todo/scheduler-production-polling-loop-design.md.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Per CLAUDE.md's docs/ role split (todo = pending, done = shipped into the
SDD/code): all seven Graph Engineering series items with a real
implementation (B, D, E, F, G, H -- A had no separate doc, resolved
directly in graphlemon_architecture.md; C is explicitly won't-do) now have
their requirements+design docs moved from docs/todo/ to docs/done/,
git mv'd to preserve history.

Every cross-reference to the 12 moved files -- across the other moved
docs themselves, graph-engineering-open-items.md, 2 Skill files, and 9
src/*.py docstrings/comments -- updated from docs/todo/<file> to
docs/done/<file>. Verified no stale docs/todo/ references to any of the
12 filenames remain anywhere in the repo.

graph-engineering-open-items.md itself stays in docs/todo/ (not moved) --
it's a living index that keeps gaining new entries from later
comparison-note/gap-analysis passes (H itself was added this way, and a
new item already appeared while this cleanup was in progress), not a
single finished conclusion. Its own header now reflects that all of
A/B/D/E/F/G/H are done and explains why it wasn't moved. Also fixed two
pre-existing stray path references (missing todo/done/ prefix entirely,
predating this cleanup) in its G section.

Full src suite (625 tests) passes -- this is a pure documentation/comment
change, no behavior touched.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Implements docs/todo/role-package-abstraction-design.md. Requirements
review (docs/todo/graph-engineering-open-items.md's H section) had already
corrected the requirements doc's own dependency analysis before this;
scope confirmed with user 2026-08-06: relax all 4 org_node CHECK
constraints in one pass, not staged.

Format is JSON, not YAML: 05 §5.7's pure-stdlib constraint rules out
PyYAML, and unlike Skill files (whose YAML frontmatter only Claude Code
itself ever parses, never this codebase's own Python), role.json genuinely
needs to be read by agent_profile.py -- json.load is the zero-risk choice
over a hand-rolled parser.

role.json is a REFERENCE manifest, not a content copy: identity.soul_md_path/
agents_md_path still point at the untouched agents/<role>/SOUL.md/AGENTS.md
files. This is what makes "deployed output must stay identical" close to
automatic -- render_agent_definition()'s body-assembly logic doesn't
change at all, only the frontmatter's data source does (hardcoded
FRONTMATTER dict -> parsed role.json). Verified byte-for-byte: diffed the
old (git HEAD) render_agent_definition() output against the new one for
all three roles, zero differences.

agent_profile.py: ORG_NODES is now discovered by scanning roles/ for
role.json files (_discover_role_ids()) instead of a hardcoded 3-tuple;
FRONTMATTER/_DESCRIPTIONS/_MIGRATED_FROM_PROFILE dicts are gone. Every
public function takes an optional roles_dir override specifically so a
test can prove "a new role needs zero code changes" against a throwaway
fixture directory (NewRoleWithoutCodeChangeTest) rather than asserting it
by inspection.

schema.py: skill.owner_org_node/task.assigned_org_node/work_graph_run.org_node/
agent_profile.org_node relaxed from CHECK(x IN ('planner','coder','validator'))
to CHECK(x != ''). Real blast radius found during design (bigger than the
requirements doc's own risk note): task/work_graph_run/skill are
referenced by several other tables' foreign keys, but since only the
parent tables' own CHECK text changes (child FK definitions are untouched),
only these 4 parent tables need rebuilding -- followed SQLite's documented
procedure for schema changes under active foreign keys (foreign_keys=OFF
for the duration, rebuild, foreign_key_check to confirm nothing went
dangling, foreign_keys=ON). migrate_relax_org_node_constraints() depends
on D's migrate_add_work_graph_fork_columns() having run (called
unconditionally, idempotent) so work_graph_run's rebuild has the right
column set either way. ux_work_graph_run_active and task's 3 indexes
verified to survive the rebuild.

work_graph_run_query.py's own separately-hardcoded _VALID_ORG_NODES
(a real dependency the requirements doc's risk section missed) now reuses
agent_profile.ORG_NODES instead of maintaining a second copy.

Explicitly NOT done (scoped out during requirements review, not an
oversight): frontend/app.js's 4 hardcoded 3-role spots (route regex, sort
order, dropdown, lookup object) -- a new role still won't be recognized by
the PH3 console. A `role` table with real FK referential integrity was
also not built; this uses the simpler "just relax the enum" approach.

Full src suite (623 tests) + supervisor suite (125 tests) pass with zero
regressions.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Implements docs/todo/work-graph-time-travel-debugging-design.md, backend
and console together (user confirmed 2026-08-06).

Key finding that shrank the v1 scope: ux_work_graph_run_active is a
PARTIAL unique index (only covers non-terminal statuses), not a blanket
one -- forking from a parent run that's already reached a terminal status
never collides with it at all. v1 restricts forking to exactly that case,
so no index/constraint change is needed, only two new nullable columns
(parent_run_id, fork_of_node_exec_id).

Resolved the requirements doc's FR-1 ambiguity ("override input_payload,
re-trigger execution") as OUTPUT SUBSTITUTION instead: the fork point's
own output_payload is replaced with the human-supplied content directly,
it is never re-executed via a new claude -p call -- only the strict
downstream subgraph actually re-runs. This sidesteps reconciling the
different build_prompt/input_payload conventions each caller uses
(Planner's artifact_type lookup, E's generic task_prompt field, F's jury)
-- work_graph_fork.py only needs the persisted graph_manifest and each
node's output_payload.

Implementation needs ZERO changes to the execution engine: pre-seeding
every non-downstream node as a 'done' work_node_execution row before
calling the unmodified run_graph() on the unmodified graph_manifest is
enough -- run_work_node()'s existing REQ-06 breakpoint-recovery
short-circuit (already-'done' nodes return their persisted output without
calling run_fn) does all the "reuse without re-executing" work for free.
_reconstruct_build_prompt supports two graph shapes generically from
persisted state alone (Planner's canonical doc chain via
e2e_pipeline._planner_prompt_builder, and E's generic task_prompt field);
anything else raises ForkGraphShapeUnsupportedError rather than producing
broken prompts.

New: src/work_graph_fork.py (create_fork/execute_fork/fork_work_graph_run/
start_fork_in_background, the last splitting fast DB-only setup from slow
real execution so an HTTP caller gets run_id back immediately, same
pattern task_execution.start_task_execution_in_background already uses).
work_graph_run gets parent_run_id/fork_of_node_exec_id (SCHEMA_SQL +
migrate_add_work_graph_fork_columns for existing databases).
work_graph_run_query.py/task_query.py surface both fields. New endpoint
POST /api/work-nodes/:node_exec_id/fork. Console: a "从这里分叉" form on
each 'done' node in v_run_detail, a fork badge linking back to the parent
run, and a 🔀 marker on v_graph_console's run list -- no diff/merge UI
(non-goal #3), read-only branch visibility only.

Verified end-to-end for real: started the actual backend, sent a real
HTTP fork request, confirmed the new run persisted with correct
parent_run_id/fork_of_node_exec_id lineage.

Tests cover requirements doc acceptance standards 1-3 (PRD swapped ->
SDD/Benchmark genuinely re-execute, BRD untouched; original run completely
unaffected; the same node forked twice produces independent branches) plus
schema migration idempotency, query-layer field surfacing, the HTTP route,
and 3 frontend smoke-test assertions. Full src suite (608 tests) + all 3
frontend smoke tests pass.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Implements docs/todo/validator-loop-until-dry-skill-design.md. Pure Skill +
Agent-definition wiring, no execution-engine changes -- the convergence
loop runs inside a single Work Node's one claude -p session (multi-turn
tool use), not a graph_manifest-level cycle, so topological_layers()/
run_graph() are untouched.

New Skill skills/unknown-scope-defect-sweep/SKILL.md: K=2 consecutive
no-new-finding rounds to stop, 20-round hard cap, dedup key must be a
structural identifier (file path + line, route path, package name) not a
natural-language description (the same finding gets reworded call to
call, so text-keyed dedup never converges), dedup set is "everything ever
seen" not "everything confirmed" (the common bug the 14-step reference
names: a rejected candidate reappearing as a false "new finding" every
round). Each candidate is verified via qa-task-validation's existing
PASS/FAIL/INCONCLUSIVE framework -- deliberately NOT root_cause_trace's
three-perspective jury (F), since that answers a different question
(root cause AFTER a failure) from this one (is this candidate real at
all) -- documented in the design doc's own tradeoff note.

Wired into agent_profile.py's FRONTMATTER["validator"]["skills"] and
agents/validator/AGENTS.md's "必须加载" section with a concrete trigger
signal (task description can't state an exact expected count/boundary
up front).

Only FR-4 (Skill wired into the rendered .claude/agents/validator.md) has
deterministic, mockable behavior -- new test
test_validator_references_unknown_scope_defect_sweep_skill. Acceptance
criteria 1-3 (actually finds everything, actually dedups correctly,
actually stops at the hard cap) need real multi-turn claude -p behavior,
out of scope for mocked tests (RUN_REAL_AGENT_E2E=1 territory), same
category of limitation as F's own real-concurrency-timing note.

Full src suite (587 tests) passes with zero regressions.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Reviewed docs/todo/role-package-abstraction-requirements.md against the
actual codebase before confirming direction. Two corrections to the
document's own risk analysis:

1. Its dependency list for agent_profile.ORG_NODES/FRONTMATTER was wrong
   in one place and missing the largest one: work_graph_orchestrate.py
   doesn't import agent_profile at all (only org_node_trigger.py and
   project_scaffold.py::deploy_agent_definitions() do -- smaller real
   footprint than claimed). But the document missed two real hardcoded
   3-role dependencies entirely: src/work_graph_run_query.py's
   _VALID_ORG_NODES, and -- the actually large one -- four separate
   hardcoded spots in frontend/app.js (a route regex, a sort-order array,
   a task-creation <option> dropdown, and a lookup object literal).
2. Acceptance criterion 1 ("a new role needs no code/schema changes")
   only covers schema.py + agent_profile.py per its own non-goal #1 (a new
   role doesn't need real Org Graph wiring) -- the console genuinely won't
   recognize it either way. Flagged so the design doc states this
   explicitly instead of letting "no code changes" read as end-to-end.

Confirmed for design phase with these two additions folded in; the
document's own open questions (§8) are left for the design pass, not
pre-decided here.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Implements docs/todo/verifier-fan-out-jury-design.md. root_cause_trace.py:
_tally_perspectives() extracts the existing >=2/3 rule out of
judge_root_cause so the closure-based version and the new
judge_root_cause_via_work_graph() share one implementation, not two that
could drift apart. The fan-in vote stays plain Python (zero_token
aggregation per the design-dynamic-work-graph Skill's own edge_type
distinction) rather than becoming a fourth Work Node -- no reason to pay
for an Agent call just to count votes.

judge_root_cause_via_work_graph(): each perspective becomes an
independent Work Node with real schema-constrained structured output
({"category", "detail"} directly), not real_validator_perspective's
manual "CATEGORY:"/"DETAIL:" text-line parsing. judge_root_cause /
real_validator_perspective are kept, not removed -- documented as the
lightweight fallback for scenarios that don't need worktree/Work Graph
overhead.

org_level_loop.run_fix_loop (the one real production caller) now calls
the Work Graph version; the fixer/verify calls are unchanged flat
trigger_org_node calls.

Real bug found and fixed along the way: this is the first production code
path that actually triggers a run_graph() multi-node CONCURRENT layer
(every graph_manifest before F was single-node or a strict serial chain,
so G's ThreadPoolExecutor path had never really run in production).
scheduler.py::run_goal_to_completion and
task_execution.py::start_task_execution_in_background both created their
conn without check_same_thread=False -- fixed both, since their conn can
now reach run_graph()'s worker threads via run_fix_loop.

Test boundary migrated in test_org_level_loop.py/test_scheduler.py: the
3 perspective calls are now mocked at work_graph_execute.write_workflow_script/
execute_work_graph (matching test_work_graph_orchestrate.py's pattern)
instead of org_node_trigger.trigger_org_node. New tests cover: three real
independent Work Node executions, judgment-rule equivalence with the
closure version, clean per-node input_payload, and concurrent wall-clock
overlap across the three perspectives.

Full src suite (586 tests) + supervisor suite (125 tests) pass with zero
regressions.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Implements docs/todo/dynamic-work-graph-design-skill-design.md. G already
answered FR-2 (how to mark a parallel-eligible node group): no new field
needed -- topological_layers() infers independence purely from
depends_on/edges, so the new Skill's job is just "express real
dependencies honestly, don't add fake ones."

New Skill skills/design-dynamic-work-graph/SKILL.md: the node-boundary /
zero_token-vs-needs_judgment / per-node self-contained task_prompt
methodology, wired into agent_profile.py's FRONTMATTER for coder and
validator (Planner's own doc-chain path is untouched -- FR-4 only names
Coder/Validator).

work_graph_plan.py: new _GENERIC_PROMPT_TEMPLATE + generic_node_prompt_builder(),
used whenever org_node != 'planner'. Planner keeps its existing fixed
brd/prd/sdd/benchmark/... chain (_PROMPT_TEMPLATE) completely unchanged --
zero regression risk to the one real production planning path.

Scoped per the requirements doc's non-goal #1 ("prove the mechanism works,
don't cover every real scenario yet"): does NOT wire into
e2e_pipeline.py::implement_code()'s existing single-node production path.
New tests prove the mechanism itself: a Coder scenario that naturally
splits into 3 independent modules produces a correctly-shaped
graph_manifest, topological_layers() puts all three in one concurrent
layer (ties E directly to G), and generic_node_prompt_builder drives it
through run_graph() to completion.

Full src suite (580 tests) passes with zero regressions.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Implements docs/todo/work-graph-parallel-execution-design.md: run_graph()
now schedules graph_manifest nodes layer-by-layer (topological_layers(),
Kahn's algorithm grouped by round instead of flattened), running a
multi-node layer's nodes concurrently via ThreadPoolExecutor (default 4
workers) with a barrier between layers; a single-node layer stays
synchronous, matching every real graph_manifest observed so far.
topological_order() is now a thin flattening wrapper, unchanged behavior
(regression-tested).

_LockedConn/_FetchedRows makes the existing scattered conn.execute() call
sites in work_node_run.py/work_graph_orchestrate.py/execution_log.py safe
to call from multiple threads without touching those call sites -- note
the lock must cover the full execute()-through-fetchone()/fetchall()
cycle, not just execute() itself, or two threads can still collide on the
shared sqlite3 connection handle. That gap was caught for real (not
theoretically) by RunGraphRealSqliteConcurrencyTest during development --
committing the fixed version, not the first draft.

create_worktree() (work_node_run.py) gets a module-level lock around the
`git worktree add` call for concurrent layers touching the same repo.

New tests: layer grouping, same-layer concurrency + inter-layer barrier,
one node's failure not cancelling a concurrent sibling (WorkGraphLayerError
aggregates), breakpoint-recovery resume of a concurrent layer, a quota-wait
timeout coexisting with a sibling's real persisted output, real (non-mocked)
sqlite under concurrent writes across several rounds, and real concurrent
`git worktree add` calls. Full src (575 tests) + supervisor (125 tests)
suites pass with zero regressions.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…c231-main-checkout-drift

Three real self-hosting fixes (log content never written, skill detail
page missing real content, artifact list not clickable), two design
questions set aside per user's request (unified Planner-driven work graph
across all org agents, loop visualization on the graph), three bugs left
for a future batch, and two infrastructure bugs found and fixed along the
way (manual_resume not actually continuing execution for most block
reasons, run_goal_to_completion crashing instead of blocking on unrouted
exceptions). Also upgrades the RollbackPathBFailureTest flakiness note --
three independent real sessions have now hit it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…1a79-main-checkout-drift

…ea4d-main-checkout-drift

…d9ae-main-checkout-drift

…0a72-main-checkout-drift

…2b85-main-checkout-drift

…f2cf-main-checkout-drift

Second real self-fix (after order 2) -- 12 skills migrated verbatim from
~/.hermes/profiles/, Validator caught and redacted a leaked Feishu API
token in 3 of them via byte-diff against source. Also records two
operational lessons from actually running it (a real network blip, and my
own mistake killing a daemon thread by running it from a one-shot script)
and a flaky supervisor test failure that didn't reproduce on independent
recheck.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Unlike order 1 (fixed by editing code directly), this one was resolved by
actually running graphlemon's own Planner->Coder->Validator loop against
its own repo (task 43dc7333, see 3bebe3b onward) -- the first real
self-fix demonstration this doc's task list produced.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

check_canary_task() creates a real project + task through the candidate's
own HTTP API to verify the write path works, but only ever cleaned up its
throwaway git directory -- the DB rows it created via the API were never
removed. Every real health check permanently grew whatever DB the candidate
was using by one dead project (pointing at an already-deleted temp dir) and
one task stuck at 'queued' forever.

Found by inspecting the real ~/.graphlemon/graphlemon.db, which had 5 such
leaked 'deploy-canary' projects from earlier real health-check runs (now
archived by hand). Fix: archive the created project in a finally block that
covers every return path, including task-creation/verification failures
where only the project (not the task) got created. Best-effort -- the
archive call failing must not override the real check result.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

New target projects never got .claude/agents/*.md deployed, so their first
Planner call always failed immediately (claude --agent planner finds nothing
in cwd) -- self-target usage never noticed since those files are already
committed as part of this repo. create_project() now deploys and commits
them alongside the existing .graphlemon/ init commit.

Also: unroutable execution failures were recorded as a bare
blocked_reason='execution_error' with no way to see what actually broke.
scheduler._block_task()/task_execution._mark_blocked() now persist the real
exception message into execution_event.payload (blocked_reason itself is
unchanged -- budget_guardrail matches its exact value elsewhere).

Found via a real console Start-button failure on a freshly created project;
see docs/implementation-gap-analysis.md for the full writeup.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Users had no way to actually start a queued goal from the console --
task.status just sits at 'queued' forever. Traced the root cause through
03-C: c_task_create's own contract ("若 max_concurrent_goals 已达上限，
任务落 queued 但不触发调度") implies task creation auto-triggers
scheduling otherwise, and 03-C/04-P repeatedly assume an always-on Org
Graph scheduler that watches the queue -- src/scheduler.py builds the
mechanism but nothing ever spins it up as a running process, so no
"start" button was ever designed because none was supposed to be needed.

Design (confirmed with user, both recommended options chosen):
- Pause/Stop are a soft stop, not a real subprocess kill -- the current
  in-flight `claude` call finishes naturally, only the next phase is
  withheld. Hard-killing would need Popen + PID tracking, out of scope.
- No new task.status enum value -- reuses blocked_pending_human with
  blocked_reason='user_paused'/'user_stopped', riding the existing
  manual_resume mechanism instead of a schema change.

New src/task_execution.py: run_or_resume_task_lifecycle() resumes from
whichever phase actually hasn't run yet (queued/planning_brd_prd_sdd/
coding), start_task_execution_in_background() runs it to completion on
its own daemon thread + sqlite connection, request_pause() blocks an
in-flight task. Extended /route's manual_resume handling so resuming
from a user-initiated pause/stop actually re-continues execution in the
background, not just flips the status field back.

Real gap found while building this: backend_server.create_server() never
stored db_path on the server object -- needed for the new background
thread to open its own connection to the same db.

22 new tests (14 src/test_task_execution.py + 8 TaskExecutionRouteTest in
test_backend_server.py, HTTP wiring mocked around the already-unit-tested
background thread mechanics). Verified against a real running server via
curl: /execute 409s a non-queued task, /pause 409s a queued task and a
double-pause, succeeds on a coding task. src/ suite: 496 tests, all green.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Two new console pages requested directly by the user, beyond 02-V's
original declared non-goal ("不做完整多项目管理 UI（项目列表/切换）"):

- v_project_list: full project list + rename + archive/reactivate.
  "Import an existing directory with real code" needs no new code --
  project_scaffold.create_project() already only ever adds .graphlemon/
  inside the target folder. New: src/project_query.py (list/get/update),
  GET/PUT/DELETE /api/projects[/:id]. DELETE archives (status='archived'),
  never a physical row delete -- project_id is FK-referenced from task/
  artifact/bug_record/etc., and 01-M's own active/archived enum already
  models this as soft-delete.

- v_goal_list: extends the existing v_task_board/v_task_detail rather than
  duplicating them. Added sort/order to task_query.list_tasks() (column
  whitelist, no injection surface) and edit/delete on v_task_detail, both
  restricted to status='queued' (task_query.TaskNotEditableError, HTTP 409)
  -- editing or deleting a goal after real work has referenced it would
  leave that history pointing at something that no longer matches reality.

Implemented directly rather than through graphlemon's own self-hosting
loop -- confirmed with the user first that implement_code() always uses a
hardcoded single-node manifest (no plan_work_graph() call for Coder), so a
multi-page/multi-endpoint feature like this would've been one unreviewable
Coder session with no way to decompose it.

Real bug found on first verification pass (not caught by the existing
_insert_task test helper, which never writes execution_event): a real
task_create.create_task() always writes an execution_event row in the same
transaction (01-M's FK, no ON DELETE CASCADE) -- even a never-touched
queued task already has one, so DELETE FROM task raised
sqlite3.IntegrityError. Fixed by clearing the task's own execution_event
rows first; new regression test goes through the real create_task() path.

474 tests green. Verified end-to-end against a real running server (every
new endpoint exercised via curl) since the Chrome extension is still
unavailable this session.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Attempt 13b was the first real run to get all the way through a real
cutover -- task lifecycle, classification, staging, health check (fixed
last commit), and a genuine blue-green cutover all succeeded for real. It
correctly landed in 'monitor' and then correctly couldn't finish: the real
production exit condition (deploy_monitor.MONITORING_WINDOW_HOURS=24h OR
MONITORING_MIN_TASK_COUNT=10 real tasks in the monitored project, NOT the
disposable per-cycle canary project) isn't practical to wait out in an
interactive session. This isn't a bug -- those are sensible real defaults
-- it's a testability gap: run_self_hosting_closed_loop already exposes
monitor_timeout_cycles as an override knob for exactly this kind of
situation, but never threaded the two constants that actually gate
"monitoring_window_clean" through to run_monitor_cycle.

Added monitor_window_hours/monitor_min_task_count params, defaulting to
deploy_monitor's real constants (production behavior unchanged), threaded
through to the actual run_monitor_cycle call. New assertion locks in that
overrides actually reach it.

Also documents the full run of 9 real bugs found and fixed across 13
attempts in development-plan.md's U-042 note.

564 tests green (441 src + 123 supervisor).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Attempts 7 and 11 both failed at exactly the same spot: process_alive read
'ok' within milliseconds of Popen, then http_health hit ECONNREFUSED in the
same second and gave up. Real cause: process_alive only confirms the
process EXISTS, not that backend_server.py has finished importing modules
and actually bound its listen socket -- a real, non-zero startup gap. The
old check_http_health did one urlopen(timeout=X) call; that timeout only
bounds how long to wait for a response once connected, not how long to
wait for the port to start accepting connections at all -- ECONNREFUSED
fires essentially instantly when nothing's listening yet, regardless of
timeout.

This touches an explicit 03-C design statement ("不重试——重试属于给坏版本
第二次机会"), so confirmed with the user before changing it rather than
just reinterpreting the policy unilaterally. Resolution: tolerating normal
startup latency within ONE health-check pass isn't "retrying a failed
check" -- check_process_alive already does exactly this (polls up to 30s
for the pid to exist). http_health now does the same, polling within its
own timeout for connection-refused-class errors specifically. A genuine
non-2xx HTTP response is still a definitive answer from a live server and
still fails immediately, no polling -- that boundary is unchanged.

Two new real regression tests: a real server that starts listening 0.3s
late is correctly waited for and passes; a real 500 response still fails
immediately without being mistaken for "not up yet".

564 tests green (441 src + 123 supervisor).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…be04-main-checkout-drift

Attempt 10 crashed immediately in the fix committed last: git add -A --
. ':!.graphlemon-worktrees' throws "paths are ignored by one of your
.gitignore files" (exit 1, not just advisory text -- confirmed even with
advice.addIgnoredFile=false suppressed) the moment that path is ALSO
gitignored, which it now always is. Reproduced locally.

Plain `git add -A` (no explicit pathspec) already skips gitignored paths
correctly on its own -- the explicit exclude was redundant now that
.gitignore covers it, and actively conflicting. Simplified
ensure_worktree_committed() back down. Also added the same .gitignore
entry to test_work_node_run.py's _init_git_repo() fixture so real-worktree
tests match production config instead of coincidentally not hitting this.

562 tests green (441 src + 121 supervisor).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Attempts 8 and 9 both hit "local changes would be overwritten by merge"
again, despite the earlier _check_prd_consistency worktree-isolation fix.
Root cause is deeper: git worktrees deliberately SHARE project identity
with their main checkout (same .git, just another checkout) -- a real
`claude -p` call's memory:project write doesn't reliably respect the cwd it
was invoked with, and can land directly on the main checkout regardless of
which specific call triggered it. Worktree-wrapping one call (as done for
_check_prd_consistency) only moves around which call happens to trigger
this; it doesn't fix the underlying issue.

ensure_worktree_committed() now also runs against repo_path itself (not
just the worktree) right before every merge_worktree() call, in both
work_node_run.run_work_node() and e2e_pipeline._check_prd_consistency().
Same already-tested mechanism, now applied generically regardless of which
real call caused the drift or why.

Fixing this surfaced a second real bug in the fix itself: running `git add
-A` against repo_path while a node's own .graphlemon-worktrees/<id>/
checkout is still present on disk picks it up as a gitlink-like entry,
which then shows as "deleted" the instant remove_worktree() runs right
after. Fixed by excluding .graphlemon-worktrees/ from both the dirty-check
and the add via pathspec (not relying on the caller repo's own .gitignore),
plus added it to graphlemon's own .gitignore for correctness generally.

New regression test: test_drift_on_main_checkout_is_auto_committed_before_merge
reproduces a run_fn writing directly to repo_path instead of its worktree,
and confirms a second node's own merge isn't affected by it.

562 tests green (441 src + 121 supervisor).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

The 7th real self-hosting attempt was the first to actually reach a real
health check against a real candidate process -- it correctly failed and
triggered rollback_path_a for real: candidate terminated, git revert landed
cleanly. But the candidate's `git worktree add --detach` checkout
(candidate_instance_dir, tracked since U-037 specifically to know where
this lives) was never removed -- left permanently on disk and still
registered in `git worktree list`. A confirmed-bad candidate is never
needed again after path A, unlike a successful cutover's candidate (which
correctly becomes previous_instance_dir, kept until the monitoring window
closes) -- these are different lifecycles that shouldn't share the same
"leave it" behavior.

rollback_path_a gains an optional candidate_instance_dir parameter
(default None, existing tests unaffected) and removes the worktree before
git revert when provided. self_hosting_e2e.py now threads through the
value stage_deployment() already returns. New regression test uses a real
`git worktree add --detach` checkout and confirms both the directory and
its git worktree registration are gone after rollback.

Also cleaned up 3 orphaned processes left by earlier test_deploy_rollback.py
runs (backing temp dirs already gone) and the real leftover candidate
worktree from this run.

561 tests green (440 src + 121 supervisor).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

The 6th real self-hosting attempt (first to reach the full 4-node doc-writing
graph after the previous five bugs) hit a genuinely new failure: this real
graph_manifest had no edges at all, not just wrong field names -- worse than
the case _normalize_graph_manifest already handled. With zero dependency
info to normalize from, topological_order() scheduled 'benchmark' before
'sdd' (alphabetically first among zero-in-degree nodes), and build_prompt()
crashed with KeyError: 'sdd_content' trying to format a template needing
sdd's not-yet-produced output.

_PROMPT_TEMPLATE always asks for exactly the same fixed 4-node linear chain
-- that's a constant architectural fact, not something that should be
re-inferred from the LLM's output every call. _normalize_graph_manifest now
deterministically enforces brd->prd->sdd->benchmark whenever all four
canonical artifact_types are present, regardless of what depends_on/edges
the LLM did or didn't provide. Same "机制用代码" principle as
ensure_worktree_committed and the earlier normalize logic.

Three new regression tests: zero dependency info still gets the chain
enforced, node_id != artifact_type still resolves to the real node_id (not
a guessed string), and a partial set (missing one of the four types) is
correctly left alone rather than guessed at.

560 tests green (440 src + 120 supervisor).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

The 4th real self-hosting attempt (first to reach the actual doc-writing
Work Graph after the previous three bugs) hit a real git merge failure:
"local changes would be overwritten by merge", exit code 2.

Root cause: _check_prd_consistency() calls execute_work_graph() directly
against target_folder_path, completely bypassing work_node_run.run_work_node's
worktree isolation. It uses agentType='planner' with memory:project, which
writes a real update to .claude/agent-memory/planner/MEMORY.md in whatever
cwd it runs with. Every prior real run of this check only ever targeted a
throwaway clone, so this never mattered -- until self-targeting, where cwd
IS the main checkout: the memory write landed as an uncommitted local
change on master, which then blocked the SDD node's own (correctly
isolated) worktree merge right after it.

Fix: apply 05 §5.8's same unconditional self-target worktree rule here,
reusing work_node_run's existing create_worktree/ensure_worktree_committed/
merge_worktree/remove_worktree rather than inventing a new mechanism.
Non-self-target calls are unaffected (verified by both regression tests).

Also cleaned up the real, now-recovered state from the failed 4th attempt:
merged a real SDD-node worktree branch that got stranded uncommitted
(conflict was just two near-identical memory file wordings, resolved by
hand), and removed three orphaned worktrees left by earlier test runs.

557 tests green (437 src + 120 supervisor).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

The third real self-hosting attempt (first to actually reach stage_deployment
itself, past the earlier bootstrap/commit bugs) hit an uncaught
FileNotFoundError: _copy_prod_db() assumed prod_db_path always exists and
shutil.copy2()'d it unconditionally. graphlemon has never actually run as a
persistent service, so ~/.graphlemon/graphlemon.db never existed.
stage_deployment() already handles this exact class of "no prior state"
bootstrap case for previous_commit_hash/previous_instance_dir -- prod_db_path
never got the same treatment, and every existing test seeds a prod db in
setUp so this was never exercised.

Fix: when prod_db_path doesn't exist, create a fresh schema'd db instead of
copying -- semantically correct, since a genuinely-first deployment has no
production data to inherit. New regression test (BootstrapProdDbTest) covers
it. 555 tests green (435 src + 120 supervisor).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

A real self-hosting run (U-042, first real exercise of needs_worktree=True
since U-035 forces it unconditionally for self-target) exposed a real bug
RealGitWorktreeIsolationTest never caught: its own run_fn always committed
its edits, but a real Coder isn't instructed anywhere to git commit its own
changes. The Coder genuinely edited src/org_loop_route.py and verified it
in its own test_report -- but never committed, so merge_worktree()'s `git
merge` was a true no-op ("Already up to date"), and remove_worktree() then
silently discarded the dirty worktree along with the real work. code_diff
came out empty; the change never reached the branch at all.

Adds ensure_worktree_committed(): checks `git status --porcelain` in the
worktree before merging and auto-commits any dirty state found. A no-op
when the worktree is legitimately clean (e.g. a pure investigation task).
Same "判断用 Agent，机制用代码" principle already used elsewhere in this
codebase -- don't rely on persona prompt-engineering for a mechanical
correctness guarantee that the system itself should hold.

Two new regression tests: one reproducing the exact "edited but never
committed" scenario, one confirming a legitimately-clean worktree still
merges fine. 554 tests green (435 src + 119 supervisor).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Adds src/deployment_query.py (list_deployments/get_current_deployment/
get_deployment -- no project_id filter needed, deployment has no project_id
column since there's exactly one graphlemon self-target) + three new
backend_server.py routes (GET /api/deployments, /current, /:id, the last
with recent deploy_health_signal history) + a new v_deployment_status page
in frontend/app.js polling every 5s (deployment.status changes don't go
through c_ws_subscribe -- Deploy Supervisor writes deployment/
deploy_health_signal directly, no execution_event of its own).

No 02-V spec exists for this view (PH5 was added after 02-V was written) --
DOM/class conventions follow the existing pages' own patterns. Verified
against a real running server via urllib (browser extension unavailable
this session), plus 9 deployment_query unit tests and 5 real HTTP route
tests.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Adds supervisor/self_hosting_e2e.py: run_self_hosting_closed_loop() chains
every PH5 module built this session (U-034~U-041) with the Graph Engine's
own scheduler (U-032) into one real, callable path matching self-development-
feasibility.md's 9-step scenario -- self-target Goal -> Planner/Coder/
Validator -> core_engine classification -> stage -> health check -> cutover
(or rollback path A if health check fails) -> monitor until either
monitoring_window_clean (finalized) or anomaly_detected (rollback path B).
Both real outcome paths are branches of the same call, not two separate
scripted scenarios. Explicitly documented as this acceptance scenario's own
orchestration glue, not a claim about Deploy Supervisor's eventual
production poll loop (U-036's discovery-loop hook is still unfilled;
building that is future work).

Mocked wiring tests (6, run in normal regression) stub each already
independently-tested module boundary and only verify the branching logic.

The real, literal acceptance test is built and ready (RealSelfHostingClosedLoopTest,
reproduces the document's own "loop_round > 3 should be >= 3" example) but
gated behind RUN_REAL_SELF_HOSTING_E2E=1 -- deliberately a separate,
distinctly-named env var from this project's usual RUN_REAL_AGENT_E2E, since
this is the highest-blast-radius action in the project (can replace the
running graphlemon process and git-revert commits on this real repo).
Per this Unit's own explicit build instruction, not executed as part of
implementing it -- doing so requires checking with the user first.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Adds supervisor/deploy_rollback.py: rollback_path_a (pre-cutover, health
check failed -- terminate candidate, delete its one-time DB copy, git
revert, record a bug_record(core_engine) audit trail) and rollback_path_b
(post-cutover, anomaly detected -- stop the bad Live, revive
previous_instance_dir, re-run the same three health-check signals directly
against it, git revert only once re-verification passes).

Design choices follow established conventions rather than inventing new
ones: no unified rollback() entry point (deployment has no PID columns,
same reason deploy_stage/deploy_cutover already require callers to pass
pids explicitly); ROLLBACK_FAILED is the raised error name while the
persisted deployment.status is 'blocked_pending_human' (same pattern
CheckoutFailedError/CutoverFailedError already use). git revert conflicts
fold into the same rollback-itself-failed fallback both paths share, per
design doc §10 open question 3's own recommendation, with `git revert
--abort` run afterward so a failed attempt doesn't leave the repo mid-conflict.

This is PH5's highest-rework-risk Unit per development-plan.md's own flag,
so its own "rollback itself fails" fallback is tested for real, not assumed:
a genuinely broken previous_instance_dir (a real empty directory with no
Python package in it) that really fails to bind its port, and a genuinely
conflicting git history (a later commit touching the same line the
self-modification changed) that really fails `git revert` -- both correctly
land on blocked_pending_human without retrying forever or silently
succeeding, and the conflict case leaves the repo clean afterward.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Adds supervisor/deploy_monitor.py: run_monitor_cycle() is one sampling
pass, called repeatedly by Deploy Supervisor's own poll loop (03-C's own
text: not a one-shot call). process_alive/http_health/canary_task/
validator_approval_rate sampled in order, stopping at the first anomaly.

validator_approval_rate's "approve rate" has no literal definition anywhere
in 01-M/03-C/04-P (no such execution_event type exists) -- derived instead
from a verified structural fact in scheduler.finish_task_lifecycle: PASS
closes straight to done with no bug_record ever created, FAIL/INCONCLUSIVE
always creates one via judge_root_cause. A task counts as approved iff no
bug_record(source='validator') exists for it. Baseline is computed by the
same function looking back MONITORING_WINDOW_HOURS from cutover_at rather
than persisted as a new column; baseline=None (cold start) never flags an
anomaly rather than defaulting to 0.

Crash-count/consecutive-http-fail windows are approximated by sample count
(N = CRASH_WINDOW / MONITOR_SAMPLE_INTERVAL) rather than a true rolling
time window -- documented simplification, correct as long as sampling stays
roughly interval-paced.

Caught my own design misreading via the tests: process_alive's monitoring-
phase anomaly condition is MAX_CRASH_COUNT crashes within CRASH_WINDOW (a
crash-loop detector), not "currently not alive" -- unlike U-038's health
check where any single failure fails closed. A single dead reading during
monitoring is recorded but must not trigger an immediate rollback by itself.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>