Implements docs/todo/work-graph-parallel-execution-design.md: run_graph()
now schedules graph_manifest nodes layer-by-layer (topological_layers(),
Kahn's algorithm grouped by round instead of flattened), running a
multi-node layer's nodes concurrently via ThreadPoolExecutor (default 4
workers) with a barrier between layers; a single-node layer stays
synchronous, matching every real graph_manifest observed so far.
topological_order() is now a thin flattening wrapper, unchanged behavior
(regression-tested).
_LockedConn/_FetchedRows makes the existing scattered conn.execute() call
sites in work_node_run.py/work_graph_orchestrate.py/execution_log.py safe
to call from multiple threads without touching those call sites -- note
the lock must cover the full execute()-through-fetchone()/fetchall()
cycle, not just execute() itself, or two threads can still collide on the
shared sqlite3 connection handle. That gap was caught for real (not
theoretically) by RunGraphRealSqliteConcurrencyTest during development --
committing the fixed version, not the first draft.
create_worktree() (work_node_run.py) gets a module-level lock around the
`git worktree add` call for concurrent layers touching the same repo.
New tests: layer grouping, same-layer concurrency + inter-layer barrier,
one node's failure not cancelling a concurrent sibling (WorkGraphLayerError
aggregates), breakpoint-recovery resume of a concurrent layer, a quota-wait
timeout coexisting with a sibling's real persisted output, real (non-mocked)
sqlite under concurrent writes across several rounds, and real concurrent
`git worktree add` calls. Full src (575 tests) + supervisor (125 tests)
suites pass with zero regressions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
