宜博 / lemongraph

"""U-032: the scheduler 04-P/01-M/06-验收与测试锚点.md repeatedly presume but
no Unit before this one ever built. REQ-09's boundary case ("多个任务同时有
一个 blocked_pending_human、其余 queued 时，调度器应继续调度 queued 任务")
and 01-M's own indexing rationale ("索引...用于调度器扫描") both assume an
autonomous dispatcher exists; before this it didn't.
`org_node_lifecycle._check_concurrency()` (U-010) already gates a *single*
trigger call against `max_concurrent_goals`, but nothing ever called it
across a real queue of multiple Goals, and `e2e_pipeline.run_end_to_end()`
(U-014) drives exactly one task, synchronously, and was only ever called
from tests.

Found and closed in the same pass (2026-08-02): `run_end_to_end()` itself
deliberately stops at 'validating' with a computed-but-unapplied verdict
("Criteria 4-7 ... explicitly out of this Unit's scope" per its own
docstring) -- a scheduler whose job is "drive Goals to completion" can't
leave every task sitting at 'validating' forever, so `finish_task_lifecycle()`
below applies the PASS -> `close_task_done` / FAIL -> `org_level_loop.
run_fix_loop` step `run_end_to_end` always left undone, without changing
`run_end_to_end`'s own tested behavior or scope.

Threading model: each worker thread opens its OWN sqlite connection
(`schema.connect(db_path)`) rather than sharing one `conn` object across
threads -- U-027 already found the hard way (SIGSEGV) that a shared sqlite3
connection needs a hand-held lock around every operation including close();
replicating that discipline here would just serialize all DB work anyway and
defeat the point of concurrency. Each connection sets a `busy_timeout` so
concurrent writers on the shared file retry instead of raising "database is
locked".

**Second gap found and closed in the same pass (2026-08-02, PH4-closure
audit before starting PH5)**: `work_node_run.CorruptPersistedStateError`
(REQ-06 failure), `work_graph_execute.ClassifierBlockedError` (REQ-07
failure), and `work_node_run.UsageLimitWaitTimeoutError` (REQ-14 failure)
were each correctly raised and each correctly marked their own
`work_node_execution.status='blocked'` (the last one via U-031's catch-all
in `work_node_run.run_work_node`) -- but nothing ever caught them to apply
the *task*-level consequence 06-验收与测试锚点.md's failure text for all
three literally names: `task.status='blocked_pending_human'` with a specific
`blocked_reason`. `work_node_run.py`'s own docstring already flagged this as
a deliberate layering choice ("callers should route it to
task.status='blocked_pending_human'") -- but no caller ever did. This is
exactly the layer `budget_guardrail.check_budget()` already does the
equivalent translation at for its own case (REQ-10), so `_block_task()`
below follows that same established pattern (UPDATE + `task_status_changed`
execution_event) rather than inventing a new one.

**U-035 self-target audit trail (2026-08-02)**: resolved a real conflict
between supervisor-auto-rollback-design.md §1 (Deploy Supervisor triggers on
"self-target + Validator PASS + merge", no bug_record involved) and 03-C
§3.11/04-P §4.6's literal text (gated on an *existing*
`bug_record.root_cause_category=core_engine`) -- confirmed with the user
that a plain goal-type self-improvement task (task_create.py: `c_task_create`
only ever makes task_type='goal') never gets a bug_record on its happy path
at all, so the literal gate would silently defeat PH5's entire purpose for
that whole task class. Resolution (user-confirmed): trigger on self-target+
PASS alone; `bug_record.record_core_engine_change()` is created as an
audit-trail byproduct of that trigger, not a precondition for it. `deployment`
row creation itself (the actual hand-off) stays in U-037's `c_deploy_
supervisor_stage` as its own Done criterion already specifies -- this module
only detects + records, it doesn't stage a candidate. `p_deploy_supervisor_
blue_green` (04-P §4.6) never touches `task.status` at all -- it's a fully
separate state machine on `deployment`, matching supervisor-auto-rollback-
design.md §2's "no shared failure domain" between Deploy Supervisor and the
Graph Engine -- so `close_task_done()` still runs unconditionally on PASS
(the Coder's work IS genuinely complete and merged), the self-target check
is a purely additive side effect alongside it, not a replacement for it.
"""

import json
import uuid
import concurrent.futures

from src import (
    bug_record, core_engine_classify, e2e_pipeline, org_level_loop, org_loop_route, paths, schema,
    settings_config, task_create, work_graph_execute, work_graph_orchestrate, work_node_run,
)

# Rationale: see module docstring's "Threading model" section (lines 21-28) above.
_BUSY_TIMEOUT_MS = 30000
_FIX_LOOP_TIMEOUT = 1200

_BLOCKED_REASON_FOR_EXCEPTION = (
    (work_node_run.CorruptPersistedStateError, "corrupt_persisted_state"),
    (work_graph_execute.ClassifierBlockedError, "classifier_blocked"),
    (work_node_run.UsageLimitWaitTimeoutError, "usage_limit_wait_timeout"),
    (work_graph_orchestrate.ConsistencyCheckFailedError, "prd_consistency_check_failed"),
    (e2e_pipeline.PlannerReviewLoopLimitReachedError, "planner_review_loop_limit_reached"),
)
_ROUTABLE_EXCEPTIONS = tuple(exc for exc, _ in _BLOCKED_REASON_FOR_EXCEPTION)


def _blocked_reason_for(exc):
    """2026-08-07: `ConsistencyCheckFailedError` used to be raised only by
    the PRD-internal-contradiction check, but commit 07a0a27 (2026-08-06)
    generalized the same exception/retry machinery to the test_script
    mechanical compile-check too -- without this, both show up in the
    console as the same misleading 'prd_consistency_check_failed' label
    (see docs/done/validator-test-script-narrative-output-issue.md), even
    when the real failure has nothing to do with the PRD.
    ponytail: distinguishes by substring-matching the failing node's LLM-
    chosen node_id (e.g. 'n_test_script'/'validator_test_script') rather
    than a real typed discriminator threaded through verify_fn -- good
    enough because every real graph_manifest observed so far names its
    test_script node descriptively; ceiling is a node named ambiguously
    silently falling back to the PRD label. Upgrade path if that's ever
    wrong in practice: have verify_fn return a third 'kind' value instead
    of inferring from node_id text."""
    if isinstance(exc, work_graph_orchestrate.ConsistencyCheckFailedError):
        if "test_script" in (exc.node_id or ""):
            return "test_script_verification_failed"
        return "prd_consistency_check_failed"
    for exc_type, reason in _BLOCKED_REASON_FOR_EXCEPTION:
        if isinstance(exc, exc_type):
            return reason
    raise AssertionError("no blocked_reason registered for " + type(exc).__name__)


def _block_task(conn, task_id, blocked_reason, detail=None):
    """`detail` (2026-08-03): the real exception message, when there is one --
    `blocked_reason` itself stays a short routing code (budget_guardrail's
    auto-resume query matches on its exact value, e.g. 'budget_exceeded'), so
    this doesn't touch it. Without `detail` an unroutable failure (anything
    not in `_ROUTABLE_EXCEPTIONS`) landed as a bare 'execution_error' with
    zero diagnostic content anywhere -- found for real via the console's
    manual Start button on a freshly created project (see
    project_scaffold.py's agent-definitions fix, same incident)."""
    task_row = conn.execute("SELECT project_id, status FROM task WHERE task_id = ?", (task_id,)).fetchone()
    payload = {"from_status": task_row[1], "to_status": "blocked_pending_human"}
    if detail:
        payload["detail"] = detail
    conn.execute(
        "UPDATE task SET status = 'blocked_pending_human', blocked_reason = ?, "
        "updated_at = CURRENT_TIMESTAMP WHERE task_id = ?",
        (blocked_reason, task_id),
    )
    conn.execute(
        "INSERT INTO execution_event (event_id, project_id, task_id, event_type, payload) "
        "VALUES (?, ?, ?, 'task_status_changed', ?)",
        (str(uuid.uuid4()), task_row[0], task_id, json.dumps(payload)),
    )
    conn.commit()


def _record_core_engine_audit_if_self_target(conn, task_id, target_folder_path, run_result):
    """U-035 (05 §5.8): additive side effect on a self-target task's PASS
    closure -- see module docstring's "self-target audit trail" note for
    why this doesn't gate close_task_done, just follows it."""
    if not paths.is_self_target(target_folder_path):
        return
    code_diff_id = run_result.get("code_artifact_ids", {}).get("code_diff")
    if not code_diff_id:
        return
    diff_text = e2e_pipeline._read_artifact_content(conn, target_folder_path, code_diff_id)
    changed_files = core_engine_classify.parse_changed_files_from_diff(diff_text)
    if core_engine_classify.classify_core_engine_change(changed_files):
        bug_record.record_core_engine_change(
            conn, task_id,
            "self-target change touched core-engine files: " + ", ".join(sorted(set(changed_files))),
        )


def finish_task_lifecycle(conn, run_result, target_folder_path, goal_text):
    """PASS -> `close_task_done`; FAIL/INCONCLUSIVE -> `org_level_loop.
    run_fix_loop` (root-cause judge + reroute + fix + reverify, U-017).
    Adds 'final_status' to `run_result` and returns it.

    2026-08-12 (Org Graph unification Phase 2c): the FAIL/INCONCLUSIVE
    branch below is effectively dead code for tasks going through the
    current e2e_pipeline.py flow -- `validate_implementation_with_own_
    graph()`'s verdict is now a self-planned node with `judgment.kind=
    'mechanical'`/`on_fail.action='dynamic_loop_back'` (see that function's
    own docstring), which retries internally via work_graph_orchestrate.py's
    Reflection-review + [Fix-Skill] + loop-back machinery; a normal return
    from that function can therefore only ever carry verdict='PASS' -- a
    goal that can't resolve within the retry budget raises
    ConsistencyCheckFailedError instead, which routes to
    blocked_pending_human upstream of this function ever being called with
    a non-PASS verdict. Kept, not deleted, per minimal-change philosophy
    (a caller using the older, still-live `validate_implementation()`/
    hardcoded-manifest path -- none currently do -- would still need it)."""
    task_id = run_result["task_id"]
    verdict = run_result["verdict"]["verdict"]
    if verdict == "PASS":
        org_loop_route.close_task_done(conn, task_id)
        _record_core_engine_audit_if_self_target(conn, task_id, target_folder_path, run_result)
        run_result["final_status"] = "done"
        return run_result

    benchmark_id = run_result["doc_artifact_ids"]["benchmark"]
    benchmark_content = e2e_pipeline._read_artifact_content(conn, target_folder_path, benchmark_id)
    fix_result = org_level_loop.run_fix_loop(
        conn, task_id, target_folder_path,
        bug_description="Validator reported {} for goal: {}".format(verdict, goal_text),
        context=run_result["verdict"]["raw_response"],
        fix_prompt=e2e_pipeline._READY_PROMPT,
        verify_prompt=e2e_pipeline._VALIDATOR_PROMPT.format(
            goal_text=goal_text, benchmark_content=benchmark_content
        ),
        timeout=_FIX_LOOP_TIMEOUT,
    )
    run_result["final_status"] = fix_result["final_status"]
    return run_result


def run_goal_to_completion(db_path, target_folder_path, goal_text):
    """One Goal, its own sqlite connection, start to a terminal task.status
    (done or blocked_pending_human). This is the unit of work each scheduler
    worker thread runs.

    Creates the task itself (rather than delegating to e2e_pipeline.
    run_end_to_end) so task_id is in scope even if e2e_pipeline.
    run_task_lifecycle raises partway through -- needed to route
    CorruptPersistedStateError/ClassifierBlockedError/UsageLimitWaitTimeoutError
    to task.status='blocked_pending_human' (see module docstring's "Second
    gap" note).

    **2026-08-04**: any OTHER exception (e.g. work_graph_execute.
    WorkGraphExecuteError from a real API hiccup -- found for real: a
    "Response stalled mid-stream" error crashed the calling process
    entirely) used to propagate uncaught, same as before this note --
    `run_scheduler_once`'s own try/except only stops it from taking down
    OTHER goals in the same batch, it does NOT apply the task-level
    blocked_pending_human consequence either, so the task itself was left
    stuck in limbo: not running, not blocked, `blocked_reason` still NULL,
    nothing a human could `manual_resume` from the console. This mirrors
    the exact fallback `task_execution._mark_blocked()` already has for its
    own (Start-button) entry point -- both entry points now guarantee a
    task always lands on a real terminal-ish status with a real error
    detail, never a caller-crashes-and-the-task-is-just-stuck outcome.
    """
    # check_same_thread=False (2026-08-06, F): this conn can reach
    # work_graph_orchestrate.run_graph() via finish_task_lifecycle ->
    # org_level_loop.run_fix_loop -> root_cause_trace.
    # judge_root_cause_via_work_graph, whose 3-perspective layer runs on
    # its OWN ThreadPoolExecutor worker threads (G), not this function's
    # thread -- a plain schema.connect() here would raise "SQLite objects
    # created in a thread can only be used in that same thread" the first
    # time a real multi-node concurrent layer actually runs (previously
    # never exercised in production: every graph_manifest before F was
    # either single-node or a strictly serial chain).
    conn = schema.connect(db_path, check_same_thread=False)
    conn.execute("PRAGMA busy_timeout = {}".format(_BUSY_TIMEOUT_MS))
    try:
        project_id = conn.execute(
            "SELECT project_id FROM project WHERE target_folder_path = ?", (target_folder_path,)
        ).fetchone()[0]
        task_id = task_create.create_task(conn, project_id, goal_text)["task_id"]
        try:
            result = e2e_pipeline.run_task_lifecycle(conn, task_id, target_folder_path, goal_text)
        except _ROUTABLE_EXCEPTIONS as e:
            blocked_reason = _blocked_reason_for(e)
            _block_task(conn, task_id, blocked_reason, detail=str(e))
            return {"task_id": task_id, "final_status": "blocked_pending_human", "blocked_reason": blocked_reason}
        except Exception as e:
            _block_task(conn, task_id, "execution_error", detail=str(e))
            return {"task_id": task_id, "final_status": "blocked_pending_human", "blocked_reason": "execution_error"}
        return finish_task_lifecycle(conn, result, target_folder_path, goal_text)
    finally:
        conn.close()


def run_scheduler_once(db_path, goals, max_workers=None):
    """`goals`: [(target_folder_path, goal_text), ...], one entry per Goal to
    dispatch this round. Runs up to `max_workers` concurrently (default:
    `org_concurrency_config.max_concurrent_goals`, the same knob
    `org_node_lifecycle._check_concurrency` reads -- read fresh here so a
    config change takes effect on the next scheduler round without a
    restart); goals beyond that queue inside the executor and start as slots
    free up, which is what actually makes "raise max_concurrent_goals -> more
    Goals genuinely run in parallel" (U-032's Done line) true, rather than
    something only exercised by an exception-driven retry loop.

    A goal that raises (including `LoopLimitReachedError` once `loop_round`
    hits 3 -> `blocked_pending_human`, which IS the expected outcome of that
    path, not a scheduler bug) doesn't stop the others -- 04-P's "调度器继续
    执行队列中其他任务" applies to a raised exception in one goal exactly the
    same as it does to one that's already `blocked_pending_human` going in.
    Returns a list of {'goal_text', 'result'} or {'goal_text', 'error'} in
    `goals` order.
    """
    if max_workers is None:
        conn = schema.connect(db_path)
        try:
            config = settings_config.get_concurrency_config(conn)
        finally:
            conn.close()
        max_workers = config["max_concurrent_goals"] if config else 1

    results = [None] * len(goals)
    with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as pool:
        futures = {
            pool.submit(run_goal_to_completion, db_path, target_folder_path, goal_text): i
            for i, (target_folder_path, goal_text) in enumerate(goals)
        }
        for future in concurrent.futures.as_completed(futures):
            i = futures[future]
            goal_text = goals[i][1]
            try:
                results[i] = {"goal_text": goal_text, "result": future.result()}
            except Exception as e:
                results[i] = {"goal_text": goal_text, "error": str(e)}
    return results