Commit 7c46a2f281a9ad93c5abfd4fc56e7473efec00eb
1 parent
bd1404db
Phase 2b: Coder self-plans its own implementation graph via append_to_run_id
Implements org-graph-unification-phase2-universal-self-planning-design.md §2.3, §3, §6 -- Coder's implement_code() no longer runs one hardcoded synthetic "implement" node; it self-plans via plan_work_graph(conn, "coder", ..., append_to_run_id=run_id), appending its own nodes onto Planner's existing work_graph_run instead of creating a separate one. - work_graph_plan.py: per-org_node suggested node-type catalogs (_CODER_NODE_CATALOG/_VALIDATOR_NODE_CATALOG, "starting points, not requirements" same as _PLANNER_NODE_CATALOG), new judgment/on_fail/ promotes_to_artifact fields in the generic planning prompt. New plan_work_graph(..., append_to_run_id=...) parameter: merges newly planned nodes into an existing run (node_id collision -> raise, never silent overwrite), returns new_node_ids so callers can avoid reprocessing nodes an earlier stage already promoted to artifacts. New shared dispatch_build_prompt() for graphs mixing Planner-shaped (artifact_type) and Coder-shaped (task_prompt) nodes side by side. - e2e_pipeline.py: _write_completed_node_artifacts() now also promotes nodes declaring promotes_to_artifact (not just Planner's artifact_type nodes), with an only_node_ids scope so a merged-graph caller doesn't re-touch artifacts an earlier stage already wrote. implement_code() reworked around self-planning; code_diff stays Python-computed (a real git diff, not any node's own content -- no node can know the diff ahead of running it), test_report is promoted from whatever the self-planned graph declares, falling back to a synthesized concatenation if it declares none. needs_worktree=True now passed to run_graph() for Coder (found missing entirely during this rework -- harmless with the old single-node manifest, a real 03-C §3.6 gap once concurrent code-writing nodes are possible). Also fixed a real bug found while wiring this up: _review_planner_output_with_retries()'s "reuse existing artifacts" branch hardcoded run_id=None, even though every real artifact row already has a valid produced_by_run_id -- Coder had nothing to append into on that resume path. - work_graph_fork.py: _reconstruct_build_prompt now dispatches per-node (via the new shared dispatch_build_prompt) instead of requiring the whole graph to be one uniform shape -- a real regression Phase 2b would otherwise have introduced silently, since a forked merged graph can now genuinely mix both node shapes. Sequencing note (unchanged from Phase 2a's own finding): work_graph_run's org_node column removal and the ux_work_graph_run_active index narrowing (design doc §2.1/§2.2) stay deferred to Phase 2c -- Validator still creates its own separate run until Validator also appends into the shared one, so narrowing the index to task_id-only now would collide with it. python3 -m unittest discover -s src (752 tests) + discover -s supervisor -p "test_*.py" (126 tests) both green, zero regressions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Showing
8 changed files
with
857 additions
and
113 deletions
src/e2e_pipeline.py
| @@ -140,6 +140,12 @@ _DOC_PROMPTS = { | @@ -140,6 +140,12 @@ _DOC_PROMPTS = { | ||
| 140 | _CODE_INPUT_SCHEMA = {"type": "object", "required": ["node_id"]} | 140 | _CODE_INPUT_SCHEMA = {"type": "object", "required": ["node_id"]} |
| 141 | _CODE_OUTPUT_SCHEMA = {"type": "object", "required": ["response"]} | 141 | _CODE_OUTPUT_SCHEMA = {"type": "object", "required": ["response"]} |
| 142 | 142 | ||
| 143 | +# 2026-08-12 (Phase 2 §6): retired as the per-node prompt now that Coder | ||
| 144 | +# self-plans (implement_code() no longer runs one synthetic "implement" | ||
| 145 | +# node) -- its guidance is folded into _CODER_PLANNING_CONTEXT below, | ||
| 146 | +# which feeds plan_work_graph()'s planning call instead of a single | ||
| 147 | +# execution call. Kept here, unused, only because deleting it isn't this | ||
| 148 | +# change's job and a future revert would want it back verbatim. | ||
| 143 | _CODER_PROMPT = ( | 149 | _CODER_PROMPT = ( |
| 144 | "Implement this goal per the approved SDD below. Write real code and real tests, run the " | 150 | "Implement this goal per the approved SDD below. Write real code and real tests, run the " |
| 145 | "tests yourself, and only stop once they pass (or once you've exhausted reasonable attempts " | 151 | "tests yourself, and only stop once they pass (or once you've exhausted reasonable attempts " |
| @@ -160,6 +166,32 @@ _CODER_PROMPT = ( | @@ -160,6 +166,32 @@ _CODER_PROMPT = ( | ||
| 160 | "Test script:\n\n{test_script_content}" | 166 | "Test script:\n\n{test_script_content}" |
| 161 | ) | 167 | ) |
| 162 | 168 | ||
| 169 | +# 2026-08-12 (Phase 2 §6): fed as `goal_text` into plan_work_graph(conn, | ||
| 170 | +# "coder", ..., append_to_run_id=run_id) -- reuses the existing | ||
| 171 | +# {goal_text} substitution point in _generic_prompt_template() rather than | ||
| 172 | +# adding a new parameter to plan_work_graph() just for Coder's extra | ||
| 173 | +# context (SDD/Benchmark/test dataset/script), which Planner's own | ||
| 174 | +# _PROMPT_TEMPLATE call never needed. Same .replace("{goal_text}", ...) | ||
| 175 | +# convention _PROMPT_TEMPLATE/_generic_prompt_template() already use (not | ||
| 176 | +# .format() -- see work_graph_plan.py's own comment on why), so this | ||
| 177 | +# string's remaining braces (none currently, but keep this in mind if | ||
| 178 | +# edited) must stay literal, not format fields. | ||
| 179 | +_CODER_PLANNING_CONTEXT = ( | ||
| 180 | + "{goal_text}\n\n" | ||
| 181 | + "Approved SDD:\n\n{sdd_content}\n\n" | ||
| 182 | + "Benchmark (test dataset/script may be 'N/A' if this goal has none -- that's expected, not " | ||
| 183 | + "an error):\n\n{benchmark_content}\n\n" | ||
| 184 | + "Test dataset:\n\n{test_dataset_content}\n\n" | ||
| 185 | + "Test script:\n\n{test_script_content}\n\n" | ||
| 186 | + "Guidance: write real code and real tests, run the tests yourself as part of your own " | ||
| 187 | + "nodes' work, and don't consider the goal done until they pass for real (or you've " | ||
| 188 | + "exhausted reasonable attempts and a node explains why not). If the Planner's test script " | ||
| 189 | + "above is real (not N/A), materialize it as a real file under this project's own test " | ||
| 190 | + "conventions and make it pass for real -- it is not optional, it's part of this goal's " | ||
| 191 | + "acceptance bar; the Planner's script is a floor, write additional tests of your own if it " | ||
| 192 | + "doesn't cover everything the SDD requires." | ||
| 193 | +) | ||
| 194 | + | ||
| 163 | _VALIDATOR_PROMPT = ( | 195 | _VALIDATOR_PROMPT = ( |
| 164 | "Independently verify the implementation for this goal against the Benchmark below: run " | 196 | "Independently verify the implementation for this goal against the Benchmark below: run " |
| 165 | "the tests yourself, don't just trust the Coder's claims. Reply with a verdict line " | 197 | "the tests yourself, don't just trust the Coder's claims. Reply with a verdict line " |
| @@ -486,8 +518,30 @@ def _failed_node_ids(exc): | @@ -486,8 +518,30 @@ def _failed_node_ids(exc): | ||
| 486 | return set() | 518 | return set() |
| 487 | 519 | ||
| 488 | 520 | ||
| 521 | +def _resolve_node_artifact_type(node, node_artifact_types): | ||
| 522 | + """A node promotes to an `artifact` row if EITHER: it's in the caller's | ||
| 523 | + precomputed `node_artifact_types` dict (Planner's/Validator's fixed | ||
| 524 | + catalogs, where every node is a promotable artifact by construction -- | ||
| 525 | + unchanged behavior), OR it declares `artifact_type` directly on | ||
| 526 | + itself, OR (2026-08-12, Phase 2 §2.3) it declares `promotes_to_ | ||
| 527 | + artifact: {"artifact_type": ...}` (Coder's/Validator's generic | ||
| 528 | + self-planned nodes -- most of which should declare NEITHER, meaning | ||
| 529 | + "this is an internal graph step, not a final deliverable"). Returns | ||
| 530 | + None for a node that isn't promoting anything -- not an error, the | ||
| 531 | + expected case for most self-planned nodes.""" | ||
| 532 | + if node["node_id"] in node_artifact_types: | ||
| 533 | + return node_artifact_types[node["node_id"]] | ||
| 534 | + if "artifact_type" in node: | ||
| 535 | + return node["artifact_type"] | ||
| 536 | + promotes = node.get("promotes_to_artifact") | ||
| 537 | + if promotes: | ||
| 538 | + return promotes.get("artifact_type") | ||
| 539 | + return None | ||
| 540 | + | ||
| 541 | + | ||
| 489 | def _write_completed_node_artifacts( | 542 | def _write_completed_node_artifacts( |
| 490 | - conn, target_folder_path, task_id, run_id, graph_manifest, node_artifact_types, ext_for, run_graph_error=None, | 543 | + conn, target_folder_path, task_id, run_id, graph_manifest, node_artifact_types, ext_for, |
| 544 | + run_graph_error=None, only_node_ids=None, | ||
| 491 | ): | 545 | ): |
| 492 | """Writes an artifact for every node that's genuinely done AND | 546 | """Writes an artifact for every node that's genuinely done AND |
| 493 | verify_fn-accepted, sourced from `work_node_execution` in the DB -- | 547 | verify_fn-accepted, sourced from `work_node_execution` in the DB -- |
| @@ -511,6 +565,15 @@ def _write_completed_node_artifacts( | @@ -511,6 +565,15 @@ def _write_completed_node_artifacts( | ||
| 511 | nodes are excluded outright: their row is schema-valid ('done') but | 565 | nodes are excluded outright: their row is schema-valid ('done') but |
| 512 | verify_fn never accepted either attempt, so neither is trustworthy | 566 | verify_fn never accepted either attempt, so neither is trustworthy |
| 513 | content to persist. | 567 | content to persist. |
| 568 | + | ||
| 569 | + 2026-08-12 (Phase 2 §2.3/§6): a node with no resolvable artifact_type | ||
| 570 | + (see `_resolve_node_artifact_type`) is silently skipped, not an | ||
| 571 | + error -- most Coder/Validator self-planned nodes are internal graph | ||
| 572 | + steps, not final deliverables. `only_node_ids` (default None = every | ||
| 573 | + node in `graph_manifest`, unchanged behavior) restricts processing to | ||
| 574 | + a subset -- needed once a caller works over a MERGED graph (Coder's | ||
| 575 | + `append_to_run_id`) and must avoid re-processing/re-promoting nodes an | ||
| 576 | + EARLIER stage already wrote artifacts for. | ||
| 514 | """ | 577 | """ |
| 515 | done_rows = conn.execute( | 578 | done_rows = conn.execute( |
| 516 | "SELECT node_id, output_payload FROM work_node_execution WHERE run_id = ? AND status = 'done'", | 579 | "SELECT node_id, output_payload FROM work_node_execution WHERE run_id = ? AND status = 'done'", |
| @@ -529,7 +592,11 @@ def _write_completed_node_artifacts( | @@ -529,7 +592,11 @@ def _write_completed_node_artifacts( | ||
| 529 | node_id = node["node_id"] | 592 | node_id = node["node_id"] |
| 530 | if node_id not in done_outputs: | 593 | if node_id not in done_outputs: |
| 531 | continue | 594 | continue |
| 532 | - artifact_type = node_artifact_types[node_id] | 595 | + if only_node_ids is not None and node_id not in only_node_ids: |
| 596 | + continue | ||
| 597 | + artifact_type = _resolve_node_artifact_type(node, node_artifact_types) | ||
| 598 | + if artifact_type is None: | ||
| 599 | + continue | ||
| 533 | deps = [ | 600 | deps = [ |
| 534 | artifact_ids[node_artifact_types[dep_id]] | 601 | artifact_ids[node_artifact_types[dep_id]] |
| 535 | for dep_id in (node.get("depends_on") or []) | 602 | for dep_id in (node.get("depends_on") or []) |
| @@ -743,7 +810,19 @@ def _review_planner_output_with_retries(conn, task_id, goal_text, target_folder_ | @@ -743,7 +810,19 @@ def _review_planner_output_with_retries(conn, task_id, goal_text, target_folder_ | ||
| 743 | artifact_ids = task_execution._latest_artifact_ids( | 810 | artifact_ids = task_execution._latest_artifact_ids( |
| 744 | conn, task_id, task_execution._DOC_ARTIFACT_TYPES | 811 | conn, task_id, task_execution._DOC_ARTIFACT_TYPES |
| 745 | ) | 812 | ) |
| 746 | - plan_result = {"run_id": None, "artifact_ids": artifact_ids} | 813 | + # Phase 2 §6.2 (2026-08-12, found while wiring implement_code()'s |
| 814 | + # new append_to_run_id requirement): this used to hardcode | ||
| 815 | + # run_id=None -- harmless for Phase 1 (nothing downstream read | ||
| 816 | + # it), but Phase 2's Coder self-planning needs a REAL run_id to | ||
| 817 | + # append into. Every real artifact row already has one | ||
| 818 | + # (produced_by_run_id is NOT NULL) -- read it off the sdd | ||
| 819 | + # artifact (required a few lines below anyway, guaranteed | ||
| 820 | + # present whenever `existing` is truthy) rather than inventing | ||
| 821 | + # a new lookup path. | ||
| 822 | + sdd_run_id = conn.execute( | ||
| 823 | + "SELECT produced_by_run_id FROM artifact WHERE artifact_id = ?", (artifact_ids["sdd"],) | ||
| 824 | + ).fetchone()[0] | ||
| 825 | + plan_result = {"run_id": sdd_run_id, "artifact_ids": artifact_ids} | ||
| 747 | else: | 826 | else: |
| 748 | plan_result = plan_documents(conn, task_id, goal_text + feedback_suffix, target_folder_path) | 827 | plan_result = plan_documents(conn, task_id, goal_text + feedback_suffix, target_folder_path) |
| 749 | artifact_ids = plan_result["artifact_ids"] | 828 | artifact_ids = plan_result["artifact_ids"] |
| @@ -792,57 +871,124 @@ def plan_and_review_documents(conn, task_id, goal_text, target_folder_path, time | @@ -792,57 +871,124 @@ def plan_and_review_documents(conn, task_id, goal_text, target_folder_path, time | ||
| 792 | 871 | ||
| 793 | 872 | ||
| 794 | def implement_code( | 873 | def implement_code( |
| 795 | - conn, task_id, sdd_content, goal_text, target_folder_path, timeout=1200, | 874 | + conn, task_id, run_id, sdd_content, goal_text, target_folder_path, timeout=1200, |
| 796 | benchmark_content="N/A: not provided", test_dataset_content="N/A: not provided", | 875 | benchmark_content="N/A: not provided", test_dataset_content="N/A: not provided", |
| 797 | test_script_content="N/A: not provided", | 876 | test_script_content="N/A: not provided", |
| 798 | ): | 877 | ): |
| 799 | - """Architecture §11 criterion 2: a real Coder call via c_work_node_run | ||
| 800 | - (work_graph_orchestrate.run_graph with a synthetic single-node | ||
| 801 | - manifest -- see module docstring for why there's no real Coder graph | ||
| 802 | - planner), then registers whatever the Coder actually committed as | ||
| 803 | - code_diff (the diff itself) and test_report (the Coder's own response | ||
| 804 | - text, which per its migrated SOUL.md is required to include real test | ||
| 805 | - commands + real output) artifacts. | 878 | + """Phase 2 §6 (2026-08-12): Coder now self-plans its own implementation |
| 879 | + graph via `plan_work_graph(..., append_to_run_id=run_id)` -- `run_id` | ||
| 880 | + is Planner's own work_graph_run (created by `plan_and_review_ | ||
| 881 | + documents()`), no longer a fresh row this function creates itself. | ||
| 882 | + Coder's self-planned nodes get appended into that SAME run, not a | ||
| 883 | + separate one -- see work_graph_plan.py's own append_to_run_id | ||
| 884 | + docstring for the merge mechanics. Replaces the old synthetic | ||
| 885 | + single-node "implement" manifest (architecture §11 criterion 2's | ||
| 886 | + original, pre-self-planning shape). | ||
| 887 | + | ||
| 888 | + `code_diff` stays a Python-computed artifact (a real `git diff` across | ||
| 889 | + whatever Coder's whole self-planned graph actually changed on disk, | ||
| 890 | + end to end) -- not something any self-planned node's own `content` | ||
| 891 | + field could meaningfully hold, since no node knows the diff ahead of | ||
| 892 | + running it. `test_report` is promoted from whichever self-planned | ||
| 893 | + node(s) declared `promotes_to_artifact: {"artifact_type": | ||
| 894 | + "test_report"}` (the catalog's own "integration_check" entry suggests | ||
| 895 | + this); if the LLM's plan didn't include one (the catalog is | ||
| 896 | + suggestions, not requirements), falls back to concatenating every | ||
| 897 | + self-planned node's own content in topological order -- same spirit | ||
| 898 | + as the old single-node manifest's `response` field always being | ||
| 899 | + Coder's own narrative of what it did, just synthesized instead of | ||
| 900 | + directly given by one node. | ||
| 901 | + | ||
| 902 | + `needs_worktree=True` passed to run_graph() (2026-08-12, found while | ||
| 903 | + doing this rework -- NOT previously passed here at all, even though | ||
| 904 | + Coder is the one org_node that actually writes real code): with a | ||
| 905 | + single synthetic node this was harmless (nothing to collide with | ||
| 906 | + inside one run), but a genuinely self-planned graph can have 2+ | ||
| 907 | + code-writing nodes in the SAME topological layer running concurrently | ||
| 908 | + -- 03-C §3.6's "并行写代码的节点必须先申请 git_worktree_ref" applies for | ||
| 909 | + real now, not just in theory. | ||
| 806 | 910 | ||
| 807 | 2026-08-06: benchmark_content/test_dataset_content/test_script_content | 911 | 2026-08-06: benchmark_content/test_dataset_content/test_script_content |
| 808 | (default 'N/A: not provided' for callers that predate this, e.g. a | 912 | (default 'N/A: not provided' for callers that predate this, e.g. a |
| 809 | - resumed task from before this change) are threaded into _CODER_PROMPT | ||
| 810 | - so the Coder must make the Planner's real test script pass for real, | ||
| 811 | - not just write whatever tests it invents on its own. | 913 | + resumed task from before this change) are folded into |
| 914 | + _CODER_PLANNING_CONTEXT so Coder's planning call sees the Planner's | ||
| 915 | + real test script and must make it pass for real, not just write | ||
| 916 | + whatever tests it invents on its own. | ||
| 812 | """ | 917 | """ |
| 813 | before_sha = _current_commit(target_folder_path) | 918 | before_sha = _current_commit(target_folder_path) |
| 814 | - run_id = str(uuid.uuid4()) | ||
| 815 | - manifest = {"nodes": [{"node_id": "implement", "depends_on": []}], "edges": []} | ||
| 816 | - conn.execute( | ||
| 817 | - "INSERT INTO work_graph_run (run_id, task_id, org_node, graph_manifest, status) " | ||
| 818 | - "VALUES (?, ?, 'coder', ?, 'running')", | ||
| 819 | - (run_id, task_id, json.dumps(manifest)), | ||
| 820 | - ) | ||
| 821 | - conn.commit() | ||
| 822 | 919 | ||
| 823 | - outputs = work_graph_orchestrate.run_graph( | ||
| 824 | - conn, run_id, "coder", "coder-implementation", target_folder_path, manifest, | ||
| 825 | - lambda node, outputs: _CODER_PROMPT.format( | ||
| 826 | - goal_text=goal_text, sdd_content=sdd_content, benchmark_content=benchmark_content, | ||
| 827 | - test_dataset_content=test_dataset_content, test_script_content=test_script_content, | ||
| 828 | - ), | ||
| 829 | - _CODE_INPUT_SCHEMA, _CODE_OUTPUT_SCHEMA, timeout=timeout, | 920 | + planning_context = _CODER_PLANNING_CONTEXT.format( |
| 921 | + goal_text=goal_text, sdd_content=sdd_content, benchmark_content=benchmark_content, | ||
| 922 | + test_dataset_content=test_dataset_content, test_script_content=test_script_content, | ||
| 923 | + ) | ||
| 924 | + plan_result = work_graph_plan.plan_work_graph( | ||
| 925 | + conn, "coder", task_id, planning_context, target_folder_path, | ||
| 926 | + timeout=300, append_to_run_id=run_id, | ||
| 927 | + ) | ||
| 928 | + graph_manifest = plan_result["graph_manifest"] | ||
| 929 | + new_node_ids = set(plan_result["new_node_ids"]) | ||
| 930 | + | ||
| 931 | + # Covers BOTH node shapes that can now coexist in this merged graph: | ||
| 932 | + # Planner's own nodes (artifact_type) and Coder's just-appended ones | ||
| 933 | + # (promotes_to_artifact) -- used for both _planner_prompt_builder's | ||
| 934 | + # prompt construction (only its own artifact_type entries are ever | ||
| 935 | + # queried) and _write_completed_node_artifacts' artifact-type/deps | ||
| 936 | + # resolution below. | ||
| 937 | + node_artifact_types = {} | ||
| 938 | + for n in graph_manifest["nodes"]: | ||
| 939 | + if "artifact_type" in n: | ||
| 940 | + node_artifact_types[n["node_id"]] = n["artifact_type"] | ||
| 941 | + elif n.get("promotes_to_artifact"): | ||
| 942 | + node_artifact_types[n["node_id"]] = n["promotes_to_artifact"]["artifact_type"] | ||
| 943 | + | ||
| 944 | + build_prompt = work_graph_plan.dispatch_build_prompt( | ||
| 945 | + _planner_prompt_builder(goal_text, node_artifact_types), work_graph_plan.generic_node_prompt_builder, | ||
| 830 | ) | 946 | ) |
| 831 | - response = outputs["implement"]["response"] | 947 | + run_graph_error = None |
| 948 | + outputs = None | ||
| 949 | + try: | ||
| 950 | + outputs = work_graph_orchestrate.run_graph( | ||
| 951 | + conn, run_id, "coder", "coder-implementation", target_folder_path, graph_manifest, | ||
| 952 | + build_prompt, _DOC_INPUT_SCHEMA, _DOC_OUTPUT_SCHEMA, timeout=timeout, | ||
| 953 | + needs_worktree=True, mechanical_check_registry={"check_script_compiles": check_script_compiles}, | ||
| 954 | + ) | ||
| 955 | + except Exception as e: | ||
| 956 | + run_graph_error = e | ||
| 832 | 957 | ||
| 833 | diff = subprocess.run( | 958 | diff = subprocess.run( |
| 834 | ["git", "diff", before_sha, "HEAD"], cwd=target_folder_path, capture_output=True, text=True | 959 | ["git", "diff", before_sha, "HEAD"], cwd=target_folder_path, capture_output=True, text=True |
| 835 | ).stdout | 960 | ).stdout |
| 836 | - | ||
| 837 | diff_result = artifact_write.write_artifact( | 961 | diff_result = artifact_write.write_artifact( |
| 838 | conn, target_folder_path, "code_diff", task_id, run_id, | 962 | conn, target_folder_path, "code_diff", task_id, run_id, |
| 839 | ".graphlemon/runs/{}/code_diff.patch".format(task_id), diff or "(no diff -- see test_report)", | 963 | ".graphlemon/runs/{}/code_diff.patch".format(task_id), diff or "(no diff -- see test_report)", |
| 840 | ) | 964 | ) |
| 841 | - report_result = artifact_write.write_artifact( | ||
| 842 | - conn, target_folder_path, "test_report", task_id, run_id, | ||
| 843 | - ".graphlemon/runs/{}/test_report.md".format(task_id), response, | 965 | + |
| 966 | + artifact_ids = _write_completed_node_artifacts( | ||
| 967 | + conn, target_folder_path, task_id, run_id, graph_manifest, node_artifact_types, | ||
| 968 | + _planner_ext_for, run_graph_error, only_node_ids=new_node_ids, | ||
| 844 | ) | 969 | ) |
| 845 | - return {"code_diff": diff_result["artifact_id"], "test_report": report_result["artifact_id"]} | 970 | + |
| 971 | + if run_graph_error is not None: | ||
| 972 | + raise run_graph_error | ||
| 973 | + | ||
| 974 | + if "test_report" not in artifact_ids: | ||
| 975 | + # The self-planned graph didn't include a node declaring | ||
| 976 | + # promotes_to_artifact: {"artifact_type": "test_report"} -- | ||
| 977 | + # synthesize one so callers can still rely on this key always | ||
| 978 | + # being present (same contract the old single-node manifest gave | ||
| 979 | + # unconditionally via its one node's `response` field). | ||
| 980 | + report_content = "\n\n---\n\n".join( | ||
| 981 | + "### {}\n\n{}".format(node_id, (outputs or {}).get(node_id, {}).get("content", "")) | ||
| 982 | + for node_id in new_node_ids | ||
| 983 | + if node_id in (outputs or {}) | ||
| 984 | + ) or "(Coder's self-planned graph produced no synthesizable report content)" | ||
| 985 | + report_result = artifact_write.write_artifact( | ||
| 986 | + conn, target_folder_path, "test_report", task_id, run_id, | ||
| 987 | + ".graphlemon/runs/{}/test_report.md".format(task_id), report_content, | ||
| 988 | + ) | ||
| 989 | + artifact_ids["test_report"] = report_result["artifact_id"] | ||
| 990 | + | ||
| 991 | + return {"code_diff": diff_result["artifact_id"], "test_report": artifact_ids["test_report"]} | ||
| 846 | 992 | ||
| 847 | 993 | ||
| 848 | def _parse_verdict_line(response): | 994 | def _parse_verdict_line(response): |
| @@ -1093,7 +1239,7 @@ def run_task_lifecycle(conn, task_id, target_folder_path, goal_text): | @@ -1093,7 +1239,7 @@ def run_task_lifecycle(conn, task_id, target_folder_path, goal_text): | ||
| 1093 | 1239 | ||
| 1094 | org_node_lifecycle.trigger_org_node_full(conn, "coder", task_id, target_folder_path, _READY_PROMPT) | 1240 | org_node_lifecycle.trigger_org_node_full(conn, "coder", task_id, target_folder_path, _READY_PROMPT) |
| 1095 | code_artifact_ids = implement_code( | 1241 | code_artifact_ids = implement_code( |
| 1096 | - conn, task_id, sdd_content, goal_text, target_folder_path, | 1242 | + conn, task_id, plan_result["run_id"], sdd_content, goal_text, target_folder_path, |
| 1097 | benchmark_content=benchmark_content, test_dataset_content=test_dataset_content, | 1243 | benchmark_content=benchmark_content, test_dataset_content=test_dataset_content, |
| 1098 | test_script_content=test_script_content, | 1244 | test_script_content=test_script_content, |
| 1099 | ) | 1245 | ) |
src/task_execution.py
| @@ -165,7 +165,7 @@ def run_or_resume_task_lifecycle(conn, task_id, target_folder_path, goal_text): | @@ -165,7 +165,7 @@ def run_or_resume_task_lifecycle(conn, task_id, target_folder_path, goal_text): | ||
| 165 | _check_not_paused(conn, task_id) | 165 | _check_not_paused(conn, task_id) |
| 166 | org_node_lifecycle.trigger_org_node_full(conn, "coder", task_id, target_folder_path, e2e_pipeline._READY_PROMPT) | 166 | org_node_lifecycle.trigger_org_node_full(conn, "coder", task_id, target_folder_path, e2e_pipeline._READY_PROMPT) |
| 167 | code_artifact_ids = e2e_pipeline.implement_code( | 167 | code_artifact_ids = e2e_pipeline.implement_code( |
| 168 | - conn, task_id, sdd_content, goal_text, target_folder_path, | 168 | + conn, task_id, plan_result["run_id"], sdd_content, goal_text, target_folder_path, |
| 169 | benchmark_content=benchmark_content, test_dataset_content=test_dataset_content, | 169 | benchmark_content=benchmark_content, test_dataset_content=test_dataset_content, |
| 170 | test_script_content=test_script_content, | 170 | test_script_content=test_script_content, |
| 171 | ) | 171 | ) |
src/test_e2e_pipeline.py
| @@ -99,6 +99,56 @@ def _fake_plan_subprocess_run(*args, **kwargs): | @@ -99,6 +99,56 @@ def _fake_plan_subprocess_run(*args, **kwargs): | ||
| 99 | return subprocess.CompletedProcess(args=["claude"], returncode=0, stdout=json.dumps(payload), stderr="") | 99 | return subprocess.CompletedProcess(args=["claude"], returncode=0, stdout=json.dumps(payload), stderr="") |
| 100 | 100 | ||
| 101 | 101 | ||
| 102 | +# Phase 2 §6 (2026-08-12): Coder now self-plans too (implement_code() calls | ||
| 103 | +# plan_work_graph(conn, "coder", ..., append_to_run_id=...), going through | ||
| 104 | +# the SAME mocked subprocess boundary as Planner's own planning call) -- | ||
| 105 | +# depends_on Planner's sdd node so it lands in a LATER topological layer | ||
| 106 | +# than Planner's already-'done' nodes, not the same one (both being | ||
| 107 | +# in-degree-zero would trigger run_graph()'s concurrent ThreadPoolExecutor | ||
| 108 | +# layer path purely as a mocking artifact, unrelated to what this test is | ||
| 109 | +# actually checking). | ||
| 110 | +_FAKE_CODER_GRAPH_MANIFEST = { | ||
| 111 | + "nodes": [{ | ||
| 112 | + "node_id": "impl", "depends_on": ["sdd"], "task_prompt": "implement it", | ||
| 113 | + "promotes_to_artifact": {"artifact_type": "test_report"}, | ||
| 114 | + }], | ||
| 115 | + "edges": [], | ||
| 116 | +} | ||
| 117 | + | ||
| 118 | + | ||
| 119 | +_real_subprocess_run = subprocess.run # captured before any test patches subprocess.run globally | ||
| 120 | + | ||
| 121 | + | ||
| 122 | +def _fake_plan_subprocess_run_dispatch(*args, **kwargs): | ||
| 123 | + """Real plan_work_graph() dispatches by the --agent argv value -- this | ||
| 124 | + mock does the same, so both Planner's and Coder's planning calls (which | ||
| 125 | + now share the same mocked subprocess.run patch target) each get their | ||
| 126 | + own realistic fake manifest instead of one hardcoded response serving | ||
| 127 | + both callers (which would hand Coder Planner's own brd/prd/... nodes, | ||
| 128 | + colliding with the ones already in the run it's appending onto). | ||
| 129 | + | ||
| 130 | + patch("src.work_graph_plan.subprocess.run", ...) patches subprocess.run | ||
| 131 | + GLOBALLY, not just within work_graph_plan.py -- `subprocess` is the same | ||
| 132 | + singleton module object everywhere it's imported, so this side_effect | ||
| 133 | + also intercepts every OTHER real subprocess.run call this test's own | ||
| 134 | + call chain makes (artifact_write.write_artifact's `git add`/`git | ||
| 135 | + commit`, e2e_pipeline's `git diff`/`git rev-parse`, etc). Those aren't | ||
| 136 | + `claude --agent ... -p ...` calls -- pass them through to the REAL | ||
| 137 | + subprocess.run instead of returning a canned fake "claude planning | ||
| 138 | + response" CompletedProcess for them too (which happened to not visibly | ||
| 139 | + break anything before this dispatch function existed, only because | ||
| 140 | + nothing downstream read those specific calls' .stdout in a way that | ||
| 141 | + exposed it -- still wrong to keep doing on purpose now that this | ||
| 142 | + function actually inspects argv).""" | ||
| 143 | + argv = args[0] | ||
| 144 | + if not argv or argv[0] != "claude": | ||
| 145 | + return _real_subprocess_run(*args, **kwargs) | ||
| 146 | + org_node = argv[argv.index("--agent") + 1] | ||
| 147 | + manifest = _FAKE_CODER_GRAPH_MANIFEST if org_node == "coder" else _FAKE_GRAPH_MANIFEST | ||
| 148 | + payload = {"is_error": False, "result": "```json\n" + json.dumps(manifest) + "\n```"} | ||
| 149 | + return subprocess.CompletedProcess(args=["claude"], returncode=0, stdout=json.dumps(payload), stderr="") | ||
| 150 | + | ||
| 151 | + | ||
| 102 | def _fake_plan_subprocess_run_with_test_script(*args, **kwargs): | 152 | def _fake_plan_subprocess_run_with_test_script(*args, **kwargs): |
| 103 | payload = { | 153 | payload = { |
| 104 | "is_error": False, | 154 | "is_error": False, |
| @@ -117,7 +167,11 @@ class PipelineTestBase(unittest.TestCase): | @@ -117,7 +167,11 @@ class PipelineTestBase(unittest.TestCase): | ||
| 117 | self.tmp = tempfile.TemporaryDirectory() | 167 | self.tmp = tempfile.TemporaryDirectory() |
| 118 | self.addCleanup(self.tmp.cleanup) | 168 | self.addCleanup(self.tmp.cleanup) |
| 119 | subprocess.run(["git", "init"], cwd=self.tmp.name, check=True, capture_output=True) | 169 | subprocess.run(["git", "init"], cwd=self.tmp.name, check=True, capture_output=True) |
| 120 | - self.conn = schema.connect(":memory:") | 170 | + # check_same_thread=False (2026-08-12, Phase 2 §6) -- see |
| 171 | + # test_task_execution.py's TaskExecutionTestBase for why: a merged | ||
| 172 | + # graph's concurrent-layer path can now be reached even in tests | ||
| 173 | + # that don't otherwise care about G's parallel execution. | ||
| 174 | + self.conn = schema.connect(":memory:", check_same_thread=False) | ||
| 121 | schema.create_schema(self.conn) | 175 | schema.create_schema(self.conn) |
| 122 | project_scaffold.create_project(self.conn, self.tmp.name, init_git_if_missing=False) | 176 | project_scaffold.create_project(self.conn, self.tmp.name, init_git_if_missing=False) |
| 123 | 177 | ||
| @@ -125,6 +179,94 @@ class PipelineTestBase(unittest.TestCase): | @@ -125,6 +179,94 @@ class PipelineTestBase(unittest.TestCase): | ||
| 125 | self.conn.close() | 179 | self.conn.close() |
| 126 | 180 | ||
| 127 | 181 | ||
| 182 | +class WriteCompletedNodeArtifactsTest(PipelineTestBase): | ||
| 183 | + """Phase 2 §2.3/§6 (2026-08-12) direct coverage: promotes_to_artifact | ||
| 184 | + resolution and only_node_ids scoping, tested directly rather than only | ||
| 185 | + incidentally through plan_documents()/implement_code() integration | ||
| 186 | + tests.""" | ||
| 187 | + | ||
| 188 | + def setUp(self): | ||
| 189 | + super().setUp() | ||
| 190 | + self.conn.execute( | ||
| 191 | + "INSERT INTO task (task_id, project_id, goal_text, task_type, status, assigned_org_node) " | ||
| 192 | + "VALUES ('task-1', (SELECT project_id FROM project), 'goal', 'goal', 'coding', 'coder')" | ||
| 193 | + ) | ||
| 194 | + self.conn.execute( | ||
| 195 | + "INSERT INTO work_graph_run (run_id, task_id, org_node, graph_manifest, status) " | ||
| 196 | + "VALUES ('run-1', 'task-1', 'coder', '{}', 'running')" | ||
| 197 | + ) | ||
| 198 | + self.conn.execute( | ||
| 199 | + "INSERT INTO skill (skill_id, owner_org_node, file_path, current_git_commit_sha, source, status) " | ||
| 200 | + "VALUES ('coder-implementation', 'coder', 'skills/coder-implementation/', 'unknown', 'new', 'active')" | ||
| 201 | + ) | ||
| 202 | + self.conn.commit() | ||
| 203 | + | ||
| 204 | + def _mark_done(self, node_id, content): | ||
| 205 | + self.conn.execute( | ||
| 206 | + "INSERT INTO work_node_execution (node_exec_id, run_id, node_id, subagent_type, skill_ref, " | ||
| 207 | + "input_schema_ref, output_schema_ref, input_payload, output_payload, status) " | ||
| 208 | + "VALUES (?, 'run-1', ?, 'coder', 'coder-implementation', 'in.json', 'out.json', '{}', ?, 'done')", | ||
| 209 | + ("node-exec-" + node_id, node_id, json.dumps({"content": content})), | ||
| 210 | + ) | ||
| 211 | + self.conn.commit() | ||
| 212 | + | ||
| 213 | + def test_node_with_no_artifact_type_or_promotes_to_artifact_is_skipped(self): | ||
| 214 | + # most self-planned Coder nodes are internal graph steps, not final | ||
| 215 | + # deliverables -- must be silently skipped, not an error. | ||
| 216 | + manifest = {"nodes": [{"node_id": "explore", "depends_on": []}], "edges": []} | ||
| 217 | + self._mark_done("explore", "some intermediate reasoning") | ||
| 218 | + artifact_ids = e2e_pipeline._write_completed_node_artifacts( | ||
| 219 | + self.conn, self.tmp.name, "task-1", "run-1", manifest, {}, lambda t: "md", | ||
| 220 | + ) | ||
| 221 | + self.assertEqual(artifact_ids, {}) | ||
| 222 | + count = self.conn.execute("SELECT COUNT(*) FROM artifact WHERE task_id = 'task-1'").fetchone()[0] | ||
| 223 | + self.assertEqual(count, 0) | ||
| 224 | + | ||
| 225 | + def test_promotes_to_artifact_node_is_persisted(self): | ||
| 226 | + manifest = { | ||
| 227 | + "nodes": [{ | ||
| 228 | + "node_id": "final_report", "depends_on": [], | ||
| 229 | + "promotes_to_artifact": {"artifact_type": "test_report"}, | ||
| 230 | + }], | ||
| 231 | + "edges": [], | ||
| 232 | + } | ||
| 233 | + self._mark_done("final_report", "# Real report content") | ||
| 234 | + node_artifact_types = {"final_report": "test_report"} | ||
| 235 | + artifact_ids = e2e_pipeline._write_completed_node_artifacts( | ||
| 236 | + self.conn, self.tmp.name, "task-1", "run-1", manifest, node_artifact_types, lambda t: "md", | ||
| 237 | + ) | ||
| 238 | + self.assertEqual(set(artifact_ids), {"test_report"}) | ||
| 239 | + content = self.conn.execute( | ||
| 240 | + "SELECT file_path FROM artifact WHERE artifact_id = ?", (artifact_ids["test_report"],) | ||
| 241 | + ).fetchone()[0] | ||
| 242 | + with open(os.path.join(self.tmp.name, content)) as f: | ||
| 243 | + self.assertEqual(f.read(), "# Real report content") | ||
| 244 | + | ||
| 245 | + def test_only_node_ids_scopes_to_new_nodes_skipping_earlier_promotable_ones(self): | ||
| 246 | + # a merged graph (Planner's already-artifact'd nodes + Coder's just- | ||
| 247 | + # appended ones) must not re-promote/re-process nodes an EARLIER | ||
| 248 | + # stage already wrote artifacts for, even though they'd otherwise | ||
| 249 | + # be individually promotable. | ||
| 250 | + manifest = { | ||
| 251 | + "nodes": [ | ||
| 252 | + {"node_id": "sdd", "artifact_type": "sdd", "depends_on": []}, | ||
| 253 | + { | ||
| 254 | + "node_id": "impl", "depends_on": ["sdd"], | ||
| 255 | + "promotes_to_artifact": {"artifact_type": "test_report"}, | ||
| 256 | + }, | ||
| 257 | + ], | ||
| 258 | + "edges": [], | ||
| 259 | + } | ||
| 260 | + self._mark_done("sdd", "# SDD content") | ||
| 261 | + self._mark_done("impl", "# Report content") | ||
| 262 | + node_artifact_types = {"sdd": "sdd", "impl": "test_report"} | ||
| 263 | + artifact_ids = e2e_pipeline._write_completed_node_artifacts( | ||
| 264 | + self.conn, self.tmp.name, "task-1", "run-1", manifest, node_artifact_types, lambda t: "md", | ||
| 265 | + only_node_ids={"impl"}, | ||
| 266 | + ) | ||
| 267 | + self.assertEqual(set(artifact_ids), {"test_report"}) # NOT "sdd" -- outside only_node_ids | ||
| 268 | + | ||
| 269 | + | ||
| 128 | class PlanDocumentsTest(PipelineTestBase): | 270 | class PlanDocumentsTest(PipelineTestBase): |
| 129 | def test_produces_five_artifacts_with_correct_lineage(self): | 271 | def test_produces_five_artifacts_with_correct_lineage(self): |
| 130 | self.conn.execute( | 272 | self.conn.execute( |
| @@ -228,7 +370,7 @@ class PlanDocumentsTest(PipelineTestBase): | @@ -228,7 +370,7 @@ class PlanDocumentsTest(PipelineTestBase): | ||
| 228 | class RunEndToEndTest(PipelineTestBase): | 370 | class RunEndToEndTest(PipelineTestBase): |
| 229 | def test_full_pipeline_wiring(self): | 371 | def test_full_pipeline_wiring(self): |
| 230 | with patch("src.org_node_lifecycle.trigger_org_node", side_effect=_fake_ready_trigger), \ | 372 | with patch("src.org_node_lifecycle.trigger_org_node", side_effect=_fake_ready_trigger), \ |
| 231 | - patch("src.work_graph_plan.subprocess.run", side_effect=_fake_plan_subprocess_run), \ | 373 | + patch("src.work_graph_plan.subprocess.run", side_effect=_fake_plan_subprocess_run_dispatch), \ |
| 232 | patch.object(work_graph_execute, "write_workflow_script"), \ | 374 | patch.object(work_graph_execute, "write_workflow_script"), \ |
| 233 | patch.object(work_graph_execute, "execute_work_graph") as mock_execute, \ | 375 | patch.object(work_graph_execute, "execute_work_graph") as mock_execute, \ |
| 234 | patch.object(org_node_trigger, "trigger_org_node", return_value="VERDICT: PASS\nAll scenarios pass."): | 376 | patch.object(org_node_trigger, "trigger_org_node", return_value="VERDICT: PASS\nAll scenarios pass."): |
| @@ -237,7 +379,11 @@ class RunEndToEndTest(PipelineTestBase): | @@ -237,7 +379,11 @@ class RunEndToEndTest(PipelineTestBase): | ||
| 237 | {"consistent": True, "issues": []}, # PRD consistency check (REQ-02 boundary) | 379 | {"consistent": True, "issues": []}, # PRD consistency check (REQ-02 boundary) |
| 238 | {"content": _FAKE_DOCS["sdd"]}, {"content": _FAKE_DOCS["benchmark"]}, | 380 | {"content": _FAKE_DOCS["sdd"]}, {"content": _FAKE_DOCS["benchmark"]}, |
| 239 | {"content": _FAKE_DOCS["test_dataset"]}, | 381 | {"content": _FAKE_DOCS["test_dataset"]}, |
| 240 | - {"response": "Implemented. Ran `pytest`: 3 passed."}, | 382 | + # Phase 2 §6 (2026-08-12): Coder self-plans now -- one |
| 383 | + # self-planned "impl" node (_FAKE_CODER_GRAPH_MANIFEST), | ||
| 384 | + # content-keyed like every other self-planned node, not the | ||
| 385 | + # old single synthetic node's response-keyed output. | ||
| 386 | + {"content": "Implemented. Ran `pytest`: 3 passed."}, | ||
| 241 | # Validator's OWN independent benchmark/dataset (2026-08-06) -- | 387 | # Validator's OWN independent benchmark/dataset (2026-08-06) -- |
| 242 | # deliberately different content from the Planner's, to also prove | 388 | # deliberately different content from the Planner's, to also prove |
| 243 | # nothing here accidentally reuses the Planner's copies. | 389 | # nothing here accidentally reuses the Planner's copies. |
| @@ -265,15 +411,31 @@ class RunEndToEndTest(PipelineTestBase): | @@ -265,15 +411,31 @@ class RunEndToEndTest(PipelineTestBase): | ||
| 265 | # (test_script/validator_test_script disabled, see _VALIDATOR_MANIFEST) | 411 | # (test_script/validator_test_script disabled, see _VALIDATOR_MANIFEST) |
| 266 | self.assertEqual(all_artifacts, 9) | 412 | self.assertEqual(all_artifacts, 9) |
| 267 | 413 | ||
| 268 | - # planner, coder, AND validator (2026-08-06: validator now has its own | ||
| 269 | - # real work_graph_run too, not a flat direct call) each got their own | ||
| 270 | - # work_graph_run (not sharing one) | ||
| 271 | - run_org_nodes = { | 414 | + # Phase 2 §6 (2026-08-12): Coder no longer gets its own work_graph_run |
| 415 | + # -- implement_code() appends its self-planned nodes onto Planner's | ||
| 416 | + # SAME run (plan_work_graph(..., append_to_run_id=...)), so only TWO | ||
| 417 | + # rows exist now, not three: the shared Planner+Coder run (still | ||
| 418 | + # labeled org_node='planner' -- that column reflects who ORIGINATED | ||
| 419 | + # the run, not every org_node that has since contributed nodes to it; | ||
| 420 | + # Phase 2c is expected to fold Validator into the same run too, at | ||
| 421 | + # which point this column stops being a reliable "who's involved" | ||
| 422 | + # signal at all, see design doc §2.1) and Validator's own separate run. | ||
| 423 | + run_rows = self.conn.execute( | ||
| 424 | + "SELECT run_id, org_node FROM work_graph_run WHERE task_id = ?", (result["task_id"],) | ||
| 425 | + ).fetchall() | ||
| 426 | + self.assertEqual(len(run_rows), 2) | ||
| 427 | + self.assertEqual({row[1] for row in run_rows}, {"planner", "validator"}) | ||
| 428 | + | ||
| 429 | + # Coder's self-planned "impl" node genuinely landed in the SAME run | ||
| 430 | + # as Planner's nodes, not a separate one. | ||
| 431 | + planner_run_id = next(row[0] for row in run_rows if row[1] == "planner") | ||
| 432 | + node_ids = { | ||
| 272 | row[0] for row in self.conn.execute( | 433 | row[0] for row in self.conn.execute( |
| 273 | - "SELECT org_node FROM work_graph_run WHERE task_id = ?", (result["task_id"],) | 434 | + "SELECT node_id FROM work_node_execution WHERE run_id = ?", (planner_run_id,) |
| 274 | ).fetchall() | 435 | ).fetchall() |
| 275 | } | 436 | } |
| 276 | - self.assertEqual(run_org_nodes, {"planner", "coder", "validator"}) | 437 | + self.assertIn("impl", node_ids) |
| 438 | + self.assertIn("sdd", node_ids) | ||
| 277 | 439 | ||
| 278 | 440 | ||
| 279 | class LooksLikeNarrationTest(unittest.TestCase): | 441 | class LooksLikeNarrationTest(unittest.TestCase): |
| @@ -437,7 +599,7 @@ class RunTaskLifecycleMissingArtifactTypesTest(PipelineTestBase): | @@ -437,7 +599,7 @@ class RunTaskLifecycleMissingArtifactTypesTest(PipelineTestBase): | ||
| 437 | def test_prd_and_benchmark_missing_does_not_raise_key_error(self): | 599 | def test_prd_and_benchmark_missing_does_not_raise_key_error(self): |
| 438 | captured = {} | 600 | captured = {} |
| 439 | 601 | ||
| 440 | - def fake_implement_code(conn, task_id, sdd_content, goal_text, target_folder_path, **kwargs): | 602 | + def fake_implement_code(conn, task_id, run_id, sdd_content, goal_text, target_folder_path, **kwargs): |
| 441 | captured["benchmark_content"] = kwargs.get("benchmark_content") | 603 | captured["benchmark_content"] = kwargs.get("benchmark_content") |
| 442 | return {"code_diff": "diff-1", "test_report": "report-1"} | 604 | return {"code_diff": "diff-1", "test_report": "report-1"} |
| 443 | 605 |
src/test_task_execution.py
| @@ -14,7 +14,10 @@ import threading | @@ -14,7 +14,10 @@ import threading | ||
| 14 | import unittest | 14 | import unittest |
| 15 | from unittest.mock import patch | 15 | from unittest.mock import patch |
| 16 | 16 | ||
| 17 | -from src import e2e_pipeline, org_node_lifecycle, org_node_trigger, project_scaffold, schema, scheduler, task_execution, work_graph_execute | 17 | +from src import ( |
| 18 | + e2e_pipeline, org_node_lifecycle, org_node_trigger, project_scaffold, schema, scheduler, | ||
| 19 | + task_execution, work_graph_execute, work_graph_plan, | ||
| 20 | +) | ||
| 18 | 21 | ||
| 19 | 22 | ||
| 20 | def _fake_ready_trigger(org_node, prompt, target_folder_path, timeout=300): | 23 | def _fake_ready_trigger(org_node, prompt, target_folder_path, timeout=300): |
| @@ -22,12 +25,46 @@ def _fake_ready_trigger(org_node, prompt, target_folder_path, timeout=300): | @@ -22,12 +25,46 @@ def _fake_ready_trigger(org_node, prompt, target_folder_path, timeout=300): | ||
| 22 | return "Ready." | 25 | return "Ready." |
| 23 | 26 | ||
| 24 | 27 | ||
| 28 | +# Phase 2 §6 (2026-08-12): implement_code() now self-plans via | ||
| 29 | +# plan_work_graph(conn, "coder", ..., append_to_run_id=...) instead of | ||
| 30 | +# running one hardcoded synthetic node -- these tests aren't about self- | ||
| 31 | +# planning mechanics (that's test_e2e_pipeline.py's job), so this fakes | ||
| 32 | +# the planning subprocess with the smallest possible one-node manifest, | ||
| 33 | +# keeping the rest of each test's mock_execute.side_effect shape close to | ||
| 34 | +# what it was before this change (one node execution slot for Coder, | ||
| 35 | +# {"content": ...} instead of the old single-node manifest's {"response": ...}). | ||
| 36 | +_FAKE_CODER_MANIFEST = { | ||
| 37 | + "nodes": [{ | ||
| 38 | + "node_id": "impl", "depends_on": [], "task_prompt": "implement it", | ||
| 39 | + "promotes_to_artifact": {"artifact_type": "test_report"}, | ||
| 40 | + }], | ||
| 41 | + "edges": [], | ||
| 42 | +} | ||
| 43 | + | ||
| 44 | + | ||
| 45 | +def _fake_coder_plan_subprocess_run(*args, **kwargs): | ||
| 46 | + payload = {"is_error": False, "result": "```json\n" + json.dumps(_FAKE_CODER_MANIFEST) + "\n```"} | ||
| 47 | + return subprocess.CompletedProcess(args=["claude"], returncode=0, stdout=json.dumps(payload), stderr="") | ||
| 48 | + | ||
| 49 | + | ||
| 25 | class TaskExecutionTestBase(unittest.TestCase): | 50 | class TaskExecutionTestBase(unittest.TestCase): |
| 26 | def setUp(self): | 51 | def setUp(self): |
| 27 | self.tmp = tempfile.TemporaryDirectory() | 52 | self.tmp = tempfile.TemporaryDirectory() |
| 28 | self.addCleanup(self.tmp.cleanup) | 53 | self.addCleanup(self.tmp.cleanup) |
| 29 | subprocess.run(["git", "init"], cwd=self.tmp.name, check=True, capture_output=True) | 54 | subprocess.run(["git", "init"], cwd=self.tmp.name, check=True, capture_output=True) |
| 30 | - self.conn = schema.connect(":memory:") | 55 | + # check_same_thread=False (2026-08-12, Phase 2 §6): a merged graph |
| 56 | + # can legitimately put an already-'done' node (Planner's, short- | ||
| 57 | + # circuiting instantly) in the SAME topological layer as a newly- | ||
| 58 | + # appended node with no depends_on between them (e.g. Coder's own | ||
| 59 | + # self-planned node happens not to depend on Planner's) -- | ||
| 60 | + # work_graph_orchestrate.run_graph()'s multi-node-layer path always | ||
| 61 | + # uses a ThreadPoolExecutor once len(pending) > 1, regardless of how | ||
| 62 | + # fast one of them finishes. Real production connections already do | ||
| 63 | + # this (scheduler.py/task_execution.py's own start_task_execution_ | ||
| 64 | + # in_background, same "G" reasoning) -- this test fixture just | ||
| 65 | + # hadn't needed it before self-planning could produce concurrent | ||
| 66 | + # layers spanning old + new nodes. | ||
| 67 | + self.conn = schema.connect(":memory:", check_same_thread=False) | ||
| 31 | schema.create_schema(self.conn) | 68 | schema.create_schema(self.conn) |
| 32 | project = project_scaffold.create_project(self.conn, self.tmp.name, init_git_if_missing=False) | 69 | project = project_scaffold.create_project(self.conn, self.tmp.name, init_git_if_missing=False) |
| 33 | self.project_id = project["project_id"] | 70 | self.project_id = project["project_id"] |
| @@ -45,9 +82,18 @@ class TaskExecutionTestBase(unittest.TestCase): | @@ -45,9 +82,18 @@ class TaskExecutionTestBase(unittest.TestCase): | ||
| 45 | 82 | ||
| 46 | def _write_artifact(self, task_id, artifact_type, content): | 83 | def _write_artifact(self, task_id, artifact_type, content): |
| 47 | run_id = "run-" + artifact_type | 84 | run_id = "run-" + artifact_type |
| 85 | + # Phase 2 §6 (2026-08-12): graph_manifest is no longer a throwaway | ||
| 86 | + # '{}' -- implement_code() now reads it back (plan_work_graph's | ||
| 87 | + # append_to_run_id mode) whenever it appends onto the run the sdd | ||
| 88 | + # artifact points at, so it needs a real "nodes"/"edges" shape, not | ||
| 89 | + # an empty object real production data would never actually have. | ||
| 90 | + manifest = json.dumps({ | ||
| 91 | + "nodes": [{"node_id": artifact_type, "artifact_type": artifact_type, "depends_on": []}], | ||
| 92 | + "edges": [], | ||
| 93 | + }) | ||
| 48 | self.conn.execute( | 94 | self.conn.execute( |
| 49 | "INSERT OR IGNORE INTO work_graph_run (run_id, task_id, org_node, graph_manifest, status) " | 95 | "INSERT OR IGNORE INTO work_graph_run (run_id, task_id, org_node, graph_manifest, status) " |
| 50 | - "VALUES (?, ?, 'planner', '{}', 'done')", (run_id, task_id), | 96 | + "VALUES (?, ?, 'planner', ?, 'done')", (run_id, task_id, manifest), |
| 51 | ) | 97 | ) |
| 52 | rel_path = ".graphlemon/runs/{}/{}.md".format(task_id, artifact_type) | 98 | rel_path = ".graphlemon/runs/{}/{}.md".format(task_id, artifact_type) |
| 53 | abs_path = os.path.join(self.tmp.name, rel_path) | 99 | abs_path = os.path.join(self.tmp.name, rel_path) |
| @@ -122,12 +168,13 @@ class ResumeFromPlanningTest(TaskExecutionTestBase): | @@ -122,12 +168,13 @@ class ResumeFromPlanningTest(TaskExecutionTestBase): | ||
| 122 | self._write_artifact("task-1", "benchmark", "# Benchmark content") | 168 | self._write_artifact("task-1", "benchmark", "# Benchmark content") |
| 123 | 169 | ||
| 124 | with patch.object(org_node_lifecycle, "trigger_org_node", side_effect=_fake_ready_trigger), \ | 170 | with patch.object(org_node_lifecycle, "trigger_org_node", side_effect=_fake_ready_trigger), \ |
| 171 | + patch.object(work_graph_plan.subprocess, "run", side_effect=_fake_coder_plan_subprocess_run), \ | ||
| 125 | patch.object(work_graph_execute, "write_workflow_script"), \ | 172 | patch.object(work_graph_execute, "write_workflow_script"), \ |
| 126 | patch.object(work_graph_execute, "execute_work_graph") as mock_execute, \ | 173 | patch.object(work_graph_execute, "execute_work_graph") as mock_execute, \ |
| 127 | patch.object(org_node_trigger, "trigger_org_node", return_value="VERDICT: PASS") as mock_trigger, \ | 174 | patch.object(org_node_trigger, "trigger_org_node", return_value="VERDICT: PASS") as mock_trigger, \ |
| 128 | patch.object(e2e_pipeline, "plan_documents") as mock_plan: | 175 | patch.object(e2e_pipeline, "plan_documents") as mock_plan: |
| 129 | mock_execute.side_effect = [ | 176 | mock_execute.side_effect = [ |
| 130 | - {"response": "done, tests pass"}, # coder's implement node | 177 | + {"content": "done, tests pass"}, # coder's self-planned "impl" node |
| 131 | # validator's OWN benchmark/test_dataset/test_script graph (2026-08-06) | 178 | # validator's OWN benchmark/test_dataset/test_script graph (2026-08-06) |
| 132 | {"content": "# Validator's own Benchmark"}, | 179 | {"content": "# Validator's own Benchmark"}, |
| 133 | {"content": "N/A: no dataset needed"}, | 180 | {"content": "N/A: no dataset needed"}, |
| @@ -169,12 +216,13 @@ class ResumeFromReviewingTest(TaskExecutionTestBase): | @@ -169,12 +216,13 @@ class ResumeFromReviewingTest(TaskExecutionTestBase): | ||
| 169 | return _fake_ready_trigger(org_node, prompt, target_folder_path, timeout) | 216 | return _fake_ready_trigger(org_node, prompt, target_folder_path, timeout) |
| 170 | 217 | ||
| 171 | with patch.object(org_node_lifecycle, "trigger_org_node", side_effect=tracking_ready_trigger), \ | 218 | with patch.object(org_node_lifecycle, "trigger_org_node", side_effect=tracking_ready_trigger), \ |
| 219 | + patch.object(work_graph_plan.subprocess, "run", side_effect=_fake_coder_plan_subprocess_run), \ | ||
| 172 | patch.object(work_graph_execute, "write_workflow_script"), \ | 220 | patch.object(work_graph_execute, "write_workflow_script"), \ |
| 173 | patch.object(work_graph_execute, "execute_work_graph") as mock_execute, \ | 221 | patch.object(work_graph_execute, "execute_work_graph") as mock_execute, \ |
| 174 | patch.object(org_node_trigger, "trigger_org_node", return_value="VERDICT: PASS") as mock_trigger, \ | 222 | patch.object(org_node_trigger, "trigger_org_node", return_value="VERDICT: PASS") as mock_trigger, \ |
| 175 | patch.object(e2e_pipeline, "plan_documents") as mock_plan: | 223 | patch.object(e2e_pipeline, "plan_documents") as mock_plan: |
| 176 | mock_execute.side_effect = [ | 224 | mock_execute.side_effect = [ |
| 177 | - {"response": "done, tests pass"}, # coder's implement node | 225 | + {"content": "done, tests pass"}, # coder's self-planned "impl" node |
| 178 | {"content": "# Validator's own Benchmark"}, | 226 | {"content": "# Validator's own Benchmark"}, |
| 179 | {"content": "N/A: no dataset needed"}, | 227 | {"content": "N/A: no dataset needed"}, |
| 180 | {"content": "import sys\nsys.exit(0)\n"}, | 228 | {"content": "import sys\nsys.exit(0)\n"}, |
| @@ -209,16 +257,22 @@ class ResumeFromPlanningWithZeroArtifactsTest(TaskExecutionTestBase): | @@ -209,16 +257,22 @@ class ResumeFromPlanningWithZeroArtifactsTest(TaskExecutionTestBase): | ||
| 209 | # writes a real, validation_status='passed' sdd artifact, since | 257 | # writes a real, validation_status='passed' sdd artifact, since |
| 210 | # org_node_lifecycle._REQUIRED_ARTIFACTS["coder"] checks the real | 258 | # org_node_lifecycle._REQUIRED_ARTIFACTS["coder"] checks the real |
| 211 | # `artifact` table, not just this function's Python return value. | 259 | # `artifact` table, not just this function's Python return value. |
| 260 | + # run_id matches what _write_artifact's own test fixture convention | ||
| 261 | + # actually creates ("run-" + artifact_type) -- a real plan_documents() | ||
| 262 | + # call's return value would likewise point at the real work_graph_run | ||
| 263 | + # row it just inserted, which Phase 2's implement_code() now needs to | ||
| 264 | + # append into. | ||
| 212 | self._write_artifact(task_id, "sdd", "# SDD\nfreshly regenerated") | 265 | self._write_artifact(task_id, "sdd", "# SDD\nfreshly regenerated") |
| 213 | - return {"run_id": "run-x", "artifact_ids": {"sdd": "art-sdd"}} | 266 | + return {"run_id": "run-sdd", "artifact_ids": {"sdd": "art-sdd"}} |
| 214 | 267 | ||
| 215 | with patch.object(org_node_lifecycle, "trigger_org_node", side_effect=_fake_ready_trigger), \ | 268 | with patch.object(org_node_lifecycle, "trigger_org_node", side_effect=_fake_ready_trigger), \ |
| 269 | + patch.object(work_graph_plan.subprocess, "run", side_effect=_fake_coder_plan_subprocess_run), \ | ||
| 216 | patch.object(work_graph_execute, "write_workflow_script"), \ | 270 | patch.object(work_graph_execute, "write_workflow_script"), \ |
| 217 | patch.object(work_graph_execute, "execute_work_graph") as mock_execute, \ | 271 | patch.object(work_graph_execute, "execute_work_graph") as mock_execute, \ |
| 218 | patch.object(org_node_trigger, "trigger_org_node", return_value="VERDICT: PASS"), \ | 272 | patch.object(org_node_trigger, "trigger_org_node", return_value="VERDICT: PASS"), \ |
| 219 | patch.object(e2e_pipeline, "plan_documents", side_effect=fake_plan_documents) as mock_plan: | 273 | patch.object(e2e_pipeline, "plan_documents", side_effect=fake_plan_documents) as mock_plan: |
| 220 | mock_execute.side_effect = [ | 274 | mock_execute.side_effect = [ |
| 221 | - {"response": "done, tests pass"}, # coder's implement node | 275 | + {"content": "done, tests pass"}, # coder's self-planned "impl" node |
| 222 | {"content": "# Validator's own Benchmark"}, | 276 | {"content": "# Validator's own Benchmark"}, |
| 223 | {"content": "N/A: no dataset needed"}, | 277 | {"content": "N/A: no dataset needed"}, |
| 224 | {"content": "import sys\nsys.exit(0)\n"}, | 278 | {"content": "import sys\nsys.exit(0)\n"}, |
src/test_work_graph_fork.py
| @@ -139,6 +139,50 @@ class ForkFromPrdAcceptanceCriterion1Test(ForkTestBase): | @@ -139,6 +139,50 @@ class ForkFromPrdAcceptanceCriterion1Test(ForkTestBase): | ||
| 139 | self.assertEqual(run_row, ("done", "run-orig", node_exec_ids["prd"])) | 139 | self.assertEqual(run_row, ("done", "run-orig", node_exec_ids["prd"])) |
| 140 | 140 | ||
| 141 | 141 | ||
| 142 | +class ForkFromMixedShapeGraphTest(ForkTestBase): | ||
| 143 | + """Phase 2 §2.1.2 (2026-08-12) regression: a graph_manifest can now | ||
| 144 | + genuinely mix Planner-shaped nodes ('artifact_type') and Coder-shaped | ||
| 145 | + nodes ('task_prompt') side by side, once plan_work_graph(..., | ||
| 146 | + append_to_run_id=...) appends Coder's self-planned nodes onto | ||
| 147 | + Planner's original run. Forking from the artifact_type node must | ||
| 148 | + still correctly reconstruct a build_prompt for the DOWNSTREAM | ||
| 149 | + task_prompt node -- before this fix, _reconstruct_build_prompt | ||
| 150 | + required the WHOLE graph to be one uniform shape and would have | ||
| 151 | + raised ForkGraphShapeUnsupportedError here.""" | ||
| 152 | + | ||
| 153 | + def test_downstream_generic_node_reexecutes_after_forking_upstream_planner_node(self): | ||
| 154 | + manifest = { | ||
| 155 | + "nodes": [ | ||
| 156 | + {"node_id": "sdd", "artifact_type": "sdd", "depends_on": []}, | ||
| 157 | + { | ||
| 158 | + "node_id": "impl", "depends_on": ["sdd"], "task_prompt": "implement per the sdd", | ||
| 159 | + "promotes_to_artifact": {"artifact_type": "test_report"}, | ||
| 160 | + }, | ||
| 161 | + ], | ||
| 162 | + "edges": [], | ||
| 163 | + } | ||
| 164 | + contents = {"sdd": {"content": "original sdd"}, "impl": {"content": "original impl"}} | ||
| 165 | + node_exec_ids = self._seed_completed_run("run-orig", "planner", manifest, contents) | ||
| 166 | + | ||
| 167 | + execute_calls = [] | ||
| 168 | + | ||
| 169 | + def fake_execute(cwd, name, args, output_schema, timeout=600): | ||
| 170 | + execute_calls.append(name) | ||
| 171 | + return {"content": "regenerated for " + name} | ||
| 172 | + | ||
| 173 | + with patch.object(work_graph_execute, "write_workflow_script"), \ | ||
| 174 | + patch.object(work_graph_execute, "execute_work_graph", side_effect=fake_execute): | ||
| 175 | + result = work_graph_fork.fork_work_graph_run( | ||
| 176 | + self.conn, node_exec_ids["sdd"], {"content": "a completely different sdd"}, "/tmp/proj-1", | ||
| 177 | + ) | ||
| 178 | + | ||
| 179 | + # only "impl" (downstream of the fork point) genuinely re-ran. | ||
| 180 | + self.assertEqual(len(execute_calls), 1) | ||
| 181 | + outputs = result["outputs"] | ||
| 182 | + self.assertEqual(outputs["sdd"], {"content": "a completely different sdd"}) | ||
| 183 | + self.assertNotEqual(outputs["impl"], {"content": "original impl"}) | ||
| 184 | + | ||
| 185 | + | ||
| 142 | class OriginalRunUnaffectedTest(ForkTestBase): | 186 | class OriginalRunUnaffectedTest(ForkTestBase): |
| 143 | """Acceptance standard 2: the original run's records are completely | 187 | """Acceptance standard 2: the original run's records are completely |
| 144 | untouched by a fork.""" | 188 | untouched by a fork.""" |
src/test_work_graph_plan.py
| @@ -399,6 +399,129 @@ class GenericPlanningPromptTest(PlanWorkGraphTestBase): | @@ -399,6 +399,129 @@ class GenericPlanningPromptTest(PlanWorkGraphTestBase): | ||
| 399 | self.assertNotIn("BRD, PRD, SDD, Benchmark", prompt) | 399 | self.assertNotIn("BRD, PRD, SDD, Benchmark", prompt) |
| 400 | self.assertIn("task_prompt", prompt) | 400 | self.assertIn("task_prompt", prompt) |
| 401 | 401 | ||
| 402 | + def test_coder_prompt_includes_its_own_suggested_catalog_not_validators(self): | ||
| 403 | + # Phase 2 §3 (2026-08-12): each org_node gets its OWN suggested | ||
| 404 | + # node-type catalog, not a shared one -- "starting points, not | ||
| 405 | + # requirements", same contract _PLANNER_NODE_CATALOG already uses. | ||
| 406 | + with patch("src.work_graph_plan.subprocess.run") as run: | ||
| 407 | + run.return_value = _completed_process( | ||
| 408 | + stdout_obj={"is_error": False, "result": _fenced_result(_CODER_THREE_MODULE_MANIFEST)} | ||
| 409 | + ) | ||
| 410 | + work_graph_plan.plan_work_graph(self.conn, "coder", "task-1", "goal", "/tmp/proj-1") | ||
| 411 | + prompt = run.call_args.args[0][run.call_args.args[0].index("-p") + 1] | ||
| 412 | + self.assertIn("Suggested node types for this goal", prompt) | ||
| 413 | + self.assertIn('"node_type": "unit_test"', prompt) | ||
| 414 | + self.assertIn('"node_type": "integration_check"', prompt) | ||
| 415 | + self.assertNotIn('"node_type": "independent_benchmark"', prompt) # validator-only | ||
| 416 | + self.assertIn("promotes_to_artifact", prompt) | ||
| 417 | + self.assertIn("dynamic_loop_back", prompt) # explicitly told NOT to use it | ||
| 418 | + | ||
| 419 | + def test_validator_prompt_includes_its_own_catalog_not_coders(self): | ||
| 420 | + with patch("src.work_graph_plan.subprocess.run") as run: | ||
| 421 | + run.return_value = _completed_process( | ||
| 422 | + stdout_obj={"is_error": False, "result": _fenced_result(_CODER_THREE_MODULE_MANIFEST)} | ||
| 423 | + ) | ||
| 424 | + work_graph_plan.plan_work_graph(self.conn, "validator", "task-1", "goal", "/tmp/proj-1") | ||
| 425 | + prompt = run.call_args.args[0][run.call_args.args[0].index("-p") + 1] | ||
| 426 | + self.assertIn('"node_type": "independent_benchmark"', prompt) | ||
| 427 | + self.assertIn('"node_type": "test_script"', prompt) | ||
| 428 | + self.assertNotIn('"node_type": "unit_test"', prompt) # coder-only | ||
| 429 | + | ||
| 430 | + def test_unknown_org_node_gets_no_catalog_section_at_all(self): | ||
| 431 | + # An org_node this hasn't been extended for yet (Phase 1's | ||
| 432 | + # planner_reviewer, for instance) still gets the generic path -- | ||
| 433 | + # just with no suggested catalog, same as before catalogs existed. | ||
| 434 | + with patch("src.work_graph_plan.subprocess.run") as run: | ||
| 435 | + run.return_value = _completed_process( | ||
| 436 | + stdout_obj={"is_error": False, "result": _fenced_result(_CODER_THREE_MODULE_MANIFEST)} | ||
| 437 | + ) | ||
| 438 | + work_graph_plan.plan_work_graph(self.conn, "planner_reviewer", "task-1", "goal", "/tmp/proj-1") | ||
| 439 | + prompt = run.call_args.args[0][run.call_args.args[0].index("-p") + 1] | ||
| 440 | + self.assertNotIn("Suggested node types for this goal", prompt) | ||
| 441 | + self.assertIn("task_prompt", prompt) # still the generic template itself | ||
| 442 | + | ||
| 443 | + | ||
| 444 | +class AppendToRunIdTest(PlanWorkGraphTestBase): | ||
| 445 | + """Phase 2 §6.1 (2026-08-12): plan_work_graph(..., append_to_run_id=...) | ||
| 446 | + merges new nodes into an EXISTING work_graph_run instead of INSERTing a | ||
| 447 | + fresh row.""" | ||
| 448 | + | ||
| 449 | + def _seed_run(self, run_id, manifest): | ||
| 450 | + self.conn.execute( | ||
| 451 | + "INSERT INTO work_graph_run (run_id, task_id, org_node, graph_manifest, status) " | ||
| 452 | + "VALUES (?, 'task-1', 'planner', ?, 'running')", | ||
| 453 | + (run_id, json.dumps(manifest)), | ||
| 454 | + ) | ||
| 455 | + self.conn.commit() | ||
| 456 | + | ||
| 457 | + def test_merges_new_nodes_into_existing_run_not_a_new_row(self): | ||
| 458 | + self._seed_run("run-1", {"nodes": [{"node_id": "sdd", "artifact_type": "sdd", "depends_on": []}], "edges": []}) | ||
| 459 | + new_manifest = {"nodes": [{"node_id": "impl", "depends_on": ["sdd"], "task_prompt": "do it"}], "edges": []} | ||
| 460 | + with patch("src.work_graph_plan.subprocess.run") as run: | ||
| 461 | + run.return_value = _completed_process( | ||
| 462 | + stdout_obj={"is_error": False, "result": _fenced_result(new_manifest)} | ||
| 463 | + ) | ||
| 464 | + result = work_graph_plan.plan_work_graph( | ||
| 465 | + self.conn, "coder", "task-1", "goal", "/tmp/proj-1", append_to_run_id="run-1", | ||
| 466 | + ) | ||
| 467 | + | ||
| 468 | + self.assertEqual(result["run_id"], "run-1") | ||
| 469 | + self.assertEqual(result["new_node_ids"], ["impl"]) | ||
| 470 | + merged_node_ids = {n["node_id"] for n in result["graph_manifest"]["nodes"]} | ||
| 471 | + self.assertEqual(merged_node_ids, {"sdd", "impl"}) | ||
| 472 | + | ||
| 473 | + # exactly one work_graph_run row for this task -- no second row created. | ||
| 474 | + row_count = self.conn.execute("SELECT COUNT(*) FROM work_graph_run WHERE task_id = 'task-1'").fetchone()[0] | ||
| 475 | + self.assertEqual(row_count, 1) | ||
| 476 | + persisted = json.loads( | ||
| 477 | + self.conn.execute("SELECT graph_manifest FROM work_graph_run WHERE run_id = 'run-1'").fetchone()[0] | ||
| 478 | + ) | ||
| 479 | + self.assertEqual({n["node_id"] for n in persisted["nodes"]}, {"sdd", "impl"}) | ||
| 480 | + | ||
| 481 | + def test_existing_node_ids_are_injected_into_the_prompt(self): | ||
| 482 | + self._seed_run("run-1", {"nodes": [{"node_id": "sdd", "artifact_type": "sdd", "depends_on": []}], "edges": []}) | ||
| 483 | + new_manifest = {"nodes": [{"node_id": "impl", "depends_on": ["sdd"], "task_prompt": "do it"}], "edges": []} | ||
| 484 | + with patch("src.work_graph_plan.subprocess.run") as run: | ||
| 485 | + run.return_value = _completed_process( | ||
| 486 | + stdout_obj={"is_error": False, "result": _fenced_result(new_manifest)} | ||
| 487 | + ) | ||
| 488 | + work_graph_plan.plan_work_graph( | ||
| 489 | + self.conn, "coder", "task-1", "goal", "/tmp/proj-1", append_to_run_id="run-1", | ||
| 490 | + ) | ||
| 491 | + prompt = run.call_args.args[0][run.call_args.args[0].index("-p") + 1] | ||
| 492 | + self.assertIn("already has these nodes", prompt) | ||
| 493 | + self.assertIn("sdd", prompt) | ||
| 494 | + self.assertIn("ADDITIONAL nodes only", prompt) | ||
| 495 | + | ||
| 496 | + def test_node_id_collision_with_existing_node_raises(self): | ||
| 497 | + self._seed_run("run-1", {"nodes": [{"node_id": "sdd", "artifact_type": "sdd", "depends_on": []}], "edges": []}) | ||
| 498 | + # the "new" plan reuses "sdd" as its own node_id -- a real collision. | ||
| 499 | + colliding_manifest = {"nodes": [{"node_id": "sdd", "depends_on": [], "task_prompt": "oops"}], "edges": []} | ||
| 500 | + with patch("src.work_graph_plan.subprocess.run") as run: | ||
| 501 | + run.return_value = _completed_process( | ||
| 502 | + stdout_obj={"is_error": False, "result": _fenced_result(colliding_manifest)} | ||
| 503 | + ) | ||
| 504 | + with self.assertRaises(work_graph_plan.PlanningFailedError): | ||
| 505 | + work_graph_plan.plan_work_graph( | ||
| 506 | + self.conn, "coder", "task-1", "goal", "/tmp/proj-1", append_to_run_id="run-1", | ||
| 507 | + ) | ||
| 508 | + # the original run's manifest must be untouched by the failed append. | ||
| 509 | + persisted = json.loads( | ||
| 510 | + self.conn.execute("SELECT graph_manifest FROM work_graph_run WHERE run_id = 'run-1'").fetchone()[0] | ||
| 511 | + ) | ||
| 512 | + self.assertEqual([n["node_id"] for n in persisted["nodes"]], ["sdd"]) | ||
| 513 | + | ||
| 514 | + def test_append_to_nonexistent_run_id_raises(self): | ||
| 515 | + new_manifest = {"nodes": [{"node_id": "impl", "depends_on": [], "task_prompt": "do it"}], "edges": []} | ||
| 516 | + with patch("src.work_graph_plan.subprocess.run") as run: | ||
| 517 | + run.return_value = _completed_process( | ||
| 518 | + stdout_obj={"is_error": False, "result": _fenced_result(new_manifest)} | ||
| 519 | + ) | ||
| 520 | + with self.assertRaises(ValueError): | ||
| 521 | + work_graph_plan.plan_work_graph( | ||
| 522 | + self.conn, "coder", "task-1", "goal", "/tmp/proj-1", append_to_run_id="no-such-run", | ||
| 523 | + ) | ||
| 524 | + | ||
| 402 | 525 | ||
| 403 | class CoderThreeIndependentModulesScenarioTest(PlanWorkGraphTestBase): | 526 | class CoderThreeIndependentModulesScenarioTest(PlanWorkGraphTestBase): |
| 404 | """E acceptance criterion 1: a Coder scenario that naturally splits into | 527 | """E acceptance criterion 1: a Coder scenario that naturally splits into |
src/work_graph_fork.py
| @@ -94,23 +94,46 @@ def _downstream_node_ids(graph_manifest, start_node_id): | @@ -94,23 +94,46 @@ def _downstream_node_ids(graph_manifest, start_node_id): | ||
| 94 | def _reconstruct_build_prompt(conn, task_id, org_node, graph_manifest): | 94 | def _reconstruct_build_prompt(conn, task_id, org_node, graph_manifest): |
| 95 | """Rebuilds a build_prompt(node, outputs) usable for the re-executed | 95 | """Rebuilds a build_prompt(node, outputs) usable for the re-executed |
| 96 | downstream subgraph, purely from persisted state -- no access to the | 96 | downstream subgraph, purely from persisted state -- no access to the |
| 97 | - original call site's Python closures. Two supported shapes (design doc | ||
| 98 | - §2): Planner's canonical doc chain (every node has 'artifact_type', | 97 | + original call site's Python closures. Two supported node shapes |
| 98 | + (design doc §2): Planner's canonical doc chain ('artifact_type', | ||
| 99 | prompt reconstructed via e2e_pipeline's own builder + task.goal_text) | 99 | prompt reconstructed via e2e_pipeline's own builder + task.goal_text) |
| 100 | - and E's generic dynamic path (every node carries its own | ||
| 101 | - self-contained 'task_prompt'). Anything else raises rather than | ||
| 102 | - silently building broken prompts.""" | 100 | + and E's generic dynamic path ('task_prompt'). A node with neither |
| 101 | + raises rather than silently building a broken prompt. | ||
| 102 | + | ||
| 103 | + 2026-08-12 (Phase 2 §2.1.2, found while designing append_to_run_id): | ||
| 104 | + this used to require the WHOLE graph to be one uniform shape (all | ||
| 105 | + nodes 'artifact_type', or all 'task_prompt') -- correct when every | ||
| 106 | + graph_manifest genuinely was one shape throughout, which stopped being | ||
| 107 | + true the moment Coder's self-planned nodes (task_prompt) can get | ||
| 108 | + appended onto Planner's original run (artifact_type) via | ||
| 109 | + plan_work_graph(..., append_to_run_id=...). Dispatches per-NODE now | ||
| 110 | + (work_graph_plan.dispatch_build_prompt) instead of once for the whole | ||
| 111 | + graph -- every pre-Phase-2 graph_manifest still has just one shape | ||
| 112 | + throughout, so this is a strict generalization, not a behavior change | ||
| 113 | + for anything that isn't a merged graph. `org_node` is unused by this | ||
| 114 | + per-node dispatch itself (kept as a parameter -- callers still pass it | ||
| 115 | + for other purposes, e.g. run_graph()'s own org_node argument; | ||
| 116 | + removing it from work_graph_run/this signature is Phase 2c's job | ||
| 117 | + alongside the actual org_node column removal, not this fix's).""" | ||
| 103 | nodes = graph_manifest["nodes"] | 118 | nodes = graph_manifest["nodes"] |
| 104 | - if org_node == "planner" and nodes and all("artifact_type" in n for n in nodes): | ||
| 105 | - goal_text = conn.execute("SELECT goal_text FROM task WHERE task_id = ?", (task_id,)).fetchone()[0] | ||
| 106 | - node_artifact_types = {n["node_id"]: n["artifact_type"] for n in nodes} | ||
| 107 | - return e2e_pipeline._planner_prompt_builder(goal_text, node_artifact_types) | ||
| 108 | - if nodes and all("task_prompt" in n for n in nodes): | ||
| 109 | - return work_graph_plan.generic_node_prompt_builder | ||
| 110 | - raise ForkGraphShapeUnsupportedError( | ||
| 111 | - "graph_manifest nodes must either all have 'artifact_type' (Planner's canonical chain) " | ||
| 112 | - "or all have 'task_prompt' (the generic dynamic-planning path) to reconstruct prompts for a fork" | ||
| 113 | - ) | 119 | + if not nodes: |
| 120 | + raise ForkGraphShapeUnsupportedError("graph_manifest has no nodes to reconstruct a build_prompt for") | ||
| 121 | + goal_text = conn.execute("SELECT goal_text FROM task WHERE task_id = ?", (task_id,)).fetchone()[0] | ||
| 122 | + node_artifact_types = {n["node_id"]: n["artifact_type"] for n in nodes if "artifact_type" in n} | ||
| 123 | + planner_builder = e2e_pipeline._planner_prompt_builder(goal_text, node_artifact_types) | ||
| 124 | + dispatch = work_graph_plan.dispatch_build_prompt(planner_builder, work_graph_plan.generic_node_prompt_builder) | ||
| 125 | + | ||
| 126 | + def build_prompt(node, outputs): | ||
| 127 | + try: | ||
| 128 | + return dispatch(node, outputs) | ||
| 129 | + except ValueError: | ||
| 130 | + raise ForkGraphShapeUnsupportedError( | ||
| 131 | + "node {} has neither 'artifact_type' (Planner's canonical chain) nor 'task_prompt' " | ||
| 132 | + "(the generic dynamic-planning path) -- can't reconstruct a prompt for a fork".format( | ||
| 133 | + node.get("node_id") | ||
| 134 | + ) | ||
| 135 | + ) | ||
| 136 | + return build_prompt | ||
| 114 | 137 | ||
| 115 | 138 | ||
| 116 | def create_fork(conn, source_node_exec_id, new_output_payload): | 139 | def create_fork(conn, source_node_exec_id, new_output_payload): |
src/work_graph_plan.py
| @@ -41,10 +41,17 @@ GRAPH_MANIFEST_SCHEMA = { | @@ -41,10 +41,17 @@ GRAPH_MANIFEST_SCHEMA = { | ||
| 41 | "required": ["node_id", "artifact_type", "depends_on"], | 41 | "required": ["node_id", "artifact_type", "depends_on"], |
| 42 | "properties": { | 42 | "properties": { |
| 43 | "node_id": {"type": "string"}, | 43 | "node_id": {"type": "string"}, |
| 44 | + # "artifact_type" (Planner's fixed-catalog nodes) and "task_prompt" | ||
| 45 | + # (Coder/Validator's generic self-planned nodes, _generic_prompt_ | ||
| 46 | + # template()) are mutually exclusive node shapes in practice -- | ||
| 47 | + # "required" above only reflects Planner's shape (this schema | ||
| 48 | + # predates the generic path); see work_graph_plan.py's own dispatch | ||
| 49 | + # helpers for how callers tell the two apart per-node. | ||
| 44 | "artifact_type": { | 50 | "artifact_type": { |
| 45 | "type": "string", | 51 | "type": "string", |
| 46 | "enum": ["brd", "prd", "sdd", "benchmark", "test_dataset", "test_script"], | 52 | "enum": ["brd", "prd", "sdd", "benchmark", "test_dataset", "test_script"], |
| 47 | }, | 53 | }, |
| 54 | + "task_prompt": {"type": "string"}, | ||
| 48 | "depends_on": {"type": "array", "items": {"type": "string"}}, | 55 | "depends_on": {"type": "array", "items": {"type": "string"}}, |
| 49 | # 2026-08-08 (Work Graph self-planning + loop-back) -- both optional, | 56 | # 2026-08-08 (Work Graph self-planning + loop-back) -- both optional, |
| 50 | # documentation only (this schema isn't jsonschema-enforced, see | 57 | # documentation only (this schema isn't jsonschema-enforced, see |
| @@ -60,10 +67,20 @@ GRAPH_MANIFEST_SCHEMA = { | @@ -60,10 +67,20 @@ GRAPH_MANIFEST_SCHEMA = { | ||
| 60 | "on_fail": { | 67 | "on_fail": { |
| 61 | "type": "object", | 68 | "type": "object", |
| 62 | "properties": { | 69 | "properties": { |
| 63 | - "action": {"type": "string", "enum": ["retry_self", "loop_back", "hard_fail"]}, | 70 | + "action": { |
| 71 | + "type": "string", | ||
| 72 | + "enum": ["retry_self", "loop_back", "hard_fail", "dynamic_loop_back"], | ||
| 73 | + }, | ||
| 64 | "max_attempts": {"type": "integer"}, | 74 | "max_attempts": {"type": "integer"}, |
| 65 | }, | 75 | }, |
| 66 | }, | 76 | }, |
| 77 | + # 2026-08-12 (Phase 2 §2.3/§4): also optional/documentation-only. | ||
| 78 | + "promotes_to_artifact": { | ||
| 79 | + "type": "object", | ||
| 80 | + "properties": {"artifact_type": {"type": "string"}}, | ||
| 81 | + }, | ||
| 82 | + "subagent_type": {"type": "string"}, | ||
| 83 | + "skill_ref": {"type": "string"}, | ||
| 67 | }, | 84 | }, |
| 68 | }, | 85 | }, |
| 69 | }, | 86 | }, |
| @@ -126,13 +143,19 @@ _PLANNER_NODE_CATALOG = [ | @@ -126,13 +143,19 @@ _PLANNER_NODE_CATALOG = [ | ||
| 126 | }, | 143 | }, |
| 127 | ] | 144 | ] |
| 128 | 145 | ||
| 129 | -# NOTE: no `str.format()` involved in building or filling this template | ||
| 130 | -# (unlike _GENERIC_PROMPT_TEMPLATE) -- the embedded catalog JSON and the | ||
| 131 | -# instructional JSON examples below (e.g. {"kind": "none"}) are full of | ||
| 132 | -# their own literal braces, which .format() would try to parse as format | ||
| 133 | -# fields and crash on. plan_work_graph() fills in goal_text with a plain | 146 | +# NOTE: no `str.format()` involved in building or filling this template -- |
| 147 | +# the embedded catalog JSON and the instructional JSON examples below | ||
| 148 | +# (e.g. {"kind": "none"}) are full of their own literal braces, which | ||
| 149 | +# .format() would try to parse as format fields and crash on. | ||
| 150 | +# plan_work_graph() fills in goal_text with a plain | ||
| 134 | # `.replace("{goal_text}", goal_text)` instead, which treats every other | 151 | # `.replace("{goal_text}", goal_text)` instead, which treats every other |
| 135 | # brace in this string as inert literal text -- no escaping needed at all. | 152 | # brace in this string as inert literal text -- no escaping needed at all. |
| 153 | +# 2026-08-12: _generic_prompt_template() (below) now follows this SAME | ||
| 154 | +# convention for the same reason -- its own catalog JSON has the same | ||
| 155 | +# literal-brace problem .format() would choke on, which the original | ||
| 156 | +# _GENERIC_PROMPT_TEMPLATE constant used to dodge by having no embedded | ||
| 157 | +# JSON at all; now that it does (Phase 2 §3's node catalogs), it needed | ||
| 158 | +# the same fix this template already had. | ||
| 136 | _PROMPT_TEMPLATE = ( | 159 | _PROMPT_TEMPLATE = ( |
| 137 | "Plan (do not execute) the Work Graph for this goal: {goal_text}\n" | 160 | "Plan (do not execute) the Work Graph for this goal: {goal_text}\n" |
| 138 | "Available node types (use any subset, in any order that makes sense -- this goal may not " | 161 | "Available node types (use any subset, in any order that makes sense -- this goal may not " |
| @@ -163,29 +186,107 @@ _PROMPT_TEMPLATE = ( | @@ -163,29 +186,107 @@ _PROMPT_TEMPLATE = ( | ||
| 163 | # _PROMPT_TEMPLATE above unchanged (zero regression risk, see | 186 | # _PROMPT_TEMPLATE above unchanged (zero regression risk, see |
| 164 | # docs/done/dynamic-work-graph-design-skill-design.md §3): that chain is a | 187 | # docs/done/dynamic-work-graph-design-skill-design.md §3): that chain is a |
| 165 | # fixed architectural constant (brd->prd->sdd->benchmark->...), not | 188 | # fixed architectural constant (brd->prd->sdd->benchmark->...), not |
| 166 | -# something that should vary call to call, whereas Coder/Validator's | ||
| 167 | -# sub-tasks have no such fixed small vocabulary of node types. Each node | ||
| 168 | -# here carries its own self-contained `task_prompt` (see the | ||
| 169 | -# design-dynamic-work-graph Skill this prompt points at) instead of an | ||
| 170 | -# `artifact_type` that a fixed lookup table like _DOC_PROMPTS could resolve. | ||
| 171 | -_GENERIC_PROMPT_TEMPLATE = ( | ||
| 172 | - "Plan (do not execute) a Work Graph for this goal: {goal_text}\n" | ||
| 173 | - "Follow the design-dynamic-work-graph Skill's methodology: only add a depends_on edge " | ||
| 174 | - "when a node's task_prompt genuinely needs another node's output content -- nodes with " | ||
| 175 | - "no real data dependency between them will be scheduled to run CONCURRENTLY, so adding " | ||
| 176 | - "an edge just to make the graph look ordered actively slows execution down. If this goal " | ||
| 177 | - "is small enough to finish in one call, output a single-node graph_manifest rather than " | ||
| 178 | - "forcing a split.\n" | ||
| 179 | - "Output a graph_manifest whose nodes each have: node_id (a short readable identifier), " | ||
| 180 | - "depends_on (list of other node_ids), and task_prompt (the FULL, self-contained prompt " | ||
| 181 | - "for that node's own execution -- assume the agent running it sees nothing except this " | ||
| 182 | - "task_prompt and its dependencies' outputs, not this planning conversation). Every edge " | ||
| 183 | - "needs edge_type: 'zero_token' for pure aggregation/reduce steps, 'needs_judgment' for " | ||
| 184 | - "anything requiring the agent to actually reason about content.\n" | ||
| 185 | - "End your reply with the graph_manifest as a single fenced ```json code block " | ||
| 186 | - "shaped exactly like {{\"nodes\": [...], \"edges\": [...]}} -- no extra wrapper key." | 189 | +# something that should vary call to call. Each node here carries its own |
| 190 | +# self-contained `task_prompt` (see the design-dynamic-work-graph Skill | ||
| 191 | +# this prompt points at) instead of an `artifact_type` that a fixed lookup | ||
| 192 | +# table like _DOC_PROMPTS could resolve. | ||
| 193 | +# | ||
| 194 | +# 2026-08-12 (Phase 2 §3, user-confirmed 2026-08-08 -- reverses this | ||
| 195 | +# comment's own prior "no fixed small vocabulary of node types" framing): | ||
| 196 | +# Coder/Validator's sub-tasks DO get a suggested (not required) catalog | ||
| 197 | +# now, same "starting points, not requirements" contract _PLANNER_NODE_ | ||
| 198 | +# CATALOG already uses -- the LLM can ignore every entry and invent its | ||
| 199 | +# own node_ids/task_prompts freely, same as before this change. | ||
| 200 | +_NODE_REVIEW_JUDGE_PROMPT = ( | ||
| 201 | + "Judge whether this node's output is genuinely what it claims to be, not a narrative " | ||
| 202 | + "description of an action taken (e.g. 'I ran/produced/completed X') standing in for the real " | ||
| 203 | + "content itself -- this is a known recurring failure mode in nodes that explore with tools " | ||
| 204 | + "before producing a final answer. ok=false if the content is narration/progress-reporting " | ||
| 205 | + "prose rather than the real deliverable (code, test output, structured analysis, etc.), or if " | ||
| 206 | + "it's otherwise clearly incomplete/wrong for what was asked. List each concrete problem in " | ||
| 207 | + "issues.\n\nContent:\n\n{content}" | ||
| 187 | ) | 208 | ) |
| 188 | 209 | ||
| 210 | +_CODER_NODE_CATALOG = [ | ||
| 211 | + {"node_type": "impl", "description": "Actual code changes implementing the SDD."}, | ||
| 212 | + { | ||
| 213 | + "node_type": "unit_test", | ||
| 214 | + "description": "New/updated unit tests covering the change -- same repo convention as existing tests.", | ||
| 215 | + "suggested_judgment": {"kind": "llm", "judge_prompt": _NODE_REVIEW_JUDGE_PROMPT}, | ||
| 216 | + }, | ||
| 217 | + { | ||
| 218 | + "node_type": "integration_check", | ||
| 219 | + "description": "Actually runs the real test suite/benchmark script against the change and reports " | ||
| 220 | + "pass/fail -- not a description of doing so.", | ||
| 221 | + "suggested_judgment": {"kind": "llm", "judge_prompt": _NODE_REVIEW_JUDGE_PROMPT}, | ||
| 222 | + "suggested_promotes_to_artifact": {"artifact_type": "test_report"}, | ||
| 223 | + }, | ||
| 224 | +] | ||
| 225 | + | ||
| 226 | +_VALIDATOR_NODE_CATALOG = [ | ||
| 227 | + { | ||
| 228 | + "node_type": "independent_benchmark", | ||
| 229 | + "description": "Validator's own held-out acceptance scenarios -- NOT reused from Planner's/Coder's " | ||
| 230 | + "benchmark, a genuine second opinion.", | ||
| 231 | + }, | ||
| 232 | + {"node_type": "test_dataset", "description": "Concrete test data, if this goal has a natural one."}, | ||
| 233 | + { | ||
| 234 | + "node_type": "test_script", | ||
| 235 | + "description": "A real, runnable script mechanically checking the independent benchmark's scenarios.", | ||
| 236 | + "suggested_judgment": {"kind": "mechanical", "check_ref": "check_script_compiles"}, | ||
| 237 | + }, | ||
| 238 | +] | ||
| 239 | + | ||
| 240 | +_GENERIC_NODE_CATALOGS = {"coder": _CODER_NODE_CATALOG, "validator": _VALIDATOR_NODE_CATALOG} | ||
| 241 | + | ||
| 242 | + | ||
| 243 | +def _generic_prompt_template(org_node): | ||
| 244 | + """Builds _GENERIC_PROMPT_TEMPLATE's replacement, parameterized by | ||
| 245 | + org_node so each gets its own suggested catalog (or none, for any | ||
| 246 | + future org_node this hasn't been extended for yet -- catalog_section | ||
| 247 | + is simply empty then, same as before this change existed). | ||
| 248 | + node_type is a label in the SUGGESTION data only, not a required | ||
| 249 | + graph_manifest node field -- node_id/task_prompt stay exactly as | ||
| 250 | + freeform as before.""" | ||
| 251 | + catalog = _GENERIC_NODE_CATALOGS.get(org_node, []) | ||
| 252 | + catalog_section = ( | ||
| 253 | + "Suggested node types for this goal (use any subset, in any order, invent your own if " | ||
| 254 | + "none fit -- these are starting points, not requirements):\n" + json.dumps(catalog) + "\n" | ||
| 255 | + if catalog else "" | ||
| 256 | + ) | ||
| 257 | + return ( | ||
| 258 | + "Plan (do not execute) a Work Graph for this goal: {goal_text}\n" | ||
| 259 | + + catalog_section + | ||
| 260 | + "Follow the design-dynamic-work-graph Skill's methodology: only add a depends_on edge " | ||
| 261 | + "when a node's task_prompt genuinely needs another node's output content -- nodes with " | ||
| 262 | + "no real data dependency between them will be scheduled to run CONCURRENTLY, so adding " | ||
| 263 | + "an edge just to make the graph look ordered actively slows execution down. If this goal " | ||
| 264 | + "is small enough to finish in one call, output a single-node graph_manifest rather than " | ||
| 265 | + "forcing a split.\n" | ||
| 266 | + "Output a graph_manifest whose nodes each have: node_id (a short readable identifier), " | ||
| 267 | + "depends_on (list of other node_ids), and task_prompt (the FULL, self-contained prompt " | ||
| 268 | + "for that node's own execution -- assume the agent running it sees nothing except this " | ||
| 269 | + "task_prompt and its dependencies' outputs, not this planning conversation), plus:\n" | ||
| 270 | + " - judgment (optional): how to tell if this node's output is acceptable -- " | ||
| 271 | + "{\"kind\": \"none\"} if nothing needs checking (the default if omitted), {\"kind\": " | ||
| 272 | + "\"llm\", \"judge_prompt\": \"...{content}...\"} for your own judgment prompt with a " | ||
| 273 | + "{content} placeholder.\n" | ||
| 274 | + " - on_fail (only meaningful if judgment.kind != \"none\"): {\"action\": \"retry_self\"}, " | ||
| 275 | + "{\"action\": \"hard_fail\"}, or {\"action\": \"loop_back\"} with a matching on_failure " | ||
| 276 | + "edge to a genuine upstream dependency (same rule _PLANNER_NODE_CATALOG's own nodes " | ||
| 277 | + "follow) -- do NOT use \"dynamic_loop_back\", that's reserved for the terminal review " | ||
| 278 | + "stage outside this graph, not for nodes you plan yourself.\n" | ||
| 279 | + " - promotes_to_artifact (optional): {\"artifact_type\": \"test_report\"} ONLY for a " | ||
| 280 | + "node producing this task's actual final deliverable text -- most nodes should omit " | ||
| 281 | + "this; it marks a node's content for persistence beyond this graph's own execution. " | ||
| 282 | + "(code_diff, if relevant, is derived separately from real file changes, not from any " | ||
| 283 | + "node's own content -- never declare artifact_type: \"code_diff\" here.)\n" | ||
| 284 | + "Every edge needs edge_type: 'zero_token' for pure aggregation/reduce steps, " | ||
| 285 | + "'needs_judgment' for anything requiring the agent to actually reason about content.\n" | ||
| 286 | + "End your reply with the graph_manifest as a single fenced ```json code block " | ||
| 287 | + "shaped exactly like {{\"nodes\": [...], \"edges\": [...]}} -- no extra wrapper key." | ||
| 288 | + ) | ||
| 289 | + | ||
| 189 | _JSON_FENCE_RE = re.compile(r"```(?:json)?\s*(\{.*?\})\s*```", re.DOTALL) | 290 | _JSON_FENCE_RE = re.compile(r"```(?:json)?\s*(\{.*?\})\s*```", re.DOTALL) |
| 190 | 291 | ||
| 191 | 292 | ||
| @@ -333,22 +434,61 @@ def _normalize_graph_manifest(manifest): | @@ -333,22 +434,61 @@ def _normalize_graph_manifest(manifest): | ||
| 333 | return manifest | 434 | return manifest |
| 334 | 435 | ||
| 335 | 436 | ||
| 336 | -def plan_work_graph(conn, org_node, task_id, goal_text, target_folder_path, timeout=300): | ||
| 337 | - """Returns {"run_id", "graph_manifest"}. Creates one `work_graph_run` row | ||
| 338 | - (status='planning') -- moving it to 'running'/executing it is out of scope. | 437 | +def plan_work_graph(conn, org_node, task_id, goal_text, target_folder_path, timeout=300, append_to_run_id=None): |
| 438 | + """Returns {"run_id", "graph_manifest", "new_node_ids"}. Creates one | ||
| 439 | + `work_graph_run` row (status='planning') -- moving it to 'running'/ | ||
| 440 | + executing it is out of scope. | ||
| 339 | 441 | ||
| 340 | 2026-08-08: `org_node == 'planner'` gets `_PROMPT_TEMPLATE` (now a | 442 | 2026-08-08: `org_node == 'planner'` gets `_PROMPT_TEMPLATE` (now a |
| 341 | genuine self-planning prompt over `_PLANNER_NODE_CATALOG`, see that | 443 | genuine self-planning prompt over `_PLANNER_NODE_CATALOG`, see that |
| 342 | template's own comment for why goal_text is filled in via `.replace()` | 444 | template's own comment for why goal_text is filled in via `.replace()` |
| 343 | rather than `.format()`). Every other org_node gets | 445 | rather than `.format()`). Every other org_node gets |
| 344 | - `_GENERIC_PROMPT_TEMPLATE` -- the cross-Org-Agent dynamic planning path | ||
| 345 | - described in the design-dynamic-work-graph Skill, whose nodes carry | ||
| 346 | - their own task_prompt instead of a fixed artifact_type. Function | ||
| 347 | - signature/return shape are unchanged either way.""" | 446 | + `_generic_prompt_template(org_node)` -- the cross-Org-Agent dynamic |
| 447 | + planning path described in the design-dynamic-work-graph Skill, whose | ||
| 448 | + nodes carry their own task_prompt instead of a fixed artifact_type. | ||
| 449 | + | ||
| 450 | + 2026-08-12 (Phase 2 §6.1, user-confirmed 2026-08-08: a parameter on | ||
| 451 | + this function, not a separate plan_work_graph_append()): with | ||
| 452 | + `append_to_run_id=None` (default), behavior is unchanged from before | ||
| 453 | + this parameter existed -- a fresh INSERT. With `append_to_run_id= | ||
| 454 | + <existing run_id>`, this call plans ADDITIONAL nodes for an | ||
| 455 | + already-existing work_graph_run (e.g. Coder appending its own | ||
| 456 | + implementation nodes onto Planner's run) instead of creating a new | ||
| 457 | + row: the existing manifest's node_ids are injected into the prompt so | ||
| 458 | + new nodes can `depends_on` them, a `node_id` collision between the | ||
| 459 | + newly-planned nodes and the existing ones raises PlanningFailedError | ||
| 460 | + (never silently renamed or overwritten), and the row is UPDATEd with | ||
| 461 | + the merged manifest instead of a new row being INSERTed. | ||
| 462 | + `new_node_ids` (only meaningfully non-empty in append mode, though | ||
| 463 | + always present for a uniform return shape) is the set of node_ids this | ||
| 464 | + specific call added -- callers that promote node output to `artifact` | ||
| 465 | + rows (e2e_pipeline.py's `_write_completed_node_artifacts`) need this | ||
| 466 | + to avoid re-processing nodes an EARLIER stage already promoted; the | ||
| 467 | + merged manifest itself doesn't distinguish "planned this call" from | ||
| 468 | + "already existed" once merged.""" | ||
| 469 | + existing_manifest = None | ||
| 470 | + if append_to_run_id: | ||
| 471 | + row = conn.execute( | ||
| 472 | + "SELECT graph_manifest FROM work_graph_run WHERE run_id = ?", (append_to_run_id,) | ||
| 473 | + ).fetchone() | ||
| 474 | + if row is None: | ||
| 475 | + raise ValueError("no such work_graph_run to append to: " + append_to_run_id) | ||
| 476 | + existing_manifest = json.loads(row[0]) | ||
| 477 | + | ||
| 348 | if org_node == "planner": | 478 | if org_node == "planner": |
| 349 | prompt = _PROMPT_TEMPLATE.replace("{goal_text}", goal_text) | 479 | prompt = _PROMPT_TEMPLATE.replace("{goal_text}", goal_text) |
| 350 | else: | 480 | else: |
| 351 | - prompt = _GENERIC_PROMPT_TEMPLATE.format(goal_text=goal_text) | 481 | + prompt = _generic_prompt_template(org_node).replace("{goal_text}", goal_text) |
| 482 | + | ||
| 483 | + if existing_manifest is not None: | ||
| 484 | + existing_ids = [n["node_id"] for n in existing_manifest["nodes"]] | ||
| 485 | + prompt = prompt.replace( | ||
| 486 | + "Plan (do not execute)", | ||
| 487 | + "This graph already has these nodes from earlier stages (you may depends_on them, but do " | ||
| 488 | + "NOT redeclare or duplicate them -- pick different node_ids for your own new nodes): " | ||
| 489 | + "{}. Plan (do not execute) ADDITIONAL nodes only".format(existing_ids), | ||
| 490 | + 1, | ||
| 491 | + ) | ||
| 352 | 492 | ||
| 353 | result = subprocess.run( | 493 | result = subprocess.run( |
| 354 | [ | 494 | [ |
| @@ -374,24 +514,48 @@ def plan_work_graph(conn, org_node, task_id, goal_text, target_folder_path, time | @@ -374,24 +514,48 @@ def plan_work_graph(conn, org_node, task_id, goal_text, target_folder_path, time | ||
| 374 | payload = json.loads(result.stdout) | 514 | payload = json.loads(result.stdout) |
| 375 | if payload.get("is_error"): | 515 | if payload.get("is_error"): |
| 376 | raise PlanningFailedError("plan failed for task {}: {}".format(task_id, payload)) | 516 | raise PlanningFailedError("plan failed for task {}: {}".format(task_id, payload)) |
| 377 | - graph_manifest = _extract_graph_manifest(payload.get("result", "")) | ||
| 378 | - if not graph_manifest: | 517 | + new_manifest = _extract_graph_manifest(payload.get("result", "")) |
| 518 | + if not new_manifest: | ||
| 379 | raise PlanningFailedError("could not extract a graph_manifest for task {}: {}".format(task_id, payload)) | 519 | raise PlanningFailedError("could not extract a graph_manifest for task {}: {}".format(task_id, payload)) |
| 380 | - graph_manifest = _normalize_graph_manifest(graph_manifest) | 520 | + new_manifest = _normalize_graph_manifest(new_manifest) |
| 521 | + | ||
| 522 | + if existing_manifest is None: | ||
| 523 | + run_id = str(uuid.uuid4()) | ||
| 524 | + conn.execute( | ||
| 525 | + "INSERT INTO work_graph_run (run_id, task_id, org_node, graph_manifest, status) " | ||
| 526 | + "VALUES (?, ?, ?, ?, 'planning')", | ||
| 527 | + (run_id, task_id, org_node, json.dumps(new_manifest)), | ||
| 528 | + ) | ||
| 529 | + conn.commit() | ||
| 530 | + return {"run_id": run_id, "graph_manifest": new_manifest, "new_node_ids": []} | ||
| 381 | 531 | ||
| 382 | - run_id = str(uuid.uuid4()) | 532 | + new_node_ids = [n["node_id"] for n in new_manifest["nodes"]] |
| 533 | + existing_ids_set = {n["node_id"] for n in existing_manifest["nodes"]} | ||
| 534 | + collision = existing_ids_set & set(new_node_ids) | ||
| 535 | + if collision: | ||
| 536 | + raise PlanningFailedError( | ||
| 537 | + "append_to_run_id={} node_id collision with existing nodes: {}".format(append_to_run_id, collision) | ||
| 538 | + ) | ||
| 539 | + merged = { | ||
| 540 | + "nodes": existing_manifest["nodes"] + new_manifest["nodes"], | ||
| 541 | + "edges": existing_manifest.get("edges", []) + new_manifest.get("edges", []), | ||
| 542 | + } | ||
| 543 | + # _validate_loop_back_targets over the MERGED graph, not just the new | ||
| 544 | + # nodes -- a new node's on_fail.loop_back could (in principle) target | ||
| 545 | + # an existing node, and _normalize_graph_manifest already validated | ||
| 546 | + # new_manifest in isolation above, not against the merged shape. | ||
| 547 | + _validate_loop_back_targets(merged) | ||
| 383 | conn.execute( | 548 | conn.execute( |
| 384 | - "INSERT INTO work_graph_run (run_id, task_id, org_node, graph_manifest, status) " | ||
| 385 | - "VALUES (?, ?, ?, ?, 'planning')", | ||
| 386 | - (run_id, task_id, org_node, json.dumps(graph_manifest)), | 549 | + "UPDATE work_graph_run SET graph_manifest = ? WHERE run_id = ?", |
| 550 | + (json.dumps(merged), append_to_run_id), | ||
| 387 | ) | 551 | ) |
| 388 | conn.commit() | 552 | conn.commit() |
| 389 | - return {"run_id": run_id, "graph_manifest": graph_manifest} | 553 | + return {"run_id": append_to_run_id, "graph_manifest": merged, "new_node_ids": new_node_ids} |
| 390 | 554 | ||
| 391 | 555 | ||
| 392 | def generic_node_prompt_builder(node, outputs): | 556 | def generic_node_prompt_builder(node, outputs): |
| 393 | """E (2026-08-06): the `build_prompt(node, outputs)` run_graph() needs | 557 | """E (2026-08-06): the `build_prompt(node, outputs)` run_graph() needs |
| 394 | - for a graph_manifest produced via _GENERIC_PROMPT_TEMPLATE -- lives | 558 | + for a graph_manifest produced via `_generic_prompt_template()` -- lives |
| 395 | here, not in e2e_pipeline.py, because it's tightly coupled to the | 559 | here, not in e2e_pipeline.py, because it's tightly coupled to the |
| 396 | `task_prompt` field convention that prompt asks for (keeping both in | 560 | `task_prompt` field convention that prompt asks for (keeping both in |
| 397 | the same module means a field rename can't silently desync producer | 561 | the same module means a field rename can't silently desync producer |
| @@ -399,3 +563,31 @@ def generic_node_prompt_builder(node, outputs): | @@ -399,3 +563,31 @@ def generic_node_prompt_builder(node, outputs): | ||
| 399 | per the design-dynamic-work-graph Skill's contract, so `outputs` | 563 | per the design-dynamic-work-graph Skill's contract, so `outputs` |
| 400 | (already-completed dependency content) isn't needed here.""" | 564 | (already-completed dependency content) isn't needed here.""" |
| 401 | return node["task_prompt"] | 565 | return node["task_prompt"] |
| 566 | + | ||
| 567 | + | ||
| 568 | +def dispatch_build_prompt(planner_builder, generic_builder): | ||
| 569 | + """2026-08-12 (Phase 2 §6.1): a build_prompt(node, outputs) that picks | ||
| 570 | + per-NODE between planner_builder (Planner's artifact_type-keyed nodes, | ||
| 571 | + e.g. e2e_pipeline._planner_prompt_builder's return value) and | ||
| 572 | + generic_builder (task_prompt-keyed nodes, e.g. generic_node_prompt_ | ||
| 573 | + builder above) -- needed once a single graph_manifest can genuinely | ||
| 574 | + contain BOTH shapes side by side, which append_to_run_id makes real | ||
| 575 | + for the first time (Planner's original nodes + Coder's/Validator's | ||
| 576 | + later self-planned ones, merged into one run). Every pre-Phase-2 | ||
| 577 | + caller only ever had ONE shape for a whole graph and dispatched once | ||
| 578 | + up front; this replaces that with a per-node check so a caller doesn't | ||
| 579 | + need to know in advance whether ANY given node in a merged graph is | ||
| 580 | + Planner-shaped or generic-shaped. Shared here (not duplicated in | ||
| 581 | + e2e_pipeline.py/work_graph_fork.py, its two real callers) so the two | ||
| 582 | + don't drift on what "neither shape" should do.""" | ||
| 583 | + def build_prompt(node, outputs): | ||
| 584 | + if "artifact_type" in node: | ||
| 585 | + return planner_builder(node, outputs) | ||
| 586 | + if "task_prompt" in node: | ||
| 587 | + return generic_builder(node, outputs) | ||
| 588 | + raise ValueError( | ||
| 589 | + "node {} has neither 'artifact_type' nor 'task_prompt' -- can't dispatch a build_prompt".format( | ||
| 590 | + node.get("node_id") | ||
| 591 | + ) | ||
| 592 | + ) | ||
| 593 | + return build_prompt |
-
mentioned in commit 82368fc6d3a38e988a10509bc8526465276529a8