Commit b160378d44043e28c5aeecbc55d90ec06fe06a84
1 parent
f4cb58b8
U-014: end-to-end pipeline (c_artifact_write + Planner->Coder->Validator orchestration)
Closes the c_artifact_write gap U-011 flagged (03-C §3.7, unowned by any Unit despite REQ-02 needing it directly): src/artifact_write.py writes content to the target project, commits it to the target's own git history, validates depends_on_artifact_ids against existing rows before writing anything (REQ-02's failure case), creates the immutable artifact row. validation_status is set to 'passed' unconditionally -- noted as a real simplification (semantic content review, e.g. REQ-02's "PRD 里出现矛盾" boundary case, needs its own LLM-judgment step this codebase doesn't build yet). src/e2e_pipeline.py orchestrates architecture doc §11's MVP criteria 1-3: plan_documents() makes 4 real sequential Planner calls (BRD, then PRD referencing BRD, then SDD referencing PRD, then Benchmark referencing SDD), writing each via c_artifact_write with correct lineage. implement_code() calls Coder once, captures the resulting `git diff` as code_diff and the Coder's own response as test_report. validate_implementation() calls Validator once and parses an explicit 'VERDICT: PASS|FAIL|INCONCLUSIVE' line. run_end_to_end() chains create_task -> plan -> implement -> validate, using U-010's trigger_org_node_full only for task.status bookkeeping (a lightweight "confirm ready" call per phase) and plain org_node_trigger calls for actual content generation, since trigger_org_node_full's contract returns a status dict, not text. Real-verified: plan_documents ran for real against "实现贪吃蛇游戏" -- all 4 documents produced, correctly lineaged, validation_status=passed, real content (>100 chars each), ~5m45s total for the 4 sequential calls. Deliberately NOT run for real: run_end_to_end's Coder/Validator half (real code generation + real test execution) is a different order of magnitude in cost/time/uncertainty than every other real call this session -- built and mock-tested, but not executed without checking first. 19 new mocked stdlib unittest cases (4 for artifact_write, 5 for the pipeline orchestration, all real/unmocked for artifact_write since git operations are fast/free/local same as U-013's worktree tests) + 1 gated real test. 127 total mocked/real-local tests across U-001-014, all passing. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Showing
7 changed files
with
573 additions
and
1 deletions
docs/development-plan.md
| @@ -140,6 +140,8 @@ | @@ -140,6 +140,8 @@ | ||
| 140 | - **U-014 [Value Path]** 完整跑通"实现贪吃蛇游戏"验收场景,从 `goal_text` 到 `task.status=done` | 140 | - **U-014 [Value Path]** 完整跑通"实现贪吃蛇游戏"验收场景,从 `goal_text` 到 `task.status=done` |
| 141 | - 依赖:U-011, U-012, U-013 | 141 | - 依赖:U-011, U-012, U-013 |
| 142 | - Done:架构文档 §11 MVP 验收标准第 1-3 条全部达成(Planner 出四件套,Coder 自测通过,Validator 归因产出 bug list) | 142 | - Done:架构文档 §11 MVP 验收标准第 1-3 条全部达成(Planner 出四件套,Coder 自测通过,Validator 归因产出 bug list) |
| 143 | + - **实现补充(2026-07-31)**:这是前面 U-009~U-013 五次"Done 标准过头"里被推迟的 happy 用例真正兑现的地方。落地时顺手把 U-011 记录的 `c_artifact_write`(03-C §3.7)缺口补上了——`src/artifact_write.py`,之前没有任何 Unit 认领。编排代码是 `src/e2e_pipeline.py`:`plan_documents`(真实调用 planner 依次产出 BRD→PRD→SDD→Benchmark,各自用 `c_artifact_write` 落盘+提交+建血缘)、`implement_code`(真实调用 coder,用会话前后的 `git diff` 作为 `code_diff`、coder 自己的回复文本作为 `test_report`)、`validate_implementation`(真实调用 validator,解析约定的 `VERDICT: PASS|FAIL|INCONCLUSIVE` 行)、`run_end_to_end`(串联全部四步) | ||
| 144 | + - **真实验证的范围(2026-07-31)**:`plan_documents` 这一段(4 次真实 planner 调用产出真实 BRD/PRD/SDD/Benchmark,血缘链验证)成本和之前几个 Unit 同量级,已经真实跑过。`run_end_to_end` 的 Coder/Validator 那一半——真实写代码、真实跑测试——量级完全不同(可能是分钟级、真金白银的开销,而且结果不确定),没有不请示直接跑,等用户决定要不要真的跑一次完整闭环 | ||
| 143 | 145 | ||
| 144 | --- | 146 | --- |
| 145 | 147 |
docs/sdd/unit_matrix.json
| @@ -26,7 +26,7 @@ | @@ -26,7 +26,7 @@ | ||
| 26 | {"id": "U-011", "phase": "PH1", "sub_phase": "PH1-S2", "name": "c_work_graph_plan 实现(只规划不执行,见 development-plan.md 2026-07-31 订正)", "classification": "Value Path", "depends_on": ["U-004", "U-010"], "coverage": {"model": ["artifact", "work_graph_run"], "view": [], "controller": ["c_work_graph_plan"], "process": [], "requirement": ["REQ-02"]}, "done": "graph_manifest 通过结构校验(brd/prd/sdd/benchmark 四节点+依赖链),work_graph_run(status=planning) 落地;REQ-02 happy 用例整体随 U-012/U-013 累计满足。c_artifact_write(03-C §3.7)目前没有任何 Unit 认领,是已知缺口"}, | 26 | {"id": "U-011", "phase": "PH1", "sub_phase": "PH1-S2", "name": "c_work_graph_plan 实现(只规划不执行,见 development-plan.md 2026-07-31 订正)", "classification": "Value Path", "depends_on": ["U-004", "U-010"], "coverage": {"model": ["artifact", "work_graph_run"], "view": [], "controller": ["c_work_graph_plan"], "process": [], "requirement": ["REQ-02"]}, "done": "graph_manifest 通过结构校验(brd/prd/sdd/benchmark 四节点+依赖链),work_graph_run(status=planning) 落地;REQ-02 happy 用例整体随 U-012/U-013 累计满足。c_artifact_write(03-C §3.7)目前没有任何 Unit 认领,是已知缺口"}, |
| 27 | {"id": "U-012", "phase": "PH1", "sub_phase": "PH1-S2", "name": "c_work_graph_execute 完整实现(见 development-plan.md 2026-07-31 范围说明)", "classification": "Value Path", "depends_on": ["U-006", "U-011"], "coverage": {"model": [], "view": [], "controller": ["c_work_graph_execute"], "process": ["p_work_graph_execution", "p_usage_limit_wait"], "requirement": ["REQ-07", "REQ-14"]}, "done": "REQ-07 boundary/failure 通过(happy 需 PH2 REQ-04/05/06 就绪后累计满足);REQ-14 happy 通过。退避重试是整图级别,非逐节点;CLASSIFIER_BLOCKED 用单次调用 permission_denials≥3 近似判定"}, | 27 | {"id": "U-012", "phase": "PH1", "sub_phase": "PH1-S2", "name": "c_work_graph_execute 完整实现(见 development-plan.md 2026-07-31 范围说明)", "classification": "Value Path", "depends_on": ["U-006", "U-011"], "coverage": {"model": [], "view": [], "controller": ["c_work_graph_execute"], "process": ["p_work_graph_execution", "p_usage_limit_wait"], "requirement": ["REQ-07", "REQ-14"]}, "done": "REQ-07 boundary/failure 通过(happy 需 PH2 REQ-04/05/06 就绪后累计满足);REQ-14 happy 通过。退避重试是整图级别,非逐节点;CLASSIFIER_BLOCKED 用单次调用 permission_denials≥3 近似判定"}, |
| 28 | {"id": "U-013", "phase": "PH1", "sub_phase": "PH1-S2", "name": "c_work_node_run + git worktree 隔离(见 development-plan.md 2026-07-31 订正)", "classification": "Value Path", "depends_on": ["U-012"], "coverage": {"model": ["work_node_execution"], "view": [], "controller": ["c_work_node_run"], "process": [], "requirement": ["REQ-03"]}, "done": "REQ-03 boundary 通过(真实 git worktree 隔离+merge,无冲突);happy 需 c_artifact_write 落地后累计满足"}, | 28 | {"id": "U-013", "phase": "PH1", "sub_phase": "PH1-S2", "name": "c_work_node_run + git worktree 隔离(见 development-plan.md 2026-07-31 订正)", "classification": "Value Path", "depends_on": ["U-012"], "coverage": {"model": ["work_node_execution"], "view": [], "controller": ["c_work_node_run"], "process": [], "requirement": ["REQ-03"]}, "done": "REQ-03 boundary 通过(真实 git worktree 隔离+merge,无冲突);happy 需 c_artifact_write 落地后累计满足"}, |
| 29 | - {"id": "U-014", "phase": "PH1", "sub_phase": "PH1-S3", "name": "贪吃蛇场景端到端验收", "classification": "Value Path", "depends_on": ["U-011", "U-012", "U-013"], "coverage": {"model": [], "view": [], "controller": [], "process": [], "requirement": []}, "done": "架构文档 §11 验收标准第 1-3 条全部达成"}, | 29 | + {"id": "U-014", "phase": "PH1", "sub_phase": "PH1-S3", "name": "贪吃蛇场景端到端验收(含补建 c_artifact_write,见 development-plan.md 2026-07-31 实现补充)", "classification": "Value Path", "depends_on": ["U-011", "U-012", "U-013"], "coverage": {"model": ["artifact"], "view": [], "controller": ["c_artifact_write"], "process": [], "requirement": []}, "done": "架构文档 §11 验收标准第 1-3 条全部达成;plan_documents(Planner 四件套+血缘)真实验证过,Coder/Validator 全流程真实跑通待用户确认后执行"}, |
| 30 | 30 | ||
| 31 | {"id": "U-015", "phase": "PH2", "sub_phase": "PH2-S1", "name": "p_validator_root_cause_trace 实现", "classification": "Risk Gate", "depends_on": ["U-013"], "coverage": {"model": ["bug_record"], "view": [], "controller": ["c_bug_record_create"], "process": ["p_validator_root_cause_trace"], "requirement": ["REQ-04"]}, "done": "REQ-04 happy/boundary/failure 通过"}, | 31 | {"id": "U-015", "phase": "PH2", "sub_phase": "PH2-S1", "name": "p_validator_root_cause_trace 实现", "classification": "Risk Gate", "depends_on": ["U-013"], "coverage": {"model": ["bug_record"], "view": [], "controller": ["c_bug_record_create"], "process": ["p_validator_root_cause_trace"], "requirement": ["REQ-04"]}, "done": "REQ-04 happy/boundary/failure 通过"}, |
| 32 | {"id": "U-016", "phase": "PH2", "sub_phase": "PH2-S1", "name": "c_org_loop_route + fast_lane/full_replan", "classification": "Risk Gate", "depends_on": ["U-015"], "coverage": {"model": [], "view": [], "controller": ["c_org_loop_route"], "process": [], "requirement": ["REQ-05", "REQ-09"]}, "done": "REQ-05/REQ-09 全部通过"}, | 32 | {"id": "U-016", "phase": "PH2", "sub_phase": "PH2-S1", "name": "c_org_loop_route + fast_lane/full_replan", "classification": "Risk Gate", "depends_on": ["U-015"], "coverage": {"model": [], "view": [], "controller": ["c_org_loop_route"], "process": [], "requirement": ["REQ-05", "REQ-09"]}, "done": "REQ-05/REQ-09 全部通过"}, |
src/artifact_write.py
0 → 100644
| 1 | +"""c_artifact_write (03-C §3.7): the only writer of `artifact` rows. | ||
| 2 | + | ||
| 3 | +Flagged as an unowned gap when scoping U-011 (REQ-02's happy AND failure | ||
| 4 | +cases both need it directly, and no Unit in development-plan.md's 43-unit | ||
| 5 | +breakdown built it). Built now because U-014 (end-to-end acceptance) is the | ||
| 6 | +first Unit that actually needs artifacts to exist for real, not seeded. | ||
| 7 | + | ||
| 8 | +Simplification, noted rather than silently assumed: `validation_status` is | ||
| 9 | +set to 'passed' unconditionally at write time. REQ-02's boundary case | ||
| 10 | +("PRD 中出现互相矛盾的场景假设时,一致性校验应在 verifying 态被拦截") describes | ||
| 11 | +semantic content review, which would need its own LLM-judgment call this | ||
| 12 | +codebase doesn't build yet (distinct from work_node_execution's schema-shape | ||
| 13 | +validation, already real since U-013) -- out of scope here, same as every | ||
| 14 | +other REQ boundary/failure case this session has deferred rather than faked. | ||
| 15 | +""" | ||
| 16 | + | ||
| 17 | +import json | ||
| 18 | +import os | ||
| 19 | +import subprocess | ||
| 20 | +import uuid | ||
| 21 | + | ||
| 22 | +_GIT_IDENTITY = ["-c", "user.name=graphlemon", "-c", "user.email=graphlemon@localhost"] | ||
| 23 | + | ||
| 24 | + | ||
| 25 | +class InvalidDependencyError(Exception): | ||
| 26 | + """REQ-02 failure case: depends_on_artifact_ids references a nonexistent artifact_id.""" | ||
| 27 | + | ||
| 28 | + | ||
| 29 | +class GitCommitFailedError(Exception): | ||
| 30 | + """GIT_COMMIT_FAILED -- 03-C §3.7.""" | ||
| 31 | + | ||
| 32 | + | ||
| 33 | +def write_artifact( | ||
| 34 | + conn, target_folder_path, artifact_type, task_id, produced_by_run_id, | ||
| 35 | + file_path, content, depends_on_artifact_ids=None, | ||
| 36 | +): | ||
| 37 | + """Writes `content` to `<target_folder_path>/<file_path>`, commits it to | ||
| 38 | + the target project's own git history (not graphlemon's -- 01-M 1.5: | ||
| 39 | + artifact.git_commit_sha is the *target project's* commit), and creates | ||
| 40 | + the immutable `artifact` row. Returns {"artifact_id", "git_commit_sha"}. | ||
| 41 | + """ | ||
| 42 | + depends_on_artifact_ids = depends_on_artifact_ids or [] | ||
| 43 | + for dep_id in depends_on_artifact_ids: | ||
| 44 | + if conn.execute("SELECT 1 FROM artifact WHERE artifact_id = ?", (dep_id,)).fetchone() is None: | ||
| 45 | + raise InvalidDependencyError("depends_on_artifact_ids references nonexistent artifact: " + dep_id) | ||
| 46 | + | ||
| 47 | + abs_path = os.path.join(target_folder_path, file_path) | ||
| 48 | + os.makedirs(os.path.dirname(abs_path), exist_ok=True) | ||
| 49 | + with open(abs_path, "w", encoding="utf-8") as f: | ||
| 50 | + f.write(content) | ||
| 51 | + | ||
| 52 | + try: | ||
| 53 | + subprocess.run( | ||
| 54 | + ["git", "add", file_path], cwd=target_folder_path, check=True, capture_output=True, text=True | ||
| 55 | + ) | ||
| 56 | + subprocess.run( | ||
| 57 | + ["git"] + _GIT_IDENTITY + [ | ||
| 58 | + "commit", "-m", "graphlemon: {} artifact for task {}".format(artifact_type, task_id) | ||
| 59 | + ], | ||
| 60 | + cwd=target_folder_path, check=True, capture_output=True, text=True, | ||
| 61 | + ) | ||
| 62 | + sha = subprocess.run( | ||
| 63 | + ["git", "rev-parse", "HEAD"], cwd=target_folder_path, check=True, capture_output=True, text=True | ||
| 64 | + ).stdout.strip() | ||
| 65 | + except subprocess.CalledProcessError as e: | ||
| 66 | + raise GitCommitFailedError(str(e)) from e | ||
| 67 | + | ||
| 68 | + artifact_id = str(uuid.uuid4()) | ||
| 69 | + conn.execute( | ||
| 70 | + "INSERT INTO artifact " | ||
| 71 | + "(artifact_id, artifact_type, task_id, produced_by_run_id, file_path, git_commit_sha, " | ||
| 72 | + "depends_on_artifact_ids, validation_status) VALUES (?, ?, ?, ?, ?, ?, ?, 'passed')", | ||
| 73 | + ( | ||
| 74 | + artifact_id, artifact_type, task_id, produced_by_run_id, file_path, sha, | ||
| 75 | + json.dumps(depends_on_artifact_ids), | ||
| 76 | + ), | ||
| 77 | + ) | ||
| 78 | + conn.commit() | ||
| 79 | + return {"artifact_id": artifact_id, "git_commit_sha": sha} |
src/e2e_pipeline.py
0 → 100644
| 1 | +"""U-014: end-to-end orchestration for architecture doc §11 MVP acceptance | ||
| 2 | +criteria 1-3 (Planner produces validated BRD/PRD/SDD/Benchmark; Coder | ||
| 3 | +implements against the SDD and self-tests pass; Validator runs | ||
| 4 | +benchmark/tests and produces a root-cause-attributed verdict). Criteria 4-7 | ||
| 5 | +(Org-level Loop, breakpoint recovery, ...) are explicitly out of this Unit's | ||
| 6 | +scope -- PH2 (U-017/U-021). | ||
| 7 | + | ||
| 8 | +Status-transition bookkeeping (task.status, precondition guards) stays | ||
| 9 | +U-010's job (org_node_lifecycle.trigger_org_node_full) -- called once per | ||
| 10 | +phase with a lightweight "confirm ready" prompt. Content generation is | ||
| 11 | +separate, plain org_node_trigger.trigger_org_node calls (no status | ||
| 12 | +side-effects) so each document/step's real text is available to write as an | ||
| 13 | +artifact -- trigger_org_node_full's own contract (a status dict, no text) | ||
| 14 | +isn't changed to accommodate this. | ||
| 15 | +""" | ||
| 16 | + | ||
| 17 | +import os | ||
| 18 | +import subprocess | ||
| 19 | + | ||
| 20 | +from src import artifact_write, org_node_lifecycle, org_node_trigger, task_create | ||
| 21 | + | ||
| 22 | +_READY_PROMPT = "Confirm you're ready to begin. Take no other action yet." | ||
| 23 | + | ||
| 24 | +_DOC_PROMPTS = { | ||
| 25 | + "brd": ( | ||
| 26 | + "Write a BRD (Business Requirement Document) for this goal, following your own " | ||
| 27 | + "SOUL.md methodology: {goal_text}\n" | ||
| 28 | + "Output ONLY the BRD content in Markdown -- no preamble, no meta-commentary about " | ||
| 29 | + "what you're about to do." | ||
| 30 | + ), | ||
| 31 | + "prd": ( | ||
| 32 | + "Here is the approved BRD for goal '{goal_text}':\n\n{brd_content}\n\n" | ||
| 33 | + "Write the PRD (Product Requirement Document) based on this BRD. " | ||
| 34 | + "Output ONLY the PRD content in Markdown." | ||
| 35 | + ), | ||
| 36 | + "sdd": ( | ||
| 37 | + "Here is the approved PRD for goal '{goal_text}':\n\n{prd_content}\n\n" | ||
| 38 | + "Write the SDD (System Design Document) based on this PRD, including a Development " | ||
| 39 | + "Unit breakdown. Output ONLY the SDD content in Markdown." | ||
| 40 | + ), | ||
| 41 | + "benchmark": ( | ||
| 42 | + "Here is the approved SDD for goal '{goal_text}':\n\n{sdd_content}\n\n" | ||
| 43 | + "Write the Benchmark/acceptance-test document based on this SDD: concrete test " | ||
| 44 | + "scenarios (happy/boundary/failure) with observable pass criteria. " | ||
| 45 | + "Output ONLY the Benchmark content in Markdown." | ||
| 46 | + ), | ||
| 47 | +} | ||
| 48 | + | ||
| 49 | +_CODER_PROMPT = ( | ||
| 50 | + "Implement this goal per the approved SDD below. Write real code and real tests, run the " | ||
| 51 | + "tests yourself, and only stop once they pass (or once you've exhausted reasonable attempts " | ||
| 52 | + "and can explain why not). Goal: {goal_text}\n\nSDD:\n\n{sdd_content}" | ||
| 53 | +) | ||
| 54 | + | ||
| 55 | +_VALIDATOR_PROMPT = ( | ||
| 56 | + "Independently verify the implementation for this goal against the Benchmark below: run " | ||
| 57 | + "the tests yourself, don't just trust the Coder's claims. Reply with a verdict line " | ||
| 58 | + "'VERDICT: PASS' or 'VERDICT: FAIL' or 'VERDICT: INCONCLUSIVE' followed by your reasoning " | ||
| 59 | + "and, if FAIL, a root_cause_category (planning|coder|skill_defect) with detail. " | ||
| 60 | + "Goal: {goal_text}\n\nBenchmark:\n\n{benchmark_content}" | ||
| 61 | +) | ||
| 62 | + | ||
| 63 | + | ||
| 64 | +def plan_documents(conn, task_id, run_id, goal_text, target_folder_path, timeout=300): | ||
| 65 | + """Architecture §11 criterion 1: real Planner calls producing BRD->PRD->SDD | ||
| 66 | + ->Benchmark, each written via c_artifact_write with correct lineage. | ||
| 67 | + Returns {"brd": artifact_id, "prd": ..., "sdd": ..., "benchmark": ...}. | ||
| 68 | + """ | ||
| 69 | + artifact_ids = {} | ||
| 70 | + contents = {} | ||
| 71 | + prev_type = None | ||
| 72 | + for artifact_type in ("brd", "prd", "sdd", "benchmark"): | ||
| 73 | + prompt = _DOC_PROMPTS[artifact_type].format( | ||
| 74 | + goal_text=goal_text, **{k + "_content": v for k, v in contents.items()} | ||
| 75 | + ) | ||
| 76 | + content = org_node_trigger.trigger_org_node("planner", prompt, target_folder_path, timeout=timeout) | ||
| 77 | + deps = [artifact_ids[prev_type]] if prev_type else [] | ||
| 78 | + result = artifact_write.write_artifact( | ||
| 79 | + conn, target_folder_path, artifact_type, task_id, run_id, | ||
| 80 | + ".graphlemon/runs/{}/{}.md".format(task_id, artifact_type), content, | ||
| 81 | + depends_on_artifact_ids=deps, | ||
| 82 | + ) | ||
| 83 | + artifact_ids[artifact_type] = result["artifact_id"] | ||
| 84 | + contents[artifact_type] = content | ||
| 85 | + prev_type = artifact_type | ||
| 86 | + return artifact_ids | ||
| 87 | + | ||
| 88 | + | ||
| 89 | +def implement_code(conn, task_id, run_id, sdd_content, goal_text, target_folder_path, timeout=1200): | ||
| 90 | + """Architecture §11 criterion 2: real Coder call, then registers whatever | ||
| 91 | + the Coder actually committed as code_diff (the diff itself) and | ||
| 92 | + test_report (the Coder's own response text, which per its migrated SOUL.md | ||
| 93 | + is required to include real test commands + real output) artifacts. | ||
| 94 | + """ | ||
| 95 | + before_sha = _current_commit(target_folder_path) | ||
| 96 | + response = org_node_trigger.trigger_org_node( | ||
| 97 | + "coder", _CODER_PROMPT.format(goal_text=goal_text, sdd_content=sdd_content), | ||
| 98 | + target_folder_path, timeout=timeout, | ||
| 99 | + ) | ||
| 100 | + diff = subprocess.run( | ||
| 101 | + ["git", "diff", before_sha, "HEAD"], cwd=target_folder_path, capture_output=True, text=True | ||
| 102 | + ).stdout | ||
| 103 | + | ||
| 104 | + diff_result = artifact_write.write_artifact( | ||
| 105 | + conn, target_folder_path, "code_diff", task_id, run_id, | ||
| 106 | + ".graphlemon/runs/{}/code_diff.patch".format(task_id), diff or "(no diff -- see test_report)", | ||
| 107 | + ) | ||
| 108 | + report_result = artifact_write.write_artifact( | ||
| 109 | + conn, target_folder_path, "test_report", task_id, run_id, | ||
| 110 | + ".graphlemon/runs/{}/test_report.md".format(task_id), response, | ||
| 111 | + ) | ||
| 112 | + return {"code_diff": diff_result["artifact_id"], "test_report": report_result["artifact_id"]} | ||
| 113 | + | ||
| 114 | + | ||
| 115 | +def validate_implementation(task_id, benchmark_content, goal_text, target_folder_path, timeout=1200): | ||
| 116 | + """Architecture §11 criterion 3: real Validator call, parses the | ||
| 117 | + 'VERDICT: PASS|FAIL|INCONCLUSIVE' line this Unit's prompt asks for. | ||
| 118 | + Returns {"verdict", "raw_response"}. | ||
| 119 | + """ | ||
| 120 | + response = org_node_trigger.trigger_org_node( | ||
| 121 | + "validator", | ||
| 122 | + _VALIDATOR_PROMPT.format(goal_text=goal_text, benchmark_content=benchmark_content), | ||
| 123 | + target_folder_path, timeout=timeout, | ||
| 124 | + ) | ||
| 125 | + verdict = "INCONCLUSIVE" | ||
| 126 | + for line in response.splitlines(): | ||
| 127 | + stripped = line.strip().upper() | ||
| 128 | + if stripped.startswith("VERDICT:"): | ||
| 129 | + for candidate in ("PASS", "FAIL", "INCONCLUSIVE"): | ||
| 130 | + if candidate in stripped: | ||
| 131 | + verdict = candidate | ||
| 132 | + break | ||
| 133 | + break | ||
| 134 | + return {"verdict": verdict, "raw_response": response} | ||
| 135 | + | ||
| 136 | + | ||
| 137 | +def _current_commit(repo_path): | ||
| 138 | + return subprocess.run( | ||
| 139 | + ["git", "rev-parse", "HEAD"], cwd=repo_path, capture_output=True, text=True | ||
| 140 | + ).stdout.strip() | ||
| 141 | + | ||
| 142 | + | ||
| 143 | +def run_end_to_end(conn, target_folder_path, goal_text): | ||
| 144 | + """Full orchestration: create_task -> plan_documents -> implement_code -> | ||
| 145 | + validate_implementation. Returns a dict with every intermediate result, | ||
| 146 | + for inspection/debugging rather than hiding it behind a single verdict. | ||
| 147 | + """ | ||
| 148 | + project_id = conn.execute("SELECT project_id FROM project WHERE target_folder_path = ?", (target_folder_path,)).fetchone()[0] | ||
| 149 | + task_id = task_create.create_task(conn, project_id, goal_text)["task_id"] | ||
| 150 | + | ||
| 151 | + org_node_lifecycle.trigger_org_node_full(conn, "planner", task_id, target_folder_path, _READY_PROMPT) | ||
| 152 | + # trigger_org_node_full itself doesn't create a work_graph_run row (that's | ||
| 153 | + # U-011's plan_work_graph, a separate call this pipeline doesn't invoke, | ||
| 154 | + # since REQ-02's graph_manifest and this pipeline's actual document | ||
| 155 | + # content are orthogonal concerns) -- artifact.produced_by_run_id is a | ||
| 156 | + # real FK, so a row has to exist; this one just records that the e2e | ||
| 157 | + # pipeline (not a real planned Work Graph) produced these artifacts | ||
| 158 | + run_id = "e2e-" + task_id | ||
| 159 | + conn.execute( | ||
| 160 | + "INSERT INTO work_graph_run (run_id, task_id, org_node, graph_manifest, status) " | ||
| 161 | + "VALUES (?, ?, 'planner', '{}', 'running')", | ||
| 162 | + (run_id, task_id), | ||
| 163 | + ) | ||
| 164 | + conn.commit() | ||
| 165 | + | ||
| 166 | + artifact_ids = plan_documents(conn, task_id, run_id, goal_text, target_folder_path) | ||
| 167 | + sdd_content = _read_artifact_content(conn, target_folder_path, artifact_ids["sdd"]) | ||
| 168 | + benchmark_content = _read_artifact_content(conn, target_folder_path, artifact_ids["benchmark"]) | ||
| 169 | + | ||
| 170 | + org_node_lifecycle.trigger_org_node_full(conn, "coder", task_id, target_folder_path, _READY_PROMPT) | ||
| 171 | + code_artifact_ids = implement_code(conn, task_id, run_id, sdd_content, goal_text, target_folder_path) | ||
| 172 | + | ||
| 173 | + org_node_lifecycle.trigger_org_node_full(conn, "validator", task_id, target_folder_path, _READY_PROMPT) | ||
| 174 | + verdict = validate_implementation(task_id, benchmark_content, goal_text, target_folder_path) | ||
| 175 | + | ||
| 176 | + return { | ||
| 177 | + "task_id": task_id, | ||
| 178 | + "doc_artifact_ids": artifact_ids, | ||
| 179 | + "code_artifact_ids": code_artifact_ids, | ||
| 180 | + "verdict": verdict, | ||
| 181 | + } | ||
| 182 | + | ||
| 183 | + | ||
| 184 | +def _read_artifact_content(conn, target_folder_path, artifact_id): | ||
| 185 | + file_path = conn.execute("SELECT file_path FROM artifact WHERE artifact_id = ?", (artifact_id,)).fetchone()[0] | ||
| 186 | + with open(os.path.join(target_folder_path, file_path), encoding="utf-8") as f: | ||
| 187 | + return f.read() |
src/test_artifact_write.py
0 → 100644
| 1 | +"""U-014 (c_artifact_write) tests: real git, no mocking needed -- writing a | ||
| 2 | +file and committing it is fast/free/local, same reasoning as U-013's | ||
| 3 | +worktree tests. | ||
| 4 | + | ||
| 5 | +Run: python3 -m unittest src.test_artifact_write -v | ||
| 6 | +""" | ||
| 7 | + | ||
| 8 | +import json | ||
| 9 | +import os | ||
| 10 | +import subprocess | ||
| 11 | +import tempfile | ||
| 12 | +import unittest | ||
| 13 | + | ||
| 14 | +from src import artifact_write, schema | ||
| 15 | + | ||
| 16 | + | ||
| 17 | +def _init_git_repo(path): | ||
| 18 | + subprocess.run(["git", "init"], cwd=path, check=True, capture_output=True) | ||
| 19 | + | ||
| 20 | + | ||
| 21 | +class ArtifactWriteTestBase(unittest.TestCase): | ||
| 22 | + def setUp(self): | ||
| 23 | + self.conn = schema.connect(":memory:") | ||
| 24 | + schema.create_schema(self.conn) | ||
| 25 | + self.tmp = tempfile.TemporaryDirectory() | ||
| 26 | + self.addCleanup(self.tmp.cleanup) | ||
| 27 | + _init_git_repo(self.tmp.name) | ||
| 28 | + self.conn.execute( | ||
| 29 | + "INSERT INTO project (project_id, name, target_folder_path, has_git_repo, status) " | ||
| 30 | + "VALUES ('proj-1', 'P', ?, 1, 'active')", | ||
| 31 | + (self.tmp.name,), | ||
| 32 | + ) | ||
| 33 | + self.conn.execute( | ||
| 34 | + "INSERT INTO task (task_id, project_id, goal_text, task_type, status, assigned_org_node) " | ||
| 35 | + "VALUES ('task-1', 'proj-1', 'goal', 'goal', 'planning_brd_prd_sdd', 'planner')" | ||
| 36 | + ) | ||
| 37 | + self.conn.execute( | ||
| 38 | + "INSERT INTO work_graph_run (run_id, task_id, org_node, graph_manifest, status) " | ||
| 39 | + "VALUES ('run-1', 'task-1', 'planner', '{}', 'running')" | ||
| 40 | + ) | ||
| 41 | + self.conn.commit() | ||
| 42 | + | ||
| 43 | + def tearDown(self): | ||
| 44 | + self.conn.close() | ||
| 45 | + | ||
| 46 | + | ||
| 47 | +class HappyPathTest(ArtifactWriteTestBase): | ||
| 48 | + def test_writes_file_commits_and_creates_row(self): | ||
| 49 | + result = artifact_write.write_artifact( | ||
| 50 | + self.conn, self.tmp.name, "brd", "task-1", "run-1", | ||
| 51 | + ".graphlemon/runs/task-1/brd.md", "# BRD\n\ncontent", | ||
| 52 | + ) | ||
| 53 | + self.assertTrue(result["artifact_id"]) | ||
| 54 | + self.assertTrue(result["git_commit_sha"]) | ||
| 55 | + | ||
| 56 | + path = os.path.join(self.tmp.name, ".graphlemon/runs/task-1/brd.md") | ||
| 57 | + with open(path) as f: | ||
| 58 | + self.assertEqual(f.read(), "# BRD\n\ncontent") | ||
| 59 | + | ||
| 60 | + log = subprocess.run( | ||
| 61 | + ["git", "log", "--oneline"], cwd=self.tmp.name, check=True, capture_output=True, text=True | ||
| 62 | + ) | ||
| 63 | + self.assertIn("brd artifact for task task-1", log.stdout) | ||
| 64 | + | ||
| 65 | + row = self.conn.execute( | ||
| 66 | + "SELECT artifact_type, task_id, validation_status, depends_on_artifact_ids " | ||
| 67 | + "FROM artifact WHERE artifact_id = ?", | ||
| 68 | + (result["artifact_id"],), | ||
| 69 | + ).fetchone() | ||
| 70 | + self.assertEqual(row[0], "brd") | ||
| 71 | + self.assertEqual(row[1], "task-1") | ||
| 72 | + self.assertEqual(row[2], "passed") | ||
| 73 | + self.assertEqual(json.loads(row[3]), []) | ||
| 74 | + | ||
| 75 | + def test_lineage_chain_brd_to_prd_to_sdd(self): | ||
| 76 | + brd = artifact_write.write_artifact( | ||
| 77 | + self.conn, self.tmp.name, "brd", "task-1", "run-1", ".graphlemon/runs/task-1/brd.md", "brd content" | ||
| 78 | + ) | ||
| 79 | + prd = artifact_write.write_artifact( | ||
| 80 | + self.conn, self.tmp.name, "prd", "task-1", "run-1", ".graphlemon/runs/task-1/prd.md", "prd content", | ||
| 81 | + depends_on_artifact_ids=[brd["artifact_id"]], | ||
| 82 | + ) | ||
| 83 | + sdd = artifact_write.write_artifact( | ||
| 84 | + self.conn, self.tmp.name, "sdd", "task-1", "run-1", ".graphlemon/runs/task-1/sdd.md", "sdd content", | ||
| 85 | + depends_on_artifact_ids=[prd["artifact_id"]], | ||
| 86 | + ) | ||
| 87 | + row = self.conn.execute( | ||
| 88 | + "SELECT depends_on_artifact_ids FROM artifact WHERE artifact_id = ?", (sdd["artifact_id"],) | ||
| 89 | + ).fetchone() | ||
| 90 | + self.assertEqual(json.loads(row[0]), [prd["artifact_id"]]) | ||
| 91 | + | ||
| 92 | + | ||
| 93 | +class InvalidDependencyTest(ArtifactWriteTestBase): | ||
| 94 | + def test_nonexistent_dependency_rejected_before_any_write(self): | ||
| 95 | + with self.assertRaises(artifact_write.InvalidDependencyError): | ||
| 96 | + artifact_write.write_artifact( | ||
| 97 | + self.conn, self.tmp.name, "sdd", "task-1", "run-1", | ||
| 98 | + ".graphlemon/runs/task-1/sdd.md", "content", | ||
| 99 | + depends_on_artifact_ids=["no-such-artifact"], | ||
| 100 | + ) | ||
| 101 | + self.assertFalse(os.path.exists(os.path.join(self.tmp.name, ".graphlemon/runs/task-1/sdd.md"))) | ||
| 102 | + self.assertEqual(self.conn.execute("SELECT COUNT(*) FROM artifact").fetchone()[0], 0) | ||
| 103 | + | ||
| 104 | + | ||
| 105 | +class GitCommitFailedTest(ArtifactWriteTestBase): | ||
| 106 | + def test_committing_identical_content_twice_raises_git_commit_failed(self): | ||
| 107 | + artifact_write.write_artifact( | ||
| 108 | + self.conn, self.tmp.name, "brd", "task-1", "run-1", ".graphlemon/runs/task-1/brd.md", "same content" | ||
| 109 | + ) | ||
| 110 | + # second write with byte-identical content -> `git commit` has nothing | ||
| 111 | + # to commit -> exits nonzero -> GitCommitFailedError | ||
| 112 | + with self.assertRaises(artifact_write.GitCommitFailedError): | ||
| 113 | + artifact_write.write_artifact( | ||
| 114 | + self.conn, self.tmp.name, "brd", "task-1", "run-1", ".graphlemon/runs/task-1/brd.md", "same content" | ||
| 115 | + ) | ||
| 116 | + | ||
| 117 | + | ||
| 118 | +if __name__ == "__main__": | ||
| 119 | + unittest.main() |
src/test_e2e_pipeline.py
0 → 100644
| 1 | +"""U-014 tests: end-to-end orchestration wiring, mocked at the subprocess | ||
| 2 | +boundary (CLAUDE.md Testing constraint) -- these test that plan_documents/ | ||
| 3 | +implement_code/validate_implementation/run_end_to_end call the right things | ||
| 4 | +in the right order and wire artifact lineage/task-status transitions | ||
| 5 | +correctly, not what a real Planner/Coder/Validator would actually produce. | ||
| 6 | +Real coverage (as much of it as is safe/affordable to run) is in | ||
| 7 | +src/test_e2e_pipeline_e2e.py (RUN_REAL_AGENT_E2E=1). | ||
| 8 | + | ||
| 9 | +Run: python3 -m unittest src.test_e2e_pipeline -v | ||
| 10 | +""" | ||
| 11 | + | ||
| 12 | +import json | ||
| 13 | +import subprocess | ||
| 14 | +import tempfile | ||
| 15 | +import unittest | ||
| 16 | +from unittest.mock import patch | ||
| 17 | + | ||
| 18 | +from src import e2e_pipeline, project_scaffold, schema | ||
| 19 | + | ||
| 20 | +_FAKE_DOCS = {"brd": "# BRD\nfake", "prd": "# PRD\nfake", "sdd": "# SDD\nfake", "benchmark": "# Benchmark\nfake"} | ||
| 21 | + | ||
| 22 | + | ||
| 23 | +def _fake_trigger(org_node, prompt, target_folder_path, timeout=300): | ||
| 24 | + if prompt == e2e_pipeline._READY_PROMPT: | ||
| 25 | + return "Ready." | ||
| 26 | + if "BRD (Business" in prompt: | ||
| 27 | + return _FAKE_DOCS["brd"] | ||
| 28 | + if "PRD (Product" in prompt: | ||
| 29 | + return _FAKE_DOCS["prd"] | ||
| 30 | + if "SDD (System" in prompt: | ||
| 31 | + return _FAKE_DOCS["sdd"] | ||
| 32 | + if "Benchmark/acceptance-test" in prompt: | ||
| 33 | + return _FAKE_DOCS["benchmark"] | ||
| 34 | + if prompt.startswith("Implement this goal"): | ||
| 35 | + return "Implemented. Ran `pytest`: 3 passed." | ||
| 36 | + if prompt.startswith("Independently verify"): | ||
| 37 | + return "VERDICT: PASS\nAll benchmark scenarios reproduced and passing." | ||
| 38 | + raise AssertionError("unexpected prompt: " + prompt[:80]) | ||
| 39 | + | ||
| 40 | + | ||
| 41 | +class PipelineTestBase(unittest.TestCase): | ||
| 42 | + def setUp(self): | ||
| 43 | + self.tmp = tempfile.TemporaryDirectory() | ||
| 44 | + self.addCleanup(self.tmp.cleanup) | ||
| 45 | + subprocess.run(["git", "init"], cwd=self.tmp.name, check=True, capture_output=True) | ||
| 46 | + self.conn = schema.connect(":memory:") | ||
| 47 | + schema.create_schema(self.conn) | ||
| 48 | + project_scaffold.create_project(self.conn, self.tmp.name, init_git_if_missing=False) | ||
| 49 | + | ||
| 50 | + def tearDown(self): | ||
| 51 | + self.conn.close() | ||
| 52 | + | ||
| 53 | + | ||
| 54 | +class PlanDocumentsTest(PipelineTestBase): | ||
| 55 | + def test_produces_four_artifacts_with_correct_lineage(self): | ||
| 56 | + self.conn.execute( | ||
| 57 | + "INSERT INTO task (task_id, project_id, goal_text, task_type, status, assigned_org_node) " | ||
| 58 | + "VALUES ('task-1', (SELECT project_id FROM project), '实现贪吃蛇游戏', 'goal', " | ||
| 59 | + "'planning_brd_prd_sdd', 'planner')" | ||
| 60 | + ) | ||
| 61 | + self.conn.execute( | ||
| 62 | + "INSERT INTO work_graph_run (run_id, task_id, org_node, graph_manifest, status) " | ||
| 63 | + "VALUES ('run-1', 'task-1', 'planner', '{}', 'running')" | ||
| 64 | + ) | ||
| 65 | + self.conn.commit() | ||
| 66 | + | ||
| 67 | + with patch("src.org_node_trigger.trigger_org_node", side_effect=_fake_trigger): | ||
| 68 | + artifact_ids = e2e_pipeline.plan_documents( | ||
| 69 | + self.conn, "task-1", "run-1", "实现贪吃蛇游戏", self.tmp.name | ||
| 70 | + ) | ||
| 71 | + | ||
| 72 | + self.assertEqual(set(artifact_ids), {"brd", "prd", "sdd", "benchmark"}) | ||
| 73 | + prd_deps = json.loads( | ||
| 74 | + self.conn.execute( | ||
| 75 | + "SELECT depends_on_artifact_ids FROM artifact WHERE artifact_id = ?", (artifact_ids["prd"],) | ||
| 76 | + ).fetchone()[0] | ||
| 77 | + ) | ||
| 78 | + self.assertEqual(prd_deps, [artifact_ids["brd"]]) | ||
| 79 | + sdd_deps = json.loads( | ||
| 80 | + self.conn.execute( | ||
| 81 | + "SELECT depends_on_artifact_ids FROM artifact WHERE artifact_id = ?", (artifact_ids["sdd"],) | ||
| 82 | + ).fetchone()[0] | ||
| 83 | + ) | ||
| 84 | + self.assertEqual(sdd_deps, [artifact_ids["prd"]]) | ||
| 85 | + | ||
| 86 | + | ||
| 87 | +class RunEndToEndTest(PipelineTestBase): | ||
| 88 | + def test_full_pipeline_wiring(self): | ||
| 89 | + with patch("src.org_node_lifecycle.trigger_org_node", side_effect=_fake_trigger), \ | ||
| 90 | + patch("src.org_node_trigger.trigger_org_node", side_effect=_fake_trigger): | ||
| 91 | + result = e2e_pipeline.run_end_to_end(self.conn, self.tmp.name, "实现贪吃蛇游戏") | ||
| 92 | + | ||
| 93 | + self.assertEqual(result["verdict"]["verdict"], "PASS") | ||
| 94 | + self.assertEqual(set(result["doc_artifact_ids"]), {"brd", "prd", "sdd", "benchmark"}) | ||
| 95 | + self.assertEqual(set(result["code_artifact_ids"]), {"code_diff", "test_report"}) | ||
| 96 | + | ||
| 97 | + task_row = self.conn.execute( | ||
| 98 | + "SELECT status FROM task WHERE task_id = ?", (result["task_id"],) | ||
| 99 | + ).fetchone() | ||
| 100 | + self.assertEqual(task_row[0], "validating") | ||
| 101 | + | ||
| 102 | + all_artifacts = self.conn.execute( | ||
| 103 | + "SELECT COUNT(*) FROM artifact WHERE task_id = ?", (result["task_id"],) | ||
| 104 | + ).fetchone()[0] | ||
| 105 | + self.assertEqual(all_artifacts, 6) # brd, prd, sdd, benchmark, code_diff, test_report | ||
| 106 | + | ||
| 107 | + | ||
| 108 | +class ValidateImplementationTest(unittest.TestCase): | ||
| 109 | + def test_parses_pass_verdict(self): | ||
| 110 | + with patch("src.org_node_trigger.trigger_org_node", return_value="VERDICT: PASS\nlooks good"): | ||
| 111 | + result = e2e_pipeline.validate_implementation("task-1", "benchmark content", "goal", "/tmp/p") | ||
| 112 | + self.assertEqual(result["verdict"], "PASS") | ||
| 113 | + | ||
| 114 | + def test_parses_fail_verdict(self): | ||
| 115 | + with patch("src.org_node_trigger.trigger_org_node", return_value="VERDICT: FAIL\nwall collision broken"): | ||
| 116 | + result = e2e_pipeline.validate_implementation("task-1", "benchmark content", "goal", "/tmp/p") | ||
| 117 | + self.assertEqual(result["verdict"], "FAIL") | ||
| 118 | + | ||
| 119 | + def test_no_verdict_line_defaults_to_inconclusive(self): | ||
| 120 | + with patch("src.org_node_trigger.trigger_org_node", return_value="I'm not sure what happened."): | ||
| 121 | + result = e2e_pipeline.validate_implementation("task-1", "benchmark content", "goal", "/tmp/p") | ||
| 122 | + self.assertEqual(result["verdict"], "INCONCLUSIVE") | ||
| 123 | + | ||
| 124 | + | ||
| 125 | +if __name__ == "__main__": | ||
| 126 | + unittest.main() |
src/test_e2e_pipeline_e2e.py
0 → 100644
| 1 | +"""U-014's real-call coverage, scoped deliberately: plan_documents (4 real | ||
| 2 | +Planner calls producing BRD/PRD/SDD/Benchmark for the snake-game scenario | ||
| 3 | +REQ-01~14 uses throughout the SDD) is cheap and fast, like every other real | ||
| 4 | +test this session (~$0.05-0.15/call). The Coder/Validator half | ||
| 5 | +(run_end_to_end in full) is a different order of magnitude -- real code | ||
| 6 | +generation + real test execution, plausibly minutes and real $ -- and isn't | ||
| 7 | +run automatically here; see the session notes for why. | ||
| 8 | + | ||
| 9 | +Run: RUN_REAL_AGENT_E2E=1 python3 -m unittest src.test_e2e_pipeline_e2e -v | ||
| 10 | +""" | ||
| 11 | + | ||
| 12 | +import json | ||
| 13 | +import subprocess | ||
| 14 | +import tempfile | ||
| 15 | +import unittest | ||
| 16 | + | ||
| 17 | +from src import agent_profile, e2e_pipeline, project_scaffold, schema, task_create | ||
| 18 | +from src.test_support import skip_unless_env | ||
| 19 | + | ||
| 20 | + | ||
| 21 | +@skip_unless_env("RUN_REAL_AGENT_E2E") | ||
| 22 | +class RealPlanDocumentsTest(unittest.TestCase): | ||
| 23 | + def test_planner_produces_four_real_lineaged_artifacts(self): | ||
| 24 | + with tempfile.TemporaryDirectory() as tmp: | ||
| 25 | + subprocess.run(["git", "init"], cwd=tmp, check=True, capture_output=True) | ||
| 26 | + conn = schema.connect(":memory:") | ||
| 27 | + schema.create_schema(conn) | ||
| 28 | + project_scaffold.create_project(conn, tmp, init_git_if_missing=False) | ||
| 29 | + agent_profile.deploy_agent_definitions(tmp) | ||
| 30 | + project_id = conn.execute("SELECT project_id FROM project").fetchone()[0] | ||
| 31 | + task_id = task_create.create_task(conn, project_id, "实现贪吃蛇游戏")["task_id"] | ||
| 32 | + conn.execute( | ||
| 33 | + "INSERT INTO work_graph_run (run_id, task_id, org_node, graph_manifest, status) " | ||
| 34 | + "VALUES ('run-1', ?, 'planner', '{}', 'running')", | ||
| 35 | + (task_id,), | ||
| 36 | + ) | ||
| 37 | + conn.commit() | ||
| 38 | + | ||
| 39 | + artifact_ids = e2e_pipeline.plan_documents(conn, task_id, "run-1", "实现贪吃蛇游戏", tmp) | ||
| 40 | + | ||
| 41 | + for artifact_type, artifact_id in artifact_ids.items(): | ||
| 42 | + row = conn.execute( | ||
| 43 | + "SELECT validation_status, file_path FROM artifact WHERE artifact_id = ?", (artifact_id,) | ||
| 44 | + ).fetchone() | ||
| 45 | + self.assertEqual(row[0], "passed") | ||
| 46 | + with open(tmp + "/" + row[1], encoding="utf-8") as f: | ||
| 47 | + content = f.read() | ||
| 48 | + self.assertTrue(len(content) > 100, "{} content suspiciously short: {!r}".format(artifact_type, content)) | ||
| 49 | + | ||
| 50 | + prd_deps = json.loads( | ||
| 51 | + conn.execute( | ||
| 52 | + "SELECT depends_on_artifact_ids FROM artifact WHERE artifact_id = ?", (artifact_ids["prd"],) | ||
| 53 | + ).fetchone()[0] | ||
| 54 | + ) | ||
| 55 | + self.assertEqual(prd_deps, [artifact_ids["brd"]]) | ||
| 56 | + | ||
| 57 | + | ||
| 58 | +if __name__ == "__main__": | ||
| 59 | + unittest.main() |