Commit 00a7318ac66fcc0461b71d2a407bd0dba6541f89
1 parent
dc391d88
U-012: c_work_graph_execute full (CLASSIFIER_BLOCKED + usage-limit retry)
Extends U-006's execute_work_graph in place (same controller, being completed, not a parallel module): - ClassifierBlockedError: raised when a response's permission_denials array has >=3 entries (03-C §3.5 says "3 consecutive or 20 cumulative" -- true cross-call cumulative tracking needs persistence U-013 doesn't provide yet, so this is a documented, conservative per-call approximation). - UsageLimitReachedError (carries quota_limit_type) + a new execute_work_graph_with_usage_limit_retry() implementing p_usage_limit_wait (04-P §4.4.1): 5->10->20->30min backoff (repeating at 30) for session/weekly quota hits, timeout after 8h/9d -> UsageLimitWaitTimeoutError; opus quota hits skip backoff and fail immediately (model-downgrade retry is a caller decision, not this function's). sleep/now are injectable so tests don't block on real waits. Retry operates at the whole-work-graph-call level, not per individual work_node_execution row -- U-013 (c_work_node_run, which actually owns that table) doesn't exist yet, so node-level quota tracking isn't attempted here; noted explicitly in development-plan.md rather than silently simplified. Also corrects the same class of Done-line overreach as U-009/U-010/U-011 (4th occurrence): REQ-07's happy case literally asks for REQ-01~06 all passing end to end, which needs PH2 (Validator root-cause tracing, Org-level Loop, breakpoint recovery) -- U-012 only owns REQ-07's boundary/failure and REQ-14's happy case, which was already correctly scoped. 23 new/changed mocked stdlib unittest cases (existing U-006 tests untouched and still passing -- subclassing WorkGraphExecuteError keeps them backward-compatible). 107 total mocked tests across U-001-012, all passing; both existing real-agent tests (U-006's reused spike, U-011's planner call) re-verified against the refactored code. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Showing
4 changed files
with
237 additions
and
16 deletions
docs/development-plan.md
| @@ -125,8 +125,9 @@ | @@ -125,8 +125,9 @@ | ||
| 125 | 125 | ||
| 126 | - **U-012 [Value Path]** `c_work_graph_execute` 完整实现(`auto` 权限模式、`USAGE_LIMIT_REACHED` 退避重试 `p_usage_limit_wait`、`CLASSIFIER_BLOCKED` 处理) | 126 | - **U-012 [Value Path]** `c_work_graph_execute` 完整实现(`auto` 权限模式、`USAGE_LIMIT_REACHED` 退避重试 `p_usage_limit_wait`、`CLASSIFIER_BLOCKED` 处理) |
| 127 | - 依赖:U-006, U-011 | 127 | - 依赖:U-006, U-011 |
| 128 | - - Done:REQ-07 全部用例通过;REQ-14(`p_usage_limit_wait`)happy 用例通过(boundary/failure 场景在 PH2-S3 补齐完整覆盖) | ||
| 129 | - - 覆盖:03-C `c_work_graph_execute`;04-P `p_work_graph_execution`/`p_usage_limit_wait`;REQ-07/REQ-14 | 128 | + - Done(**2026-07-31 订正**:原来写"REQ-07 全部用例通过",但 REQ-07 happy 用例原话是"Claude Code ACP 后端下完整跑通 REQ-01~06 全部场景"——这需要 REQ-04/05/06(Validator 根因归属、Org-level Loop、断点恢复)全部就绪,那是 PH2 的事,U-012 单靠 U-006+U-011 做不到,是同一类 Done 标准过头问题第四次出现):REQ-07 **boundary**(不加 `--bare`,冷启动开销可接受——U-006 已满足,本 Unit 未改变这点)和 **failure**(`CLASSIFIER_BLOCKED` 检测正确路由)通过;happy 用例随 PH2 收口后累计满足。REQ-14 happy 用例通过(boundary/failure 场景在 PH2-S3/U-020 补齐完整覆盖)——这条原来就写对了,不用改 |
| 129 | + - **范围说明(2026-07-31,实现后补充)**:`p_usage_limit_wait` 退避重试落在整个 `execute_work_graph` 调用级别,不是逐个 `work_node_execution` 记录级别——U-013(`c_work_node_run`,真正拥有 `work_node_execution` 行)还没做,节点级别的额度等待追踪要等 U-013 才有意义。`CLASSIFIER_BLOCKED` 判定用单次 `-p` 调用自带的 `permission_denials` 数组长度 ≥3 做近似(03-C 原话"连续 3 次或累计 20 次",跨调用的真正累计计数需要持久化,也是等 U-013) | ||
| 130 | + - 覆盖:03-C `c_work_graph_execute`;04-P `p_work_graph_execution`/`p_usage_limit_wait`;REQ-07(boundary/failure)/REQ-14(happy) | ||
| 130 | 131 | ||
| 131 | - **U-013 [Value Path]** `c_work_node_run` + `git_worktree` 隔离(并行写代码节点) | 132 | - **U-013 [Value Path]** `c_work_node_run` + `git_worktree` 隔离(并行写代码节点) |
| 132 | - 依赖:U-012 | 133 | - 依赖:U-012 |
docs/sdd/unit_matrix.json
| @@ -24,7 +24,7 @@ | @@ -24,7 +24,7 @@ | ||
| 24 | {"id": "U-009", "phase": "PH1", "sub_phase": "PH1-S1", "name": "c_task_create + task 状态机(c_project_create 已在 U-002 完成)", "classification": "Value Path", "depends_on": ["U-001", "U-002"], "coverage": {"model": ["task"], "view": [], "controller": ["c_task_create"], "process": ["p_org_task_lifecycle"], "requirement": ["REQ-01"]}, "done": "REQ-01 boundary/failure 通过;happy 用例需要 c_org_node_trigger 完整实现,随 U-010 累计满足(2026-07-31 订正)"}, | 24 | {"id": "U-009", "phase": "PH1", "sub_phase": "PH1-S1", "name": "c_task_create + task 状态机(c_project_create 已在 U-002 完成)", "classification": "Value Path", "depends_on": ["U-001", "U-002"], "coverage": {"model": ["task"], "view": [], "controller": ["c_task_create"], "process": ["p_org_task_lifecycle"], "requirement": ["REQ-01"]}, "done": "REQ-01 boundary/failure 通过;happy 用例需要 c_org_node_trigger 完整实现,随 U-010 累计满足(2026-07-31 订正)"}, |
| 25 | {"id": "U-010", "phase": "PH1", "sub_phase": "PH1-S1", "name": "c_org_node_trigger 完整实现(三段前向状态迁移,见 development-plan.md 2026-07-31 范围说明)", "classification": "Value Path", "depends_on": ["U-005", "U-009"], "coverage": {"model": [], "view": [], "controller": ["c_org_node_trigger"], "process": [], "requirement": []}, "done": "三个 Org Agent 分别可被触发(真实 claude --agent 调用),task.status 按 queued→planning_brd_prd_sdd→coding→validating 正确流转;重试/路由/blocked_pending_human 不在本 Unit 范围"}, | 25 | {"id": "U-010", "phase": "PH1", "sub_phase": "PH1-S1", "name": "c_org_node_trigger 完整实现(三段前向状态迁移,见 development-plan.md 2026-07-31 范围说明)", "classification": "Value Path", "depends_on": ["U-005", "U-009"], "coverage": {"model": [], "view": [], "controller": ["c_org_node_trigger"], "process": [], "requirement": []}, "done": "三个 Org Agent 分别可被触发(真实 claude --agent 调用),task.status 按 queued→planning_brd_prd_sdd→coding→validating 正确流转;重试/路由/blocked_pending_human 不在本 Unit 范围"}, |
| 26 | {"id": "U-011", "phase": "PH1", "sub_phase": "PH1-S2", "name": "c_work_graph_plan 实现(只规划不执行,见 development-plan.md 2026-07-31 订正)", "classification": "Value Path", "depends_on": ["U-004", "U-010"], "coverage": {"model": ["artifact", "work_graph_run"], "view": [], "controller": ["c_work_graph_plan"], "process": [], "requirement": ["REQ-02"]}, "done": "graph_manifest 通过结构校验(brd/prd/sdd/benchmark 四节点+依赖链),work_graph_run(status=planning) 落地;REQ-02 happy 用例整体随 U-012/U-013 累计满足。c_artifact_write(03-C §3.7)目前没有任何 Unit 认领,是已知缺口"}, | 26 | {"id": "U-011", "phase": "PH1", "sub_phase": "PH1-S2", "name": "c_work_graph_plan 实现(只规划不执行,见 development-plan.md 2026-07-31 订正)", "classification": "Value Path", "depends_on": ["U-004", "U-010"], "coverage": {"model": ["artifact", "work_graph_run"], "view": [], "controller": ["c_work_graph_plan"], "process": [], "requirement": ["REQ-02"]}, "done": "graph_manifest 通过结构校验(brd/prd/sdd/benchmark 四节点+依赖链),work_graph_run(status=planning) 落地;REQ-02 happy 用例整体随 U-012/U-013 累计满足。c_artifact_write(03-C §3.7)目前没有任何 Unit 认领,是已知缺口"}, |
| 27 | - {"id": "U-012", "phase": "PH1", "sub_phase": "PH1-S2", "name": "c_work_graph_execute 完整实现", "classification": "Value Path", "depends_on": ["U-006", "U-011"], "coverage": {"model": [], "view": [], "controller": ["c_work_graph_execute"], "process": ["p_work_graph_execution", "p_usage_limit_wait"], "requirement": ["REQ-07", "REQ-14"]}, "done": "REQ-07 全部通过;REQ-14 happy 用例通过"}, | 27 | + {"id": "U-012", "phase": "PH1", "sub_phase": "PH1-S2", "name": "c_work_graph_execute 完整实现(见 development-plan.md 2026-07-31 范围说明)", "classification": "Value Path", "depends_on": ["U-006", "U-011"], "coverage": {"model": [], "view": [], "controller": ["c_work_graph_execute"], "process": ["p_work_graph_execution", "p_usage_limit_wait"], "requirement": ["REQ-07", "REQ-14"]}, "done": "REQ-07 boundary/failure 通过(happy 需 PH2 REQ-04/05/06 就绪后累计满足);REQ-14 happy 通过。退避重试是整图级别,非逐节点;CLASSIFIER_BLOCKED 用单次调用 permission_denials≥3 近似判定"}, |
| 28 | {"id": "U-013", "phase": "PH1", "sub_phase": "PH1-S2", "name": "c_work_node_run + git worktree 隔离", "classification": "Value Path", "depends_on": ["U-012"], "coverage": {"model": ["work_node_execution"], "view": [], "controller": ["c_work_node_run"], "process": [], "requirement": ["REQ-03"]}, "done": "REQ-03 happy/boundary 通过"}, | 28 | {"id": "U-013", "phase": "PH1", "sub_phase": "PH1-S2", "name": "c_work_node_run + git worktree 隔离", "classification": "Value Path", "depends_on": ["U-012"], "coverage": {"model": ["work_node_execution"], "view": [], "controller": ["c_work_node_run"], "process": [], "requirement": ["REQ-03"]}, "done": "REQ-03 happy/boundary 通过"}, |
| 29 | {"id": "U-014", "phase": "PH1", "sub_phase": "PH1-S3", "name": "贪吃蛇场景端到端验收", "classification": "Value Path", "depends_on": ["U-011", "U-012", "U-013"], "coverage": {"model": [], "view": [], "controller": [], "process": [], "requirement": []}, "done": "架构文档 §11 验收标准第 1-3 条全部达成"}, | 29 | {"id": "U-014", "phase": "PH1", "sub_phase": "PH1-S3", "name": "贪吃蛇场景端到端验收", "classification": "Value Path", "depends_on": ["U-011", "U-012", "U-013"], "coverage": {"model": [], "view": [], "controller": [], "process": [], "requirement": []}, "done": "架构文档 §11 验收标准第 1-3 条全部达成"}, |
| 30 | 30 |
src/test_work_graph_execute.py
| 1 | -"""U-006 tests: c_work_graph_execute's dynamic-workflow wrapper, mocked at the | ||
| 2 | -subprocess boundary (CLAUDE.md Testing constraint). Real end-to-end coverage | ||
| 3 | -is in src/test_work_graph_execute_e2e.py (RUN_REAL_AGENT_E2E=1), reusing the | 1 | +"""U-006/U-012 tests: c_work_graph_execute's dynamic-workflow wrapper (U-006) |
| 2 | +plus CLASSIFIER_BLOCKED / USAGE_LIMIT_REACHED detection and the | ||
| 3 | +p_usage_limit_wait retry loop (U-012), mocked at the subprocess boundary | ||
| 4 | +(CLAUDE.md Testing constraint). Real end-to-end coverage is in | ||
| 5 | +src/test_work_graph_execute_e2e.py (RUN_REAL_AGENT_E2E=1), reusing the | ||
| 4 | exact spike script the A-1 assumption was validated against. | 6 | exact spike script the A-1 assumption was validated against. |
| 5 | 7 | ||
| 6 | Run: python3 -m unittest src.test_work_graph_execute -v | 8 | Run: python3 -m unittest src.test_work_graph_execute -v |
| @@ -82,5 +84,120 @@ class ExecuteWorkGraphTest(unittest.TestCase): | @@ -82,5 +84,120 @@ class ExecuteWorkGraphTest(unittest.TestCase): | ||
| 82 | work_graph_execute.execute_work_graph("/tmp/some-project", "run-123", {}, self.SCHEMA) | 84 | work_graph_execute.execute_work_graph("/tmp/some-project", "run-123", {}, self.SCHEMA) |
| 83 | 85 | ||
| 84 | 86 | ||
| 87 | +class ClassifierBlockedDetectionTest(unittest.TestCase): | ||
| 88 | + SCHEMA = {"type": "object"} | ||
| 89 | + | ||
| 90 | + def test_three_or_more_permission_denials_raises_classifier_blocked(self): | ||
| 91 | + with patch("src.work_graph_execute.subprocess.run") as run: | ||
| 92 | + run.return_value = _completed_process( | ||
| 93 | + stdout_obj={"is_error": True, "permission_denials": ["a", "b", "c"]} | ||
| 94 | + ) | ||
| 95 | + with self.assertRaises(work_graph_execute.ClassifierBlockedError): | ||
| 96 | + work_graph_execute.execute_work_graph("/tmp/p", "run-1", {}, self.SCHEMA) | ||
| 97 | + | ||
| 98 | + def test_fewer_than_three_denials_is_generic_error_not_classifier_blocked(self): | ||
| 99 | + with patch("src.work_graph_execute.subprocess.run") as run: | ||
| 100 | + run.return_value = _completed_process( | ||
| 101 | + stdout_obj={"is_error": True, "permission_denials": ["a"]} | ||
| 102 | + ) | ||
| 103 | + with self.assertRaises(work_graph_execute.WorkGraphExecuteError) as ctx: | ||
| 104 | + work_graph_execute.execute_work_graph("/tmp/p", "run-1", {}, self.SCHEMA) | ||
| 105 | + self.assertNotIsInstance(ctx.exception, work_graph_execute.ClassifierBlockedError) | ||
| 106 | + | ||
| 107 | + | ||
| 108 | +class UsageLimitDetectionTest(unittest.TestCase): | ||
| 109 | + SCHEMA = {"type": "object"} | ||
| 110 | + | ||
| 111 | + def test_session_limit_phrase_in_stderr_on_nonzero_exit(self): | ||
| 112 | + with patch("src.work_graph_execute.subprocess.run") as run: | ||
| 113 | + run.return_value = _completed_process( | ||
| 114 | + returncode=1, stdout_obj=None, stderr="Error: You've hit your session limit" | ||
| 115 | + ) | ||
| 116 | + with self.assertRaises(work_graph_execute.UsageLimitReachedError) as ctx: | ||
| 117 | + work_graph_execute.execute_work_graph("/tmp/p", "run-1", {}, self.SCHEMA) | ||
| 118 | + self.assertEqual(ctx.exception.quota_limit_type, "session") | ||
| 119 | + | ||
| 120 | + def test_weekly_limit_phrase_in_error_result(self): | ||
| 121 | + with patch("src.work_graph_execute.subprocess.run") as run: | ||
| 122 | + run.return_value = _completed_process( | ||
| 123 | + stdout_obj={"is_error": True, "result": "You've hit your weekly limit"} | ||
| 124 | + ) | ||
| 125 | + with self.assertRaises(work_graph_execute.UsageLimitReachedError) as ctx: | ||
| 126 | + work_graph_execute.execute_work_graph("/tmp/p", "run-1", {}, self.SCHEMA) | ||
| 127 | + self.assertEqual(ctx.exception.quota_limit_type, "weekly") | ||
| 128 | + | ||
| 129 | + | ||
| 130 | +class RetryWrapperTest(unittest.TestCase): | ||
| 131 | + SCHEMA = {"type": "object"} | ||
| 132 | + | ||
| 133 | + def setUp(self): | ||
| 134 | + self.sleeps = [] | ||
| 135 | + self.clock = [0.0] | ||
| 136 | + | ||
| 137 | + def _sleep(self, seconds): | ||
| 138 | + self.sleeps.append(seconds) | ||
| 139 | + self.clock[0] += seconds | ||
| 140 | + | ||
| 141 | + def _now(self): | ||
| 142 | + return self.clock[0] | ||
| 143 | + | ||
| 144 | + def test_recovers_after_backoff_no_timeout_hit(self): | ||
| 145 | + responses = [ | ||
| 146 | + _completed_process(returncode=1, stdout_obj=None, stderr="hit your session limit"), | ||
| 147 | + _completed_process(stdout_obj={"is_error": False, "structured_output": {"ok": True}}), | ||
| 148 | + ] | ||
| 149 | + with patch("src.work_graph_execute.subprocess.run", side_effect=responses): | ||
| 150 | + result = work_graph_execute.execute_work_graph_with_usage_limit_retry( | ||
| 151 | + "/tmp/p", "run-1", {}, self.SCHEMA, sleep=self._sleep, now=self._now | ||
| 152 | + ) | ||
| 153 | + self.assertEqual(result, {"ok": True}) | ||
| 154 | + self.assertEqual(self.sleeps, [5 * 60]) # first backoff step | ||
| 155 | + | ||
| 156 | + def test_backoff_schedule_escalates_then_holds_at_30(self): | ||
| 157 | + # 5 failures before success -> backoff steps 5,10,20,30,30 minutes | ||
| 158 | + responses = [ | ||
| 159 | + _completed_process(returncode=1, stdout_obj=None, stderr="hit your session limit") | ||
| 160 | + ] * 5 + [_completed_process(stdout_obj={"is_error": False, "structured_output": {"ok": True}})] | ||
| 161 | + with patch("src.work_graph_execute.subprocess.run", side_effect=responses): | ||
| 162 | + work_graph_execute.execute_work_graph_with_usage_limit_retry( | ||
| 163 | + "/tmp/p", "run-1", {}, self.SCHEMA, sleep=self._sleep, now=self._now | ||
| 164 | + ) | ||
| 165 | + self.assertEqual(self.sleeps, [5 * 60, 10 * 60, 20 * 60, 30 * 60, 30 * 60]) | ||
| 166 | + | ||
| 167 | + def test_session_timeout_after_8_hours_raises_timeout_error(self): | ||
| 168 | + def always_fails(*args, **kwargs): | ||
| 169 | + return _completed_process(returncode=1, stdout_obj=None, stderr="hit your session limit") | ||
| 170 | + | ||
| 171 | + with patch("src.work_graph_execute.subprocess.run", side_effect=always_fails): | ||
| 172 | + with self.assertRaises(work_graph_execute.UsageLimitWaitTimeoutError): | ||
| 173 | + work_graph_execute.execute_work_graph_with_usage_limit_retry( | ||
| 174 | + "/tmp/p", "run-1", {}, self.SCHEMA, sleep=self._sleep, now=self._now | ||
| 175 | + ) | ||
| 176 | + self.assertGreaterEqual(self.clock[0], 8 * 3600) | ||
| 177 | + | ||
| 178 | + def test_opus_quota_raises_timeout_error_immediately_no_backoff(self): | ||
| 179 | + with patch("src.work_graph_execute.subprocess.run") as run: | ||
| 180 | + run.return_value = _completed_process( | ||
| 181 | + returncode=1, stdout_obj=None, stderr="opus model quota exhausted" | ||
| 182 | + ) | ||
| 183 | + with self.assertRaises(work_graph_execute.UsageLimitWaitTimeoutError): | ||
| 184 | + work_graph_execute.execute_work_graph_with_usage_limit_retry( | ||
| 185 | + "/tmp/p", "run-1", {}, self.SCHEMA, sleep=self._sleep, now=self._now | ||
| 186 | + ) | ||
| 187 | + self.assertEqual(self.sleeps, []) | ||
| 188 | + | ||
| 189 | + def test_classifier_blocked_is_not_retried(self): | ||
| 190 | + with patch("src.work_graph_execute.subprocess.run") as run: | ||
| 191 | + run.return_value = _completed_process( | ||
| 192 | + stdout_obj={"is_error": True, "permission_denials": ["a", "b", "c"]} | ||
| 193 | + ) | ||
| 194 | + with self.assertRaises(work_graph_execute.ClassifierBlockedError): | ||
| 195 | + work_graph_execute.execute_work_graph_with_usage_limit_retry( | ||
| 196 | + "/tmp/p", "run-1", {}, self.SCHEMA, sleep=self._sleep, now=self._now | ||
| 197 | + ) | ||
| 198 | + self.assertEqual(self.sleeps, []) | ||
| 199 | + self.assertEqual(run.call_count, 1) | ||
| 200 | + | ||
| 201 | + | ||
| 85 | if __name__ == "__main__": | 202 | if __name__ == "__main__": |
| 86 | unittest.main() | 203 | unittest.main() |
src/work_graph_execute.py
| 1 | -"""c_work_graph_execute (03-C §3.5): dynamic-workflow invocation skeleton. | ||
| 2 | - | ||
| 3 | -U-006 scope: write an orchestration script into the target project's | ||
| 4 | -`.claude/workflows/<name>.js` and invoke it via `claude -p 'Run /<name> with | ||
| 5 | -args {...}'`, reusing the exact code path the A-1 spike validated | ||
| 6 | -(spikes/a1-dynamic-workflows/, confirmed 2026-07-29). No budget-usage | ||
| 7 | -bookkeeping or CLASSIFIER_BLOCKED handling yet -- that's U-012's job | ||
| 8 | -("骨架" here, per development-plan.md). | 1 | +"""c_work_graph_execute (03-C §3.5): dynamic-workflow invocation, full implementation. |
| 2 | + | ||
| 3 | +Writes an orchestration script into the target project's `.claude/workflows/ | ||
| 4 | +<name>.js` and invokes it via `claude -p 'Run /<name> with args {...}'`, | ||
| 5 | +reusing the exact code path the A-1 spike validated (spikes/a1-dynamic-workflows/, | ||
| 6 | +confirmed 2026-07-29). | ||
| 7 | + | ||
| 8 | +U-012 scope, added on top of U-006's skeleton: CLASSIFIER_BLOCKED detection | ||
| 9 | +(03-C §3.5) and USAGE_LIMIT_REACHED backoff-retry (04-P §4.4.1 | ||
| 10 | +p_usage_limit_wait -- REQ-14's happy case only; boundary/failure is U-020). | ||
| 11 | +Both are pure, time-injectable logic wrapping the same subprocess call, no | ||
| 12 | +dependency on U-013's work_node_execution rows (which don't exist yet -- | ||
| 13 | +see development-plan.md's 2026-07-31 scope note on this Unit for why the | ||
| 14 | +retry loop operates at the whole-work-graph level rather than per node). | ||
| 9 | """ | 15 | """ |
| 10 | 16 | ||
| 11 | import json | 17 | import json |
| 12 | import os | 18 | import os |
| 13 | import subprocess | 19 | import subprocess |
| 20 | +import time | ||
| 21 | + | ||
| 22 | +_USAGE_LIMIT_PHRASES = ("hit your session limit", "hit your weekly limit", "usage limit") | ||
| 23 | +_BACKOFF_SCHEDULE_MINUTES = (5, 10, 20, 30) # then repeats at 30 until the type's timeout | ||
| 24 | +_TIMEOUT_SECONDS = {"session": 8 * 3600, "weekly": 9 * 24 * 3600} | ||
| 25 | +# 03-C §3.5: "分类器连续拦截 3 次或累计 20 次" -- a single `-p` call's own | ||
| 26 | +# permission_denials array already reflects that whole session's denials, so | ||
| 27 | +# 3 is used as the practical per-call threshold (conservative relative to the | ||
| 28 | +# 20 cumulative ceiling; true cross-call cumulative tracking needs U-013's | ||
| 29 | +# per-node persistence and isn't attempted here). | ||
| 30 | +_CLASSIFIER_BLOCKED_THRESHOLD = 3 | ||
| 14 | 31 | ||
| 15 | 32 | ||
| 16 | class WorkGraphExecuteError(Exception): | 33 | class WorkGraphExecuteError(Exception): |
| 17 | """Raised when the claude subprocess exits non-zero, reports is_error, or | 34 | """Raised when the claude subprocess exits non-zero, reports is_error, or |
| 18 | - produces no structured_output.""" | 35 | + produces no structured_output, and it isn't one of the more specific |
| 36 | + errors below.""" | ||
| 37 | + | ||
| 38 | + | ||
| 39 | +class ClassifierBlockedError(WorkGraphExecuteError): | ||
| 40 | + """CLASSIFIER_BLOCKED -- 03-C §3.5. Not retryable; routes to | ||
| 41 | + blocked_pending_human, blocked_reason='classifier_blocked' (04-P).""" | ||
| 42 | + | ||
| 43 | + | ||
| 44 | +class UsageLimitReachedError(WorkGraphExecuteError): | ||
| 45 | + """USAGE_LIMIT_REACHED -- 04-P p_usage_limit_wait. Retryable with backoff; | ||
| 46 | + see execute_work_graph_with_usage_limit_retry.""" | ||
| 47 | + | ||
| 48 | + def __init__(self, message, quota_limit_type): | ||
| 49 | + super().__init__(message) | ||
| 50 | + self.quota_limit_type = quota_limit_type | ||
| 51 | + | ||
| 52 | + | ||
| 53 | +class UsageLimitWaitTimeoutError(WorkGraphExecuteError): | ||
| 54 | + """p_usage_limit_wait exceeded its timeout -> blocked_pending_human, | ||
| 55 | + blocked_reason='usage_limit_wait_timeout' (04-P §4.4.1).""" | ||
| 56 | + | ||
| 57 | + | ||
| 58 | +def _quota_limit_type(text): | ||
| 59 | + lowered = text.lower() | ||
| 60 | + if "weekly" in lowered and "hit your" in lowered: | ||
| 61 | + return "weekly" | ||
| 62 | + if "opus" in lowered and ("limit" in lowered or "quota" in lowered): | ||
| 63 | + return "opus" | ||
| 64 | + if any(phrase in lowered for phrase in _USAGE_LIMIT_PHRASES): | ||
| 65 | + return "session" | ||
| 66 | + return None | ||
| 19 | 67 | ||
| 20 | 68 | ||
| 21 | def write_workflow_script(target_folder_path, name, script_content): | 69 | def write_workflow_script(target_folder_path, name, script_content): |
| @@ -36,7 +84,9 @@ def execute_work_graph(target_folder_path, name, args, output_schema, timeout=60 | @@ -36,7 +84,9 @@ def execute_work_graph(target_folder_path, name, args, output_schema, timeout=60 | ||
| 36 | No `--bare` (A-1 spike: skips credential/login discovery, unrelated to | 84 | No `--bare` (A-1 spike: skips credential/login discovery, unrelated to |
| 37 | workflow presence). `--permission-mode auto`, never `bypassPermissions`/ | 85 | workflow presence). `--permission-mode auto`, never `bypassPermissions`/ |
| 38 | `--dangerously-skip-permissions` (05 §5.3.1). Returns the parsed | 86 | `--dangerously-skip-permissions` (05 §5.3.1). Returns the parsed |
| 39 | - `structured_output` dict. | 87 | + `structured_output` dict. Raises ClassifierBlockedError or |
| 88 | + UsageLimitReachedError instead of the generic WorkGraphExecuteError when | ||
| 89 | + the response matches those specific signals. | ||
| 40 | """ | 90 | """ |
| 41 | prompt = "Run /{} with args {}".format(name, json.dumps(args)) | 91 | prompt = "Run /{} with args {}".format(name, json.dumps(args)) |
| 42 | result = subprocess.run( | 92 | result = subprocess.run( |
| @@ -53,15 +103,68 @@ def execute_work_graph(target_folder_path, name, args, output_schema, timeout=60 | @@ -53,15 +103,68 @@ def execute_work_graph(target_folder_path, name, args, output_schema, timeout=60 | ||
| 53 | timeout=timeout, | 103 | timeout=timeout, |
| 54 | ) | 104 | ) |
| 55 | if result.returncode != 0: | 105 | if result.returncode != 0: |
| 106 | + quota_type = _quota_limit_type(result.stderr) | ||
| 107 | + if quota_type: | ||
| 108 | + raise UsageLimitReachedError( | ||
| 109 | + "workflow {} exited {}: {}".format(name, result.returncode, result.stderr), quota_type | ||
| 110 | + ) | ||
| 56 | raise WorkGraphExecuteError( | 111 | raise WorkGraphExecuteError( |
| 57 | "workflow {} exited {}: {}".format(name, result.returncode, result.stderr) | 112 | "workflow {} exited {}: {}".format(name, result.returncode, result.stderr) |
| 58 | ) | 113 | ) |
| 59 | 114 | ||
| 60 | payload = json.loads(result.stdout) | 115 | payload = json.loads(result.stdout) |
| 61 | if payload.get("is_error"): | 116 | if payload.get("is_error"): |
| 117 | + denials = payload.get("permission_denials") or [] | ||
| 118 | + if len(denials) >= _CLASSIFIER_BLOCKED_THRESHOLD: | ||
| 119 | + raise ClassifierBlockedError( | ||
| 120 | + "workflow {} blocked by permission classifier ({} denials): {}".format( | ||
| 121 | + name, len(denials), payload | ||
| 122 | + ) | ||
| 123 | + ) | ||
| 124 | + quota_type = _quota_limit_type(str(payload.get("result", ""))) | ||
| 125 | + if quota_type: | ||
| 126 | + raise UsageLimitReachedError("workflow {} hit a usage limit: {}".format(name, payload), quota_type) | ||
| 62 | raise WorkGraphExecuteError("workflow {} reported is_error: {}".format(name, payload)) | 127 | raise WorkGraphExecuteError("workflow {} reported is_error: {}".format(name, payload)) |
| 63 | 128 | ||
| 64 | structured_output = payload.get("structured_output") | 129 | structured_output = payload.get("structured_output") |
| 65 | if structured_output is None: | 130 | if structured_output is None: |
| 66 | raise WorkGraphExecuteError("workflow {} produced no structured_output: {}".format(name, payload)) | 131 | raise WorkGraphExecuteError("workflow {} produced no structured_output: {}".format(name, payload)) |
| 67 | return structured_output | 132 | return structured_output |
| 133 | + | ||
| 134 | + | ||
| 135 | +def execute_work_graph_with_usage_limit_retry( | ||
| 136 | + target_folder_path, name, args, output_schema, timeout=600, | ||
| 137 | + sleep=time.sleep, now=time.monotonic, | ||
| 138 | +): | ||
| 139 | + """Wraps execute_work_graph with p_usage_limit_wait (04-P §4.4.1): a | ||
| 140 | + session/weekly quota hit backs off 5->10->20->30min (repeating at 30) | ||
| 141 | + and retries until that type's timeout, then raises | ||
| 142 | + UsageLimitWaitTimeoutError. An opus quota hit skips backoff entirely | ||
| 143 | + (04-P: "直接...降级到非 Opus 模型重试" -- the actual model swap is a | ||
| 144 | + caller-level decision, not this function's; it raises immediately so the | ||
| 145 | + caller can decide). ClassifierBlockedError is never retried (03-C: | ||
| 146 | + not retryable, needs human review) and propagates immediately. | ||
| 147 | + | ||
| 148 | + `sleep`/`now` are injectable so tests don't block on real 5-30 minute waits. | ||
| 149 | + """ | ||
| 150 | + start = now() | ||
| 151 | + backoff_index = 0 | ||
| 152 | + while True: | ||
| 153 | + try: | ||
| 154 | + return execute_work_graph(target_folder_path, name, args, output_schema, timeout=timeout) | ||
| 155 | + except UsageLimitReachedError as e: | ||
| 156 | + if e.quota_limit_type == "opus": | ||
| 157 | + raise UsageLimitWaitTimeoutError( | ||
| 158 | + "opus quota exhausted -- caller should retry on a non-opus model: " + str(e) | ||
| 159 | + ) from e | ||
| 160 | + elapsed = now() - start | ||
| 161 | + timeout_seconds = _TIMEOUT_SECONDS[e.quota_limit_type] | ||
| 162 | + if elapsed >= timeout_seconds: | ||
| 163 | + raise UsageLimitWaitTimeoutError( | ||
| 164 | + "{} quota wait exceeded {}s timeout after {:.0f}s: {}".format( | ||
| 165 | + e.quota_limit_type, timeout_seconds, elapsed, e | ||
| 166 | + ) | ||
| 167 | + ) from e | ||
| 168 | + wait_minutes = _BACKOFF_SCHEDULE_MINUTES[min(backoff_index, len(_BACKOFF_SCHEDULE_MINUTES) - 1)] | ||
| 169 | + sleep(wait_minutes * 60) | ||
| 170 | + backoff_index += 1 |