/** * Goal Contract — 任务级验收契约的格式、校验、执行与判决持久化(P3 阶段 0 / T0) * * 设计见 docs/todo/system-design/27-goal-contract-format.md, * 需求见 27-p3-p4-work-graph-console.md §五 阶段 0 / §四.A.5 / §八.9, * 以及 27-e2e-task-execution-flow.md §十一(三个验收产出节点)。 * * 形态直接借 DevLoop 已跑通的 `integration/spec.json`(_spec/sdd-model.md M-6) * 与 `runs/result.json`(M-7/M-8):契约是一份 JSON、判决是一份 JSON, * 都放在约定好的固定位置,谁都不用猜。 * * §八.9 划的优先级:**外层(存哪、怎么被引用、怎么被执行、判决存哪)要定死, * 内层断言字段允许后续演进**。所以本文件的稳定面是: * * data/companies/{cid}/runs/{rootRunId}/ * ├── artifacts/ * │ ├── acceptance.json ← 唯一契约入口(三 section,各自可为 null) * │ ├── goal-benchmark.json ← 大块内容外置,契约里用 {"$ref": "..."} 指过来 * │ ├── goal-test-data/ * │ ├── goal-test-script/ * │ └── verdicts/ * │ ├── {verdictId}.json ← 一次执行一份完整判决 * │ └── index.ndjson ← append-only 索引,可查、可追溯到 run/node/attempt * └── nodes/{nodeRunId}/ ← output_ref 指向这里,即「被测产物」 * * 与 output_ref 的关联方式:执行器的入参 targetPath 就是某个 graph_run_nodes.output_ref * 解析出的绝对路径;判决里回存 run_id / node_run_id / attempt / contract_hash 四元组, * 所以任何一份判决都能双向追溯到「哪次 run 的哪个节点的第几次尝试、用哪版契约判的」。 * * 不起 LLM:benchmark / test_data 是纯声明式比对,test_script 走真实子进程。 * * ── 生产调用方(R5-F3)───────────────────────────────────────────── * 契约的产出方和消费方都是 **agent**,不是引擎——见 27-goal-contract-format.md §二bis * 「为什么接在 agent 工具层而不是引擎钩子」。所以本文件除了作为模块被 require, * 还带一个 CLI 入口(文件末尾),供 agent 在 bash 里直接调: * * node server/services/goal-contract.js put --run-dir R --goal-id G --section benchmark --file X.json * node server/services/goal-contract.js validate --run-dir R * node server/services/goal-contract.js run --run-dir R --target T * node server/services/goal-contract.js list|show --run-dir R [...] * * 走 CLI 而不是让 agent 自己写 require 脚本,是因为 agent 侧唯一稳定的执行面就是 bash * (模板 tools.md 全是 bash 片段),且 `put --file` 让「绝不在 bash 里内联 JSON」这条 * 铁律得以遵守。 */ const fs = require('fs'); const path = require('path'); const crypto = require('crypto'); const { spawn } = require('child_process'); // contract_hash 直接复用 DevLoop 的实现(只读借用,不改它一行;见 §七「明确不改」) const { hashContract } = require('./devloop/doc-store'); const DATA_DIR = path.resolve(__dirname, '../../data'); const CONTRACT_VERSION = '1'; const CONTRACT_FILE = 'acceptance.json'; const SECTIONS = ['benchmark', 'test_data', 'test_script']; // ponytail: v1 只支持 runner='node'(脚本退出码即判决),够表达一切断言。 // 上限:跑不了 vitest——实测仓库根 vitest.config 的 include 不覆盖 data/, // `vitest run <脚本>` / `--root` / `--dir` 都会被根 config 的 include 盖掉,直接 "No test files found"。 // 升级路径:真需要 vitest 时,为验收脚本生成一份临时 vitest.config 并用 `--config` 指过去, // 再把 'vitest' 加回这张白名单——契约的 runner 字段就是为此留的扩展位。 // 眼下 node 脚本里自己 spawn 任何测试框架也不受限制,所以不急。 const RUNNERS = ['node']; const MATCH_MODES = ['deep-equal', 'subset']; const DEFAULT_SCRIPT_TIMEOUT_MS = 120_000; const OUTPUT_CAP = 64 * 1024; // stdout/stderr 各留 64KB,超出截断 class ContractValidationError extends Error { constructor(errors) { super('goal contract validation failed'); this.name = 'ContractValidationError'; this.errors = errors; } } // ──────────────────────────────────────────────────────────────── // 路径 // ──────────────────────────────────────────────────────────────── /** root run 的产物根目录:data/companies/{cid}/runs/{rootRunId}/ */ function resolveRunDir(companyId, rootRunId) { return path.join(DATA_DIR, 'companies', String(companyId), 'runs', String(rootRunId)); } /** 任务级共享产物区(跨 org node 只读,写入是特权动作,见 §四.A.5) */ function resolveArtifactsDir(runDir) { return path.join(runDir, 'artifacts'); } function resolveContractPath(runDir) { return path.join(resolveArtifactsDir(runDir), CONTRACT_FILE); } function resolveVerdictsDir(runDir) { return path.join(resolveArtifactsDir(runDir), 'verdicts'); } /** 契约里的相对路径一律相对 artifacts/,且不许逃出 artifacts/(防契约指向任意可执行文件) */ function resolveInsideArtifacts(artifactsDir, rel) { const abs = path.resolve(artifactsDir, rel); const base = path.resolve(artifactsDir); if (abs !== base && !abs.startsWith(base + path.sep)) return null; return abs; } // ──────────────────────────────────────────────────────────────── // 小工具 // ──────────────────────────────────────────────────────────────── function now() { return new Date().toISOString(); } /** CONVENTIONS §2 的 ID 形态:vd-{YYYYMMDD-HHMM}-{4hex} */ function genVerdictId(d = new Date()) { const p2 = (n) => String(n).padStart(2, '0'); const stamp = `${d.getFullYear()}${p2(d.getMonth() + 1)}${p2(d.getDate())}` + `-${p2(d.getHours())}${p2(d.getMinutes())}`; return `vd-${stamp}-${crypto.randomBytes(2).toString('hex')}`; } /** key 排序的规范序列化,用于 deep-equal 比较 */ function stable(v) { if (v === null || typeof v !== 'object') return JSON.stringify(v) ?? 'null'; if (Array.isArray(v)) return '[' + v.map(stable).join(',') + ']'; return ( '{' + Object.keys(v) .sort() .map((k) => JSON.stringify(k) + ':' + stable(v[k])) .join(',') + '}' ); } function deepEqual(a, b) { return stable(a) === stable(b); } /** subset:expected 里出现的键/元素必须匹配,actual 允许有额外键 */ function matchSubset(expected, actual) { if (expected === null || typeof expected !== 'object') return expected === actual; if (Array.isArray(expected)) { if (!Array.isArray(actual) || actual.length !== expected.length) return false; return expected.every((e, i) => matchSubset(e, actual[i])); } if (actual === null || typeof actual !== 'object' || Array.isArray(actual)) return false; return Object.keys(expected).every((k) => matchSubset(expected[k], actual[k])); } function readJsonFile(abs) { try { return { ok: true, json: JSON.parse(fs.readFileSync(abs, 'utf8')) }; } catch (e) { return { ok: false, error: e.message }; } } /** CONVENTIONS §5:JSON 落盘走 tmp + rename */ function writeJsonAtomic(abs, json) { fs.mkdirSync(path.dirname(abs), { recursive: true }); const tmp = abs + '.tmp'; fs.writeFileSync(tmp, JSON.stringify(json, null, 2)); fs.renameSync(tmp, abs); return abs; } // ──────────────────────────────────────────────────────────────── // 契约:读 / 写 / 校验 // ──────────────────────────────────────────────────────────────── /** * 校验契约本身是否合法(不碰被测产物)。 * * errors 形态与 org-graph-service.validateGraph 保持一致:{rule, detail}。 * rule 取值:schema | section | reference | runner * * @param {object} contract * @returns {{ok: boolean, errors: Array<{rule: string, detail: string}>}} */ function validateContract(contract) { const errors = []; const err = (rule, detail) => errors.push({ rule, detail }); if (!contract || typeof contract !== 'object' || Array.isArray(contract)) { return { ok: false, errors: [{ rule: 'schema', detail: 'contract must be a plain object' }] }; } if (contract.version !== CONTRACT_VERSION) { err('schema', `version must be "${CONTRACT_VERSION}", got ${JSON.stringify(contract.version)}`); } if (!contract.goal_id || typeof contract.goal_id !== 'string') { err('schema', 'missing field: goal_id'); } for (const k of Object.keys(contract)) { if (!['version', 'goal_id', 'created_by', 'created_at', ...SECTIONS].includes(k)) { err('schema', `unknown top-level field: ${k}`); } } // created_by 两种形态都合法:单写者写字符串;三个产出节点各写一段时是 {section: producer} 映射 // (§一.1「三个产出节点各写自己那一段」——一个字符串存不下三个作者)。审计字段,不参与判定。 if (contract.created_by !== undefined && contract.created_by !== null) { const cb = contract.created_by; const okShape = typeof cb === 'string' || (typeof cb === 'object' && !Array.isArray(cb) && Object.values(cb).every((v) => typeof v === 'string')); if (!okShape) err('schema', 'created_by must be a string or a {section: producer} object'); } // 至少一个 section 非 null。空契约会「平凡通过」——正是 §四.D.3 要求 planner-validator // 盯的那类漏洞,在格式层面直接堵掉。 const present = SECTIONS.filter((s) => contract[s] !== undefined && contract[s] !== null); if (present.length === 0) { err('section', 'at least one of benchmark / test_data / test_script must be non-null'); } for (const s of present) { const sec = contract[s]; if (typeof sec !== 'object' || Array.isArray(sec)) { err('section', `${s} must be an object or null`); continue; } // 未解引用的 $ref 只校验形态,内容等 readContract 解开后再校验 if (sec.$ref !== undefined) { if (typeof sec.$ref !== 'string' || !sec.$ref) err('reference', `${s}.$ref must be a non-empty string`); else if (path.isAbsolute(sec.$ref) || sec.$ref.split('/').includes('..')) { err('reference', `${s}.$ref must be a relative path inside artifacts/: ${sec.$ref}`); } continue; } if (s === 'benchmark') validateBenchmark(sec, err); if (s === 'test_data') validateTestData(sec, err); if (s === 'test_script') validateTestScript(sec, err); } return { ok: errors.length === 0, errors }; } function validateBenchmark(sec, err) { if (!Array.isArray(sec.dimensions) || sec.dimensions.length === 0) { err('section', 'benchmark.dimensions must be a non-empty array'); return; } let sum = 0; const seen = new Set(); for (const d of sec.dimensions) { if (!d || typeof d.name !== 'string' || !d.name) { err('section', 'benchmark.dimensions[].name required'); continue; } if (seen.has(d.name)) err('section', `benchmark duplicate dimension: ${d.name}`); seen.add(d.name); if (typeof d.weight !== 'number' || d.weight <= 0) err('section', `benchmark.${d.name}.weight must be > 0`); else sum += d.weight; if (d.pass_threshold !== undefined && typeof d.pass_threshold !== 'number') { err('section', `benchmark.${d.name}.pass_threshold must be a number`); } } if (seen.size > 0 && Math.abs(sum - 1) > 1e-6) { err('section', `benchmark dimension weights must sum to 1, got ${sum}`); } if (sec.overall_pass_threshold !== undefined && typeof sec.overall_pass_threshold !== 'number') { err('section', 'benchmark.overall_pass_threshold must be a number'); } } function validateTestData(sec, err) { if (!Array.isArray(sec.cases) || sec.cases.length === 0) { err('section', 'test_data.cases must be a non-empty array'); return; } if (sec.match !== undefined && !MATCH_MODES.includes(sec.match)) { err('section', `test_data.match must be one of ${MATCH_MODES.join('|')}`); } const seen = new Set(); for (const c of sec.cases) { if (!c || typeof c.id !== 'string' || !c.id) { err('section', 'test_data.cases[].id required'); continue; } if (seen.has(c.id)) err('section', `test_data duplicate case id: ${c.id}`); seen.add(c.id); if (!('expect' in c)) err('section', `test_data.cases[${c.id}].expect required`); } } function validateTestScript(sec, err) { if (!RUNNERS.includes(sec.runner)) { err('runner', `test_script.runner must be one of ${RUNNERS.join('|')}, got ${JSON.stringify(sec.runner)}`); } if (typeof sec.path !== 'string' || !sec.path) { err('section', 'test_script.path required'); } else if (path.isAbsolute(sec.path) || sec.path.split('/').includes('..')) { err('reference', `test_script.path must be a relative path inside artifacts/: ${sec.path}`); } if (sec.args !== undefined && !Array.isArray(sec.args)) err('section', 'test_script.args must be an array'); if (sec.timeout_ms !== undefined && (typeof sec.timeout_ms !== 'number' || sec.timeout_ms <= 0)) { err('section', 'test_script.timeout_ms must be a positive number'); } } /** * 解开 section 上的 $ref(相对 artifacts/,不许逃出去)。 * * contract_hash 一律按**解引用后**的契约计算——否则「内联写」和「$ref 外置写」 * 同一份验收标准会算出两个 hash,判决就追溯不回契约版本了。读和写都走这里。 */ function resolveRefs(contract, artifactsDir) { const out = { ...contract }; for (const s of SECTIONS) { const sec = out[s]; if (!sec || typeof sec !== 'object' || sec.$ref === undefined) continue; const abs = resolveInsideArtifacts(artifactsDir, sec.$ref); if (!abs) throw new ContractValidationError([{ rule: 'reference', detail: `${s}.$ref escapes artifacts/: ${sec.$ref}` }]); const sub = readJsonFile(abs); if (!sub.ok) throw new ContractValidationError([{ rule: 'reference', detail: `${s}.$ref unreadable: ${sec.$ref} (${sub.error})` }]); out[s] = sub.json; } return out; } /** * 读契约并解开 $ref。 * @returns {{contract: object, path: string}|null} 没有契约文件返回 null */ function readContract(runDir) { const p = resolveContractPath(runDir); if (!fs.existsSync(p)) return null; const r = readJsonFile(p); if (!r.ok) throw new ContractValidationError([{ rule: 'schema', detail: `unreadable ${CONTRACT_FILE}: ${r.error}` }]); return { contract: resolveRefs(r.json, resolveArtifactsDir(runDir)), path: p }; } /** * 写契约(写前必校验,不通过抛 ContractValidationError;CONVENTIONS §12 同款约定)。 * 文件按原样落盘(保留 $ref),返回的 contract_hash 按解引用后的形态算。 * 因此 $ref 指向的文件必须先写好,否则这里就报错——比等到执行时才炸要早。 */ function writeContract(runDir, contract) { const v = validateContract(contract); if (!v.ok) throw new ContractValidationError(v.errors); const resolved = resolveRefs(contract, resolveArtifactsDir(runDir)); const rv = validateContract(resolved); // 解引用后的内容同样要合法 if (!rv.ok) throw new ContractValidationError(rv.errors); const p = resolveContractPath(runDir); writeJsonAtomic(p, contract); return { path: p, contract_hash: hashContract(resolved) }; } /** * 写**一段** section,合并进已有的 acceptance.json(产出侧唯一该用的入口)。 * * §一.1 的不变量是「三个产出节点(goal-benchmark / goal-test-data / goal-test-script) * 各写自己那一段,互不覆盖」,但 `writeContract` 是整份替换——三个节点先后调它, * 后写的会把先写的那两段抹掉,契约到了 org-validator 手上只剩最后一段。 * 这就是「产生 → 落库」之间被静默吞掉的那一环,所以产出侧必须走这个函数。 * * @param {string} runDir * @param {object} args {section, content, goalId, createdBy} * content 可以是内联对象,也可以是 {"$ref": "goal-benchmark.json"} * @returns {{path: string, contract_hash: string, sections: string[]}} */ function writeContractSection(runDir, { section, content, goalId, createdBy } = {}) { if (!SECTIONS.includes(section)) { throw new ContractValidationError([ { rule: 'schema', detail: `section must be one of ${SECTIONS.join('|')}, got ${JSON.stringify(section)}` }, ]); } if (!content || typeof content !== 'object' || Array.isArray(content)) { throw new ContractValidationError([{ rule: 'section', detail: `${section} content must be a plain object` }]); } // 读**原文**(不解 $ref):合并要保留别人写的 $ref 形态,解开再写回等于把外置内容抄进契约 const p = resolveContractPath(runDir); let base = {}; if (fs.existsSync(p)) { const r = readJsonFile(p); if (!r.ok) throw new ContractValidationError([{ rule: 'schema', detail: `unreadable ${CONTRACT_FILE}: ${r.error}` }]); base = r.json && typeof r.json === 'object' && !Array.isArray(r.json) ? r.json : {}; } const nextGoal = goalId || base.goal_id; if (base.goal_id && goalId && base.goal_id !== goalId) { // 一个 root run 一份契约、一个 goal(§一.1)。串了 goal 说明调用方拿错了 runDir, // 静默覆盖会让判决追溯到一个根本不是它的 goal_id 上。 throw new ContractValidationError([ { rule: 'schema', detail: `goal_id mismatch: existing ${base.goal_id}, incoming ${goalId}` }, ]); } const merged = { ...base, version: CONTRACT_VERSION, goal_id: nextGoal, created_at: base.created_at || now(), // 保留首写时间:contract_hash 含它,刷新会让 hash 无谓变动 [section]: content, }; if (createdBy) { const prev = typeof base.created_by === 'string' ? {} : (base.created_by || {}); merged.created_by = { ...prev, [section]: createdBy }; } const out = writeContract(runDir, merged); return { ...out, sections: SECTIONS.filter((s) => merged[s] != null) }; } // ──────────────────────────────────────────────────────────────── // 执行器 // ──────────────────────────────────────────────────────────────── /** * 短路失败(还没跑到逐维度/逐 case 比对就已经判定不通过)的 failures 条目。 * * 形状与正常比对产出的 failure 完全一致(case / expected / got / hint), * 这样 `executeContract` 的 `failures: active.flatMap(...)` 和 CLI 的 `fmtFailures` * 都不用为短路分支分叉——**下游只认一种形状**。 * `hint` 里带上 `reason`,让扁平汇总单看一行也能归因。 */ function shortCircuitFailure(caseName, reason, detail, expected) { return { case: caseName, expected, got: detail === undefined || detail === null ? null : String(detail), hint: `${reason}: ${SHORT_CIRCUIT_HINTS[reason] || '执行前置条件不满足,未进入逐项比对'}`, }; } const SHORT_CIRCUIT_HINTS = { target_not_json: '被测产物读不出 JSON(不存在 / 不是合法 JSON),无法比对', script_missing: '契约指向的验收脚本不在 artifacts/ 下,未起进程', script_escapes_artifacts: '验收脚本路径逃出 artifacts/,拒绝执行', }; /** * benchmark:被测产物提供各维度分数,执行器只做加权与阈值比较(纯声明式,不起 LLM)。 * 被测产物形态:{"scores": {"": 87, ...}} 或直接 {"": 87, ...} */ function runBenchmark(sec, target) { // 短路分支也必须进 failures:verdict.failures 是下游(validator agent / validator-triager) // 唯一的扁平归因入口,只留 reason 不留 failures 的话「读回」这一环就空了(§十 四环缺陷)。 if (!target.ok) { return { ok: false, reason: 'target_not_json', detail: target.error, score: null, dimensions: [], failures: [shortCircuitFailure('benchmark', 'target_not_json', target.error, '可解析为 JSON 的被测产物')], }; } const scores = (target.json && typeof target.json.scores === 'object' && target.json.scores) || target.json || {}; const dims = []; let weighted = 0; let ok = true; for (const d of sec.dimensions) { const score = scores[d.name]; const has = typeof score === 'number'; const threshold = typeof d.pass_threshold === 'number' ? d.pass_threshold : null; const passed = has && (threshold === null || score >= threshold); if (!passed) ok = false; if (has) weighted += score * d.weight; dims.push({ name: d.name, weight: d.weight, score: has ? score : null, pass_threshold: threshold, ok: passed, reason: has ? undefined : 'missing score in target', }); } const overall = typeof sec.overall_pass_threshold === 'number' ? sec.overall_pass_threshold : null; const overallOk = overall === null || weighted >= overall; if (!overallOk) ok = false; return { ok, score: Number(weighted.toFixed(4)), overall_pass_threshold: overall, overall_ok: overallOk, dimensions: dims, failures: dims.filter((d) => !d.ok).map((d) => ({ case: d.name, expected: d.pass_threshold, got: d.score, hint: d.reason || 'dimension score below pass_threshold', })), }; } /** * test_data:被测产物提供各 case 的实际输出,执行器逐 case 比对(纯声明式,不起 LLM)。 * 被测产物形态:{"results": {"": }} 或 {"results": [{"id":..,"output":..}]} */ function runTestData(sec, target) { if (!target.ok) { return { ok: false, reason: 'target_not_json', detail: target.error, match: sec.match || 'deep-equal', total: sec.cases.length, passed: 0, failed: sec.cases.length, failures: [shortCircuitFailure('test_data', 'target_not_json', target.error, '可解析为 JSON 的被测产物')], }; } const raw = (target.json && target.json.results !== undefined ? target.json.results : target.json) || {}; const actual = Array.isArray(raw) ? Object.fromEntries(raw.map((r) => [r && r.id, r && (r.output !== undefined ? r.output : r.actual)])) : raw; const mode = sec.match || 'deep-equal'; const failures = []; for (const c of sec.cases) { const got = actual[c.id]; const missing = !(c.id in actual); const passed = !missing && (mode === 'subset' ? matchSubset(c.expect, got) : deepEqual(c.expect, got)); if (!passed) { failures.push({ case: c.id, expected: c.expect, got: missing ? null : got, hint: missing ? `no result for case ${c.id} in target` : `case ${c.id} mismatch (${mode})`, }); } } return { ok: failures.length === 0, match: mode, total: sec.cases.length, passed: sec.cases.length - failures.length, failed: failures.length, failures, }; } /** 起子进程跑脚本;超时先 SIGTERM 再 SIGKILL(沿用 integrator-tools.killAll 的做法) */ function spawnScript({ cmd, args, cwd, env, timeoutMs }) { return new Promise((resolve) => { const started = Date.now(); let stdout = ''; let stderr = ''; let timedOut = false; const child = spawn(cmd, args, { cwd, env, stdio: ['ignore', 'pipe', 'pipe'] }); const cap = (buf, chunk) => (buf.length >= OUTPUT_CAP ? buf : buf + String(chunk).slice(0, OUTPUT_CAP - buf.length)); child.stdout.on('data', (c) => { stdout = cap(stdout, c); }); child.stderr.on('data', (c) => { stderr = cap(stderr, c); }); const killer = setTimeout(() => { timedOut = true; try { child.kill('SIGTERM'); } catch { /* already gone */ } setTimeout(() => { try { child.kill('SIGKILL'); } catch { /* already gone */ } }, 3000).unref(); }, timeoutMs); killer.unref(); child.on('error', (e) => { clearTimeout(killer); resolve({ exit_code: null, spawn_error: e.message, stdout, stderr, timed_out: false, duration_ms: Date.now() - started }); }); child.on('close', (code, signal) => { clearTimeout(killer); resolve({ exit_code: code, signal: signal || null, stdout, stderr, timed_out: timedOut, duration_ms: Date.now() - started }); }); }); } /** * test_script:真实子进程执行(DevLoop Integrator 起真实进程跑 e2e 的同款做法)。 * 被测产物路径通过 env ACCEPTANCE_TARGET 交给脚本,脚本退出码即判决。 */ async function runTestScript(sec, { artifactsDir, runDir, targetPath }) { const scriptAbs = resolveInsideArtifacts(artifactsDir, sec.path); if (!scriptAbs) { return { ok: false, reason: 'script_escapes_artifacts', detail: sec.path, runner: sec.runner, script: sec.path, exit_code: null, failures: [shortCircuitFailure(sec.path, 'script_escapes_artifacts', sec.path, 'artifacts/ 之内的相对脚本路径')], }; } if (!fs.existsSync(scriptAbs)) { return { ok: false, reason: 'script_missing', detail: scriptAbs, runner: sec.runner, script: sec.path, exit_code: null, failures: [shortCircuitFailure(sec.path, 'script_missing', scriptAbs, '契约指向的脚本文件存在')], }; } const res = await spawnScript({ cmd: process.execPath, // RUNNERS 白名单目前只有 'node' args: [scriptAbs, ...(sec.args || [])], cwd: sec.cwd ? resolveInsideArtifacts(artifactsDir, sec.cwd) || runDir : runDir, env: { ...process.env, ACCEPTANCE_TARGET: targetPath, ACCEPTANCE_RUN_DIR: runDir || '', ACCEPTANCE_ARTIFACTS_DIR: artifactsDir || '', }, timeoutMs: sec.timeout_ms || DEFAULT_SCRIPT_TIMEOUT_MS, }); const expectCode = (sec.pass_on && typeof sec.pass_on.exit_code === 'number') ? sec.pass_on.exit_code : 0; const ok = !res.timed_out && !res.spawn_error && res.exit_code === expectCode; return { ok, runner: sec.runner, script: sec.path, exit_code: res.exit_code, signal: res.signal, timed_out: res.timed_out, duration_ms: res.duration_ms, stdout_excerpt: res.stdout.slice(-4000), stderr_excerpt: res.stderr.slice(-4000), reason: res.timed_out ? 'script_timeout' : (res.spawn_error ? 'spawn_error' : undefined), failures: ok ? [] : [{ case: sec.path, expected: `exit ${expectCode}`, got: res.timed_out ? 'timeout' : (res.spawn_error || `exit ${res.exit_code}`), hint: (res.stderr || res.stdout).slice(-800), }], }; } /** * 基础设施抖动 → infra_flake(§四.D.2.2 的取值域)。 * 这类失败在引擎侧原地重跑、不烧任何计数,所以必须与「真的没达标」区分开。 */ const INFRA_REASONS = new Set(['script_timeout', 'spawn_error']); /** * 执行一份契约,产出判决(纯执行,不落盘)。 * * @param {object} contract 已解引用的契约(readContract 的产物 / 手写对象) * @param {string} targetPath 被测产物绝对路径(通常来自 graph_run_nodes.output_ref) * @param {object} [opts] {runDir, runId, nodeRunId, attempt} * @returns {Promise} verdict */ async function executeContract(contract, targetPath, opts = {}) { const started = Date.now(); const runDir = opts.runDir || null; const artifactsDir = runDir ? resolveArtifactsDir(runDir) : null; // 相对路径先归一:下面既拿它做 startsWith 判断又拿它做 path.relative, // 两处用不同形态会算出「判断说在 run 目录里、relative 却算出一串 ../」的结果。 const targetAbs = path.resolve(targetPath); const base = { verdict_id: genVerdictId(), goal_id: contract && contract.goal_id, contract_hash: null, run_id: opts.runId || null, node_run_id: opts.nodeRunId || null, // ⚠️ 不能写 `opts.attempt || null`:attempt 是 **0 基**(graph-engine 的 picked.attempt 从 0 起), // 第一次尝试的 0 会被 `||` 判成假值吞掉,判决四元组当场少一环、追溯不回是哪次尝试。 attempt: Number.isInteger(opts.attempt) ? opts.attempt : null, // 落在 run 目录里的产物存相对路径(整个 run 目录可整体搬走 / 归档后仍可读), // 外部路径原样存绝对路径 target: runDir && targetAbs.startsWith(path.resolve(runDir) + path.sep) ? path.relative(path.resolve(runDir), targetAbs) : targetAbs, created_at: now(), }; const v = validateContract(contract); if (!v.ok) { // 契约本身不合法:不是产物的错,归因为「输入约束没写清」,退回契约产出者。 return { ...base, ok: false, fail_reason: 'input_underspecified', sections: { benchmark: null, test_data: null, test_script: null }, failures: v.errors.map((e) => ({ case: e.rule, expected: 'valid contract', got: e.detail, hint: 'fix acceptance.json' })), duration_ms: Date.now() - started, }; } base.contract_hash = hashContract(contract); const needsJson = contract.benchmark || contract.test_data; const target = needsJson ? (fs.existsSync(targetAbs) ? readJsonFile(targetAbs) : { ok: false, error: `target not found: ${targetAbs}` }) : { ok: true, json: null }; const sections = { benchmark: null, test_data: null, test_script: null }; if (contract.benchmark) sections.benchmark = runBenchmark(contract.benchmark, target); if (contract.test_data) sections.test_data = runTestData(contract.test_data, target); if (contract.test_script) { // 脚本拿到的 ACCEPTANCE_TARGET 必须是绝对路径:子进程的 cwd 是 runDir/artifacts 下的某处, // 传相对路径过去它解析出来的是另一个文件。 sections.test_script = await runTestScript(contract.test_script, { artifactsDir, runDir, targetPath: targetAbs }); } const active = SECTIONS.map((s) => sections[s]).filter(Boolean); const ok = active.every((s) => s.ok); const infra = active.some((s) => INFRA_REASONS.has(s.reason)); return { ...base, ok, // 判决本身只回答「这次验收有没有达成」,**不**寻址任何节点:它是机器执行的合同校验, // 手上只有 target 与 sections,没有图的信息。跨层定点(target_node → fail_node_ref) // 由下游 validator-triager 依据这里的 failures 归因产出(R-17.1 定案;原 R-16 // 「回退只退到 org node 粒度」的假设已作废,引擎现在支持退到子图里的具体 node)。 fail_reason: ok ? null : (infra ? 'infra_flake' : 'acceptance_not_met'), sections, // 扁平汇总,供下游 validator-triager 归因直接消费(R-17.2 定名) failures: active.flatMap((s) => s.failures || []), duration_ms: Date.now() - started, }; } // ──────────────────────────────────────────────────────────────── // 判决持久化 // ──────────────────────────────────────────────────────────────── /** * 落盘一份判决:完整体写 verdicts/{verdictId}.json,摘要 append 到 verdicts/index.ndjson。 * index 永不改已写行(CONVENTIONS §12 的 ndjson 不变量)。 */ function persistVerdict(runDir, verdict) { const dir = resolveVerdictsDir(runDir); const p = path.join(dir, `${verdict.verdict_id}.json`); writeJsonAtomic(p, verdict); const indexPath = path.join(dir, 'index.ndjson'); const entry = { verdict_id: verdict.verdict_id, ok: verdict.ok, fail_reason: verdict.fail_reason || null, goal_id: verdict.goal_id || null, run_id: verdict.run_id || null, node_run_id: verdict.node_run_id || null, // 同 executeContract:attempt 0 基,`||` 会把第一次尝试吞成 null, // 于是 listVerdicts 读回来的索引里首轮判决永远查不到 attempt。 attempt: Number.isInteger(verdict.attempt) ? verdict.attempt : null, contract_hash: verdict.contract_hash || null, target: verdict.target || null, created_at: verdict.created_at, file: path.relative(runDir, p), }; fs.appendFileSync(indexPath, JSON.stringify(entry) + '\n'); return { path: p, indexPath, entry }; } /** 查判决索引(可按 run / node / 只看失败过滤),新的在后,limit 取末尾 N 条 */ function listVerdicts(runDir, filter = {}) { const indexPath = path.join(resolveVerdictsDir(runDir), 'index.ndjson'); if (!fs.existsSync(indexPath)) return []; let rows = fs .readFileSync(indexPath, 'utf8') .split('\n') .filter(Boolean) .map((l) => { try { return JSON.parse(l); } catch { return null; } }) .filter(Boolean); if (filter.runId) rows = rows.filter((r) => r.run_id === filter.runId); if (filter.nodeRunId) rows = rows.filter((r) => r.node_run_id === filter.nodeRunId); if (filter.onlyFailed) rows = rows.filter((r) => !r.ok); if (filter.limit) rows = rows.slice(-filter.limit); return rows; } /** 读单份完整判决 */ function readVerdict(runDir, verdictId) { const p = path.join(resolveVerdictsDir(runDir), `${verdictId}.json`); if (!fs.existsSync(p)) return null; const r = readJsonFile(p); return r.ok ? r.json : null; } /** * 一站式入口:读 runDir 的契约 → 执行 → 落盘判决。引擎侧的 validator 节点调这个。 * * @param {object} args {runDir, targetPath, runId, nodeRunId, attempt} * @returns {Promise<{verdict: object, path: string}>} */ async function runAcceptance({ runDir, targetPath, runId, nodeRunId, attempt }) { const loaded = readContract(runDir); if (!loaded) { throw new ContractValidationError([ { rule: 'reference', detail: `no ${CONTRACT_FILE} under ${resolveArtifactsDir(runDir)}` }, ]); } const verdict = await executeContract(loaded.contract, targetPath, { runDir, runId, nodeRunId, attempt }); const { path: p } = persistVerdict(runDir, verdict); return { verdict, path: p }; } // ──────────────────────────────────────────────────────────────── // CLI(agent 侧的生产入口,见文件顶部「生产调用方」) // ──────────────────────────────────────────────────────────────── const USAGE = `goal-contract — 验收契约的产出 / 校验 / 执行 / 查询(27-goal-contract-format.md) put --run-dir --section (--file | --ref ) [--goal-id ] [--created-by <节点本体名>] 产出侧:把**一段** section 合并进 artifacts/acceptance.json,不覆盖别人写的段。 validate --run-dir 只校验契约本身(空契约 / 权重不足 1 / runner 越界 / 路径逃逸都在这里被挡)。 run --run-dir --target <被测产物路径> [--run-id ] [--node-run-id ] [--attempt ] [--json] 消费侧:读契约 → 执行 → 落盘判决。缺省的 run-dir / node-run-id / attempt 会从 target 路径推。 list --run-dir [--node-run-id ] [--only-failed] [--limit ] show --run-dir --verdict-id 退出码:0 = 通过 / 操作成功;1 = **判决为 fail**(工具本身是好的);2 = 跑不出判决(参数错 / 没契约 / 契约非法)。 `; /** 从被测产物路径向上找到含 artifacts/acceptance.json 的那一层(agent 少填一个参数少错一次) */ function deriveRunDir(targetPath) { let dir = path.dirname(path.resolve(targetPath)); for (let i = 0; i < 12; i++) { if (fs.existsSync(path.join(dir, 'artifacts', CONTRACT_FILE))) return dir; const up = path.dirname(dir); if (up === dir) break; dir = up; } return null; } /** .../nodes/{nodeRunId}/attempt-{n}.json → {nodeRunId, attempt};推不出来就留 null */ function deriveTargetIds(targetPath) { const abs = path.resolve(targetPath); const parts = abs.split(path.sep); const i = parts.lastIndexOf('nodes'); const nodeRunId = i >= 0 && parts[i + 1] ? parts[i + 1] : null; const m = /^attempt-(\d+)\./.exec(path.basename(abs)); return { nodeRunId, attempt: m ? Number(m[1]) : null }; } function fmtFailures(failures) { const rows = failures || []; const out = rows.slice(0, 20).map( (f) => ` - ${f.case}: expected=${JSON.stringify(f.expected)} got=${JSON.stringify(f.got)}${f.hint ? ` | ${f.hint}` : ''}`, ); if (rows.length > 20) out.push(` … 另有 ${rows.length - 20} 条,完整清单见 verdict 文件`); // 判决为 fail 却一条 failure 都没有,本身就是要报出来的异常(旧版直接打个空的 `failures:` 标题, // agent 看到的是「fail 但没有任何可归因信息」,分不清「工具吞了」还是「真的没细节」)。 if (rows.length === 0) out.push(' (无逐项明细——看上面的 sections 行与 fail_reason)'); return out; } /** * 一段 section 的人读摘要。 * * ⚠️ **短路分支没有 score / passed / exit_code**(`target_not_json` / `script_missing` / * `script_escapes_artifacts` 直接 return,压根没跑到比对),旧版无条件打 * `score=${s.benchmark.score}`,于是照着 `tools.md` 退出码表办事的 agent 只看到 * 「benchmark=FAIL、score=undefined、failures 空」——而落盘的 verdict JSON 里 * reason 和 detail 明明白白都在。**归因信息被吞在「读回」这一环**(§十 的四环缺陷模式)。 * 所以这里按字段是否真的存在分别打,且**只要有 reason 就一定把 reason + detail 打出来**。 */ function fmtSection(name, sec) { if (!sec) return []; const out = []; if (name === 'benchmark' && typeof sec.score === 'number') { out.push(` benchmark score=${sec.score} / ${sec.overall_pass_threshold ?? '-'}`); } if (name === 'test_data' && typeof sec.total === 'number') { out.push(` test_data ${sec.passed}/${sec.total} passed (match=${sec.match})`); } // duration_ms 只有真起过进程才有:短路分支打一行 `exit=null` 会被读成「跑了、返回 null」 if (name === 'test_script' && sec.duration_ms !== undefined) { out.push(` test_script exit=${sec.exit_code} timed_out=${sec.timed_out === true} (${sec.duration_ms}ms)`); } if (sec.reason) { out.push(` ${name} reason=${sec.reason}` + (sec.detail === undefined || sec.detail === null ? '' : ` detail=${sec.detail}`)); } if (name === 'test_script' && sec.stderr_excerpt) out.push(` stderr: ${sec.stderr_excerpt.slice(-500)}`); if (out.length === 0) out.push(` ${name} ${sec.ok ? 'ok' : 'FAIL'}(判决文件里也没有更多明细)`); return out; } async function cliMain(argv) { const cmd = argv[0]; if (!cmd || cmd === 'help' || cmd === '--help' || cmd === '-h') { process.stdout.write(USAGE); return cmd ? 0 : 2; } const { values: a } = require('util').parseArgs({ args: argv.slice(1), allowPositionals: false, options: { 'run-dir': { type: 'string' }, target: { type: 'string' }, 'goal-id': { type: 'string' }, section: { type: 'string' }, file: { type: 'string' }, ref: { type: 'string' }, 'created-by': { type: 'string' }, 'run-id': { type: 'string' }, 'node-run-id': { type: 'string' }, attempt: { type: 'string' }, 'verdict-id': { type: 'string' }, 'only-failed': { type: 'boolean' }, limit: { type: 'string' }, json: { type: 'boolean' }, }, }); const needRunDir = () => { const d = a['run-dir'] || (a.target ? deriveRunDir(a.target) : null); if (!d) throw new Error('missing --run-dir (且无法从 --target 推出 run 目录)'); return d; }; if (cmd === 'put') { const runDir = needRunDir(); if (!a.file && !a.ref) throw new Error('put 需要 --file 或 --ref 其一'); const content = a.ref ? { $ref: a.ref } : JSON.parse(fs.readFileSync(a.file, 'utf8')); const r = writeContractSection(runDir, { section: a.section, content, goalId: a['goal-id'], createdBy: a['created-by'], }); process.stdout.write(`OK ${a.section} 已写入 ${r.path}\n` + `contract_hash: ${r.contract_hash}\nsections: ${r.sections.join(', ') || '(空)'}\n`); return 0; } if (cmd === 'validate') { const runDir = needRunDir(); const loaded = readContract(runDir); // $ref 解不开会在这里抛,正是要挡的 if (!loaded) throw new Error(`没有 ${CONTRACT_FILE}:${resolveArtifactsDir(runDir)}`); const v = validateContract(loaded.contract); const present = SECTIONS.filter((s) => loaded.contract[s] != null); if (v.ok) { process.stdout.write(`OK 契约合法:${loaded.path}\nsections: ${present.join(', ')}\n` + `contract_hash: ${hashContract(loaded.contract)}\n`); return 0; } process.stdout.write('INVALID 契约不合法(= fail_reason: input_underspecified,退回契约产出者)\n' + v.errors.map((e) => ` - [${e.rule}] ${e.detail}`).join('\n') + '\n'); return 1; } if (cmd === 'run') { if (!a.target) throw new Error('run 需要 --target <被测产物路径>'); const runDir = needRunDir(); const derived = deriveTargetIds(a.target); const attempt = a.attempt !== undefined ? Number(a.attempt) : derived.attempt; const { verdict, path: vp } = await runAcceptance({ runDir, targetPath: a.target, runId: a['run-id'] || null, nodeRunId: a['node-run-id'] || derived.nodeRunId, attempt: Number.isInteger(attempt) ? attempt : null, }); if (a.json) { process.stdout.write(JSON.stringify(verdict, null, 2) + '\n'); } else { const s = verdict.sections; const lines = [ `VERDICT ${verdict.ok ? 'pass' : 'fail'} fail_reason=${verdict.fail_reason || 'null'}` + ` goal=${verdict.goal_id || '?'} attempt=${verdict.attempt === null ? '?' : verdict.attempt}`, `verdict_id: ${verdict.verdict_id} contract_hash: ${verdict.contract_hash || '?'}`, `verdict_file: ${vp}`, `target: ${verdict.target}`, `sections: ${SECTIONS.map((k) => (s[k] ? `${k}=${s[k].ok ? 'ok' : 'FAIL'}` : `${k}=-`)).join(' ')}`, ]; for (const k of SECTIONS) lines.push(...fmtSection(k, s[k])); if (!verdict.ok) lines.push('failures:', ...fmtFailures(verdict.failures)); process.stdout.write(lines.join('\n') + '\n'); } return verdict.ok ? 0 : 1; } if (cmd === 'list') { const runDir = needRunDir(); const rows = listVerdicts(runDir, { nodeRunId: a['node-run-id'], runId: a['run-id'], onlyFailed: !!a['only-failed'], limit: a.limit ? Number(a.limit) : undefined, }); process.stdout.write(rows.length ? rows.map((r) => `${r.created_at} ${r.ok ? 'pass' : 'fail'} ${r.verdict_id}` + ` node=${r.node_run_id || '-'} attempt=${r.attempt === null ? '-' : r.attempt}` + ` ${r.fail_reason || ''}`).join('\n') + '\n' : '(没有判决记录)\n'); return 0; } if (cmd === 'show') { const runDir = needRunDir(); const v = readVerdict(runDir, a['verdict-id']); if (!v) throw new Error(`没有这份判决:${a['verdict-id']}`); process.stdout.write(JSON.stringify(v, null, 2) + '\n'); return v.ok ? 0 : 1; } throw new Error(`未知子命令:${cmd}\n\n${USAGE}`); } if (require.main === module) { cliMain(process.argv.slice(2)) .then((code) => process.exit(code)) .catch((e) => { // 契约非法 / 参数错 / 没契约 → 2:与「判决为 fail」的 1 分开, // 否则 agent 分不清「产物没达标」和「工具压根没跑起来」(后者是 input_underspecified)。 const detail = e instanceof ContractValidationError ? e.errors.map((x) => ` - [${x.rule}] ${x.detail}`).join('\n') : ''; process.stderr.write(`ERROR ${e.message}\n${detail}${detail ? '\n' : ''}`); process.exit(2); }); } module.exports = { // 常量 CONTRACT_VERSION, CONTRACT_FILE, SECTIONS, RUNNERS, ContractValidationError, // 路径 resolveRunDir, resolveArtifactsDir, resolveContractPath, resolveVerdictsDir, // 契约 validateContract, readContract, writeContract, writeContractSection, resolveRefs, contractHash: hashContract, // 入参必须是**已解引用**的契约 // 执行 executeContract, runAcceptance, // 判决 persistVerdict, listVerdicts, readVerdict, // CLI(测试直接跑它,保证模板里印的那几行命令是真能跑的) cliMain, deriveRunDir, deriveTargetIds, };