Root cause traced (docs/implementation-gap-analysis.md): the Benchmark
document Validator relies on for its verdict was frequently a narrative
about the work session ("The planner workflow completed and produced a
Benchmark defining 18 scenarios...") instead of the actual scenarios --
found via a real skill-detail-page task's benchmark.md, then confirmed
present since the earliest real self-hosting runs. Validator's own
judgment was, in practice, resting on thin, sometimes-vacuous prose.
Planner's work graph (brd->prd->sdd->benchmark) extends to
benchmark->test_dataset->test_script: test_dataset/test_script are always
present as nodes, but the generating subagent judges per-goal whether a
real dataset/script applies (many goals, e.g. UI wiring, legitimately
don't have one -- replying 'N/A: <reason>' is a real, expected answer, not
a failure). test_script gets a genuinely mechanical verify_fn (real
py_compile/node --check, not LLM judgment) -- the one node type where an
objective check is actually possible. All four document prompts also gained
an explicit anti-narration instruction (only 'brd' had a weaker version of
this before).
Coder's implementation prompt now receives the real benchmark/dataset/
script and must make the Planner's real script pass for real, not just
whatever tests it invents on its own.
Validator gets its own 3-node Work Graph (validator_benchmark ->
validator_test_dataset -> validator_test_script), generated independently
from SDD/PRD -- deliberately NOT reusing Coder's or Planner's own copies,
same reasoning as a held-out eval set vs. the dev set a model was tuned
against. The resulting script, if real, is actually executed against the
current repository state; a non-zero real exit code force-overrides the
verdict to FAIL in code (user-confirmed hard gate: the script is not up
for LLM debate on a failure, only on a pass can subjective judgment add
INCONCLUSIVE/risk notes).
schema.py: artifact.artifact_type's CHECK constraint gains the 5 new
types, plus migrate_expand_artifact_types() -- SQLite has no ALTER-based
constraint migration, so this is the standard rebuild-copy-swap dance,
applied for real to the live ~/.graphlemon/graphlemon.db (42 rows,
verified byte-identical count before/after, integrity_check clean).
skills/qa-task-validation.md and skills/project-sdd-planning.md already
gained a standing principle from this same incident: verify cross-task
data dependencies against real system state, not seeded fixtures alone
(see prior commit d7035fc) -- this commit is the mechanism-level half of
the same fix, not just the prose-guidance half.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
