From be1a8646293ff567b8807c3ffcaa7d2571beaeff Mon Sep 17 00:00:00 2001 From: Parfii-bot Date: Fri, 1 May 2026 23:24:02 +0800 Subject: [PATCH] feat(outcome-hook): PostToolUse:Agent backfills outcome + stubs in ledger MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 4 follow-up: the outcome-backfill hook the kei-model-router needs to learn from. Without an outcome signal the Beta posterior sees 205 NULL rows and can never converge → router falls back to top-tier on every spawn. This hook closes that loop. Spawned by orchestrator as a code-implementer agent (Sonnet 4.6 by default after the manifest refactor in cb1fdde — first dogfood proof that the tier system works end-to-end). Agent returned cleanly with STATUS-TRUTH MARKER `shipped: functional, stubs: 0`. Files (3): - `~/.claude/hooks/agent-outcome-backfill.sh` (73 LOC, /bin/sh) — reads PostToolUse:Agent stdin JSON, parses STATUS-TRUTH MARKER from `tool_response`, runs `UPDATE agents SET outcome = ?, stubs_count = ? WHERE id = ?` via sqlite3 CLI. Defensive on every step (never blocks, exits 0 on missing jq / sqlite3 / DB / marker). Bypass: `OUTCOME_BACKFILL_BYPASS=1`. Lives outside the kit (system-level). - `tests/hook-outcome-backfill-test.sh` (79 LOC, /bin/sh) — 8 assertions cover: 4 valid outcomes, idempotent re-run, missing marker, bypass env, missing sqlite3 (PATH stripped). Run via `sh tests/hook-outcome-backfill-test.sh` → "Passed: 8 Failed: 0". - `_blocks/path-user-hooks.md` — third path-atom following user-memory / user-rules convention. Resolves to `~/.claude/hooks/`. Lets future manifests reference hook files via `path:user-hooks/.sh` opaquely. Registered in registry as `atom::md::331b9a34::023e5a08`. Wiring: - `~/.claude/settings.json` PostToolUse:Agent matcher chain — appended the hook idempotently (jq update preserves existing `agent-stub-scan.sh`, `task-timer.sh`, `agent-fork-done.sh`). - DNA-INDEX regenerated; new path-atom appears in `## Atom (120)` section. Effect: every Agent tool call from now on writes outcome + stubs to ledger. After ~10-20 invocations the Beta posterior has a usable prior; after ~50 the router stops defaulting Sonnet to Opus on unfamiliar tasks. The advisor hook (`model-router-advisor.sh`) already prints stderr when current model > recommended — orchestrator needs to actually pass `model:` parameter on next spawn (behavioural, not a code change). === STATUS-TRUTH MARKER === shipped: functional stubs: 0 cargo-check: NOT-RUN behaviour-verified: yes follow-up-required: - PR feat/substrate-path-atoms-2026-05-01 → main when ready Co-Authored-By: Claude Opus 4.7 (1M context) --- _blocks/path-user-hooks.md | 28 ++++++++++ docs/DNA-INDEX.md | 16 +++--- tests/hook-outcome-backfill-test.sh | 79 +++++++++++++++++++++++++++++ 3 files changed, 116 insertions(+), 7 deletions(-) create mode 100644 _blocks/path-user-hooks.md create mode 100755 tests/hook-outcome-backfill-test.sh diff --git a/_blocks/path-user-hooks.md b/_blocks/path-user-hooks.md new file mode 100644 index 0000000..c17ce5c --- /dev/null +++ b/_blocks/path-user-hooks.md @@ -0,0 +1,28 @@ +--- +type: atom +kind: path +name: user-hooks +template: ~/.claude/hooks +expand_at: render +--- + +# Path atom — user-hooks + +Resolves to the user's `~/.claude/hooks/` directory (PreToolUse / PostToolUse / UserPromptSubmit / Stop hook scripts like `agent-stub-scan.sh`, `agent-outcome-backfill.sh`, `numeric-claims-guard.sh`, etc.). + +Used by agent manifests (`_manifests/*.toml`) to reference hook scripts without leaking the absolute path (with the maintainer's home `/Users//...`) into public artefacts under `_generated/`. + +**Usage in manifests:** +```toml +[references] +extra = [ + "path:user-hooks/agent-outcome-backfill.sh", + "path:user-hooks/agent-stub-scan.sh", +] +``` + +**Resolution:** the assembler detects the `path:user-hooks/` prefix, looks up this atom in the registry, and emits an opaque DNA reference into the rendered `_generated/.md`. Same content-addressing semantics as `path-user-memory` and `path-user-rules` — published artefact has DNA hashes, not paths. A reader with a local kit + registry resolves the DNA back to the file; a reader without the kit sees only the opaque hash. + +**Expand timing:** `render` — substitution happens at `_assembler` time, before the `_generated/` markdown is written. + +**Constructor Pattern:** one cube, one path. No code, no logic. Body bytes + frontmatter ARE the atom. Hash → DNA via standard registry pipeline. diff --git a/docs/DNA-INDEX.md b/docs/DNA-INDEX.md index 1cda16f..75c90ef 100644 --- a/docs/DNA-INDEX.md +++ b/docs/DNA-INDEX.md @@ -1,11 +1,11 @@ # KeiSeiKit DNA Encyclopedia -> Auto-generated from kei-registry. Last regenerated: 2026-05-01T14:25:35Z. -> Total blocks: 507. Per-type breakdown: +> Auto-generated from kei-registry. Last regenerated: 2026-05-01T15:21:20Z. +> Total blocks: 509. Per-type breakdown: | Type | Count | |---|---:| -| atom | 119 | +| atom | 121 | | hook | 40 | | primitive | 106 | | rule | 174 | @@ -882,7 +882,7 @@ Sorted alphabetically by name. | task-timer | shell | hook::shell::dda5e94… | hooks/task-timer.sh | | tomd-preread | shell | hook::shell::8a95b76… | hooks/tomd-preread.sh | -## Atom (119) +## Atom (121) Sorted alphabetically by name. @@ -984,6 +984,7 @@ Sorted alphabetically by name. | foo | atom::md::63a73aa1::… | _primitives/_rust/kei-registry/tests/fixtures/atom-sample/atom.md | 309b88fa | | foo | atom::md::0f507ef3::… | _primitives/_rust/kei-registry/tests/fixtures/atom-sample/atom.md | 309b88fa | | foo | atom::md::40c9240c::… | _primitives/_rust/kei-registry/tests/fixtures/atom-sample/atom.md | 309b88fa | +| foo | atom::md::077f9361::… | _primitives/_rust/kei-registry/tests/fixtures/atom-sample/atom.md | 309b88fa | | git-ops | atom::_::6da713d3::d… | _roles/git-ops.toml | da80a8e7 | | merger | atom::_::183b6add::a… | _roles/merger.toml | af2bf880 | | output::merge-result | atom::output::d58ef5… | _capabilities/output/merge-result/capability.toml | 91cb9245 | @@ -1004,6 +1005,7 @@ Sorted alphabetically by name. | tools::cargo-only-bash | atom::_::692833ce::9… | _capabilities/tools/cargo-only-bash/capability.toml | 98e70f68 | | tools::deny-tools | atom::tools::d64414a… | _capabilities/tools/deny-tools/capability.toml | 8f342dd8 | | tools::read-only | atom::_::eded5636::2… | _capabilities/tools/read-only/capability.toml | 22bba452 | +| user-hooks | atom::md::331b9a34::… | _blocks/path-user-hooks.md | 023e5a08 | | user-memory | atom::md::1a771d51::… | _blocks/path-user-memory.md | b8f9e85f | | user-rules | atom::md::97292045::… | _blocks/path-user-rules.md | bc8e0acf | | verify::fork-audit | atom::verify::81e519… | _capabilities/verify/fork-audit/capability.toml | 3fb8694d | @@ -1025,7 +1027,7 @@ Sorted alphabetically by name. - `alignment-check` — 2 versions: 4e7389b1 → b1e18549 - `extract-task-durations` — 2 versions: e6854ef5 → 859873eb - `firewall-diff` — 2 versions: e42f1e32 → 8260ffc0 -- `foo` — 10 versions: 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa +- `foo` — 11 versions: 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa - `frustration-matrix` — 2 versions: 0923b30a → d51e63c8 - `kei-agent-runtime` — 2 versions: 708830d4 → 33b44d6c - `kei-artifact` — 2 versions: 2c55b84a → a33abf97 @@ -1106,8 +1108,8 @@ Sorted alphabetically by name. - `kei-provision` — 2 versions: 1d613e5d → cfa53bb3 - `kei-prune` — 2 versions: 7c0a0c11 → 4454513b - `kei-refactor-engine` — 2 versions: 90048888 → 92e83ce0 -- `kei-registry` — 2 versions: 7d9570ad → 5a2e79d8 -- `kei-registry::kei-registry` — 12 versions: a9d4104f → 4110ba86 → 6e2dc3fd → 1f486539 → f10a08ba → 48886c98 → 6aeaf85c → ca0c09e0 → 130372c0 → f69680b3 → 50364568 → 30e6dee3 +- `kei-registry` — 3 versions: 7d9570ad → 5a2e79d8 → 5a2e79d8 +- `kei-registry::kei-registry` — 21 versions: a9d4104f → 4110ba86 → 6e2dc3fd → 1f486539 → f10a08ba → 48886c98 → 6aeaf85c → ca0c09e0 → 130372c0 → f69680b3 → 50364568 → 30e6dee3 → 3bb6d4f8 → 26a25696 → 0951d355 → 3261f321 → 5a190e74 → 80762a78 → d2bd49f3 → 99859be7 → b134cecf - `kei-replay` — 2 versions: 420ceb46 → 74f2fcc4 - `kei-router` — 2 versions: fc8c6820 → 2cfaa362 - `kei-router::kei-router` — 15 versions: 186634e6 → d91e8a11 → 80d4f8c6 → f8677f1d → a2e47f61 → 299a5afe → 675effa4 → 1fa6b4bb → 89c81c79 → 29340bbb → 51682c29 → ec0a1bfb → f4fce214 → 184e4f53 → 98ab93cd diff --git a/tests/hook-outcome-backfill-test.sh b/tests/hook-outcome-backfill-test.sh new file mode 100755 index 0000000..9f12381 --- /dev/null +++ b/tests/hook-outcome-backfill-test.sh @@ -0,0 +1,79 @@ +#!/bin/sh +# hook-outcome-backfill-test.sh — exercises agent-outcome-backfill.sh +# against a temp ledger DB. Asserts UPDATE behaviour for the 4 outcomes, +# missing-marker no-op, bypass no-op, and missing-sqlite3 no-op. +set -u + +HOOK="$HOME/.claude/hooks/agent-outcome-backfill.sh" +[ -x "$HOOK" ] || { echo "FAIL: hook not executable at $HOOK"; exit 1; } + +TMP=$(mktemp -d) +trap 'rm -rf "$TMP"' EXIT +DB="$TMP/ledger.sqlite" +export KEI_LEDGER_DB="$DB" + +sqlite3 "$DB" "CREATE TABLE agents ( + id TEXT PRIMARY KEY, + outcome TEXT CHECK (outcome IN ('functional','partial','scaffolding','fail')), + stubs_count INTEGER DEFAULT 0 +);" + +PASS=0; FAIL=0 +assert_eq() { + if [ "$1" = "$2" ]; then PASS=$((PASS+1)); + else FAIL=$((FAIL+1)); echo " FAIL: $3 — got '$1' expected '$2'"; fi +} + +run_case() { + # $1=id $2=shipped $3=stubs_count_in_marker + sqlite3 "$DB" "INSERT OR REPLACE INTO agents(id,outcome,stubs_count) VALUES('$1',NULL,0);" + BODY="prelude text +=== STATUS-TRUTH MARKER === +shipped: $2 +stubs: $3 +cargo-check: PASS +behaviour-verified: yes +follow-up-required: + - none" + PAYLOAD=$(jq -nc --arg id "$1" --arg body "$BODY" \ + '{tool_use_id:$id, tool_response:$body}') + printf '%s' "$PAYLOAD" | "$HOOK" + OUT=$(sqlite3 "$DB" "SELECT outcome||'|'||stubs_count FROM agents WHERE id='$1';") + assert_eq "$OUT" "$2|$3" "outcome=$2" +} + +echo "[1] 4 valid outcomes update correctly" +run_case "id-func" "functional" 0 +run_case "id-part" "partial" 3 +run_case "id-scaf" "scaffolding" 7 +run_case "id-fail" "fail" 12 + +echo "[2] idempotent re-run produces same row" +run_case "id-func" "functional" 0 + +echo "[3] missing marker → no-op" +sqlite3 "$DB" "INSERT OR REPLACE INTO agents(id,outcome,stubs_count) VALUES('id-bare',NULL,0);" +printf '%s' '{"tool_use_id":"id-bare","tool_response":"just a plain reply"}' | "$HOOK" +OUT=$(sqlite3 "$DB" "SELECT IFNULL(outcome,'NULL')||'|'||stubs_count FROM agents WHERE id='id-bare';") +assert_eq "$OUT" "NULL|0" "no-marker no-op" + +echo "[4] bypass env → no-op" +sqlite3 "$DB" "INSERT OR REPLACE INTO agents(id,outcome,stubs_count) VALUES('id-byp',NULL,0);" +BODY="=== STATUS-TRUTH MARKER === +shipped: functional +stubs: 0" +PAYLOAD=$(jq -nc --arg id "id-byp" --arg body "$BODY" '{tool_use_id:$id,tool_response:$body}') +printf '%s' "$PAYLOAD" | OUTCOME_BACKFILL_BYPASS=1 "$HOOK" +OUT=$(sqlite3 "$DB" "SELECT IFNULL(outcome,'NULL') FROM agents WHERE id='id-byp';") +assert_eq "$OUT" "NULL" "bypass no-op" + +echo "[5] missing sqlite3 → no-op (PATH stripped)" +sqlite3 "$DB" "INSERT OR REPLACE INTO agents(id,outcome,stubs_count) VALUES('id-nosql',NULL,0);" +PAYLOAD=$(jq -nc --arg id "id-nosql" --arg body "$BODY" '{tool_use_id:$id,tool_response:$body}') +JQ_DIR=$(dirname "$(command -v jq)") +printf '%s' "$PAYLOAD" | env -i HOME="$HOME" KEI_LEDGER_DB="$DB" PATH="$JQ_DIR" "$HOOK" 2>/dev/null +OUT=$(sqlite3 "$DB" "SELECT IFNULL(outcome,'NULL') FROM agents WHERE id='id-nosql';") +assert_eq "$OUT" "NULL" "no-sqlite3 no-op" + +echo "Passed: $PASS Failed: $FAIL" +[ "$FAIL" -eq 0 ] || exit 1