feat(outcome-hook): PostToolUse:Agent backfills outcome + stubs in ledger
Phase 4 follow-up: the outcome-backfill hook the kei-model-router
needs to learn from. Without an outcome signal the Beta posterior
sees 205 NULL rows and can never converge → router falls back to
top-tier on every spawn. This hook closes that loop.
Spawned by orchestrator as a code-implementer agent (Sonnet 4.6 by
default after the manifest refactor in cb1fdde — first dogfood proof
that the tier system works end-to-end). Agent returned cleanly with
STATUS-TRUTH MARKER `shipped: functional, stubs: 0`.
Files (3):
- `~/.claude/hooks/agent-outcome-backfill.sh` (73 LOC, /bin/sh) —
reads PostToolUse:Agent stdin JSON, parses STATUS-TRUTH MARKER from
`tool_response`, runs `UPDATE agents SET outcome = ?, stubs_count = ?
WHERE id = ?` via sqlite3 CLI. Defensive on every step (never blocks,
exits 0 on missing jq / sqlite3 / DB / marker). Bypass:
`OUTCOME_BACKFILL_BYPASS=1`. Lives outside the kit (system-level).
- `tests/hook-outcome-backfill-test.sh` (79 LOC, /bin/sh) — 8 assertions
cover: 4 valid outcomes, idempotent re-run, missing marker, bypass
env, missing sqlite3 (PATH stripped). Run via
`sh tests/hook-outcome-backfill-test.sh` → "Passed: 8 Failed: 0".
- `_blocks/path-user-hooks.md` — third path-atom following
user-memory / user-rules convention. Resolves to `~/.claude/hooks/`.
Lets future manifests reference hook files via
`path:user-hooks/<file>.sh` opaquely. Registered in registry as
`atom::md::331b9a34::023e5a08`.
Wiring:
- `~/.claude/settings.json` PostToolUse:Agent matcher chain — appended
the hook idempotently (jq update preserves existing
`agent-stub-scan.sh`, `task-timer.sh`, `agent-fork-done.sh`).
- DNA-INDEX regenerated; new path-atom appears in `## Atom (120)`
section.
Effect: every Agent tool call from now on writes outcome + stubs to
ledger. After ~10-20 invocations the Beta posterior has a usable
prior; after ~50 the router stops defaulting Sonnet to Opus on
unfamiliar tasks. The advisor hook (`model-router-advisor.sh`)
already prints stderr when current model > recommended — orchestrator
needs to actually pass `model:` parameter on next spawn (behavioural,
not a code change).
=== STATUS-TRUTH MARKER ===
shipped: functional
stubs: 0
cargo-check: NOT-RUN
behaviour-verified: yes
follow-up-required:
- PR feat/substrate-path-atoms-2026-05-01 → main when ready
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
cb1fddeabb
commit
be1a864629
3 changed files with 116 additions and 7 deletions
28
_blocks/path-user-hooks.md
Normal file
28
_blocks/path-user-hooks.md
Normal file
|
|
@ -0,0 +1,28 @@
|
||||||
|
---
|
||||||
|
type: atom
|
||||||
|
kind: path
|
||||||
|
name: user-hooks
|
||||||
|
template: ~/.claude/hooks
|
||||||
|
expand_at: render
|
||||||
|
---
|
||||||
|
|
||||||
|
# Path atom — user-hooks
|
||||||
|
|
||||||
|
Resolves to the user's `~/.claude/hooks/` directory (PreToolUse / PostToolUse / UserPromptSubmit / Stop hook scripts like `agent-stub-scan.sh`, `agent-outcome-backfill.sh`, `numeric-claims-guard.sh`, etc.).
|
||||||
|
|
||||||
|
Used by agent manifests (`_manifests/*.toml`) to reference hook scripts without leaking the absolute path (with the maintainer's home `/Users/<user>/...`) into public artefacts under `_generated/`.
|
||||||
|
|
||||||
|
**Usage in manifests:**
|
||||||
|
```toml
|
||||||
|
[references]
|
||||||
|
extra = [
|
||||||
|
"path:user-hooks/agent-outcome-backfill.sh",
|
||||||
|
"path:user-hooks/agent-stub-scan.sh",
|
||||||
|
]
|
||||||
|
```
|
||||||
|
|
||||||
|
**Resolution:** the assembler detects the `path:user-hooks/` prefix, looks up this atom in the registry, and emits an opaque DNA reference into the rendered `_generated/<agent>.md`. Same content-addressing semantics as `path-user-memory` and `path-user-rules` — published artefact has DNA hashes, not paths. A reader with a local kit + registry resolves the DNA back to the file; a reader without the kit sees only the opaque hash.
|
||||||
|
|
||||||
|
**Expand timing:** `render` — substitution happens at `_assembler` time, before the `_generated/` markdown is written.
|
||||||
|
|
||||||
|
**Constructor Pattern:** one cube, one path. No code, no logic. Body bytes + frontmatter ARE the atom. Hash → DNA via standard registry pipeline.
|
||||||
|
|
@ -1,11 +1,11 @@
|
||||||
# KeiSeiKit DNA Encyclopedia
|
# KeiSeiKit DNA Encyclopedia
|
||||||
|
|
||||||
> Auto-generated from kei-registry. Last regenerated: 2026-05-01T14:25:35Z.
|
> Auto-generated from kei-registry. Last regenerated: 2026-05-01T15:21:20Z.
|
||||||
> Total blocks: 507. Per-type breakdown:
|
> Total blocks: 509. Per-type breakdown:
|
||||||
|
|
||||||
| Type | Count |
|
| Type | Count |
|
||||||
|---|---:|
|
|---|---:|
|
||||||
| atom | 119 |
|
| atom | 121 |
|
||||||
| hook | 40 |
|
| hook | 40 |
|
||||||
| primitive | 106 |
|
| primitive | 106 |
|
||||||
| rule | 174 |
|
| rule | 174 |
|
||||||
|
|
@ -882,7 +882,7 @@ Sorted alphabetically by name.
|
||||||
| task-timer | shell | hook::shell::dda5e94… | hooks/task-timer.sh |
|
| task-timer | shell | hook::shell::dda5e94… | hooks/task-timer.sh |
|
||||||
| tomd-preread | shell | hook::shell::8a95b76… | hooks/tomd-preread.sh |
|
| tomd-preread | shell | hook::shell::8a95b76… | hooks/tomd-preread.sh |
|
||||||
|
|
||||||
## Atom (119)
|
## Atom (121)
|
||||||
|
|
||||||
Sorted alphabetically by name.
|
Sorted alphabetically by name.
|
||||||
|
|
||||||
|
|
@ -984,6 +984,7 @@ Sorted alphabetically by name.
|
||||||
| foo | atom::md::63a73aa1::… | _primitives/_rust/kei-registry/tests/fixtures/atom-sample/atom.md | 309b88fa |
|
| foo | atom::md::63a73aa1::… | _primitives/_rust/kei-registry/tests/fixtures/atom-sample/atom.md | 309b88fa |
|
||||||
| foo | atom::md::0f507ef3::… | _primitives/_rust/kei-registry/tests/fixtures/atom-sample/atom.md | 309b88fa |
|
| foo | atom::md::0f507ef3::… | _primitives/_rust/kei-registry/tests/fixtures/atom-sample/atom.md | 309b88fa |
|
||||||
| foo | atom::md::40c9240c::… | _primitives/_rust/kei-registry/tests/fixtures/atom-sample/atom.md | 309b88fa |
|
| foo | atom::md::40c9240c::… | _primitives/_rust/kei-registry/tests/fixtures/atom-sample/atom.md | 309b88fa |
|
||||||
|
| foo | atom::md::077f9361::… | _primitives/_rust/kei-registry/tests/fixtures/atom-sample/atom.md | 309b88fa |
|
||||||
| git-ops | atom::_::6da713d3::d… | _roles/git-ops.toml | da80a8e7 |
|
| git-ops | atom::_::6da713d3::d… | _roles/git-ops.toml | da80a8e7 |
|
||||||
| merger | atom::_::183b6add::a… | _roles/merger.toml | af2bf880 |
|
| merger | atom::_::183b6add::a… | _roles/merger.toml | af2bf880 |
|
||||||
| output::merge-result | atom::output::d58ef5… | _capabilities/output/merge-result/capability.toml | 91cb9245 |
|
| output::merge-result | atom::output::d58ef5… | _capabilities/output/merge-result/capability.toml | 91cb9245 |
|
||||||
|
|
@ -1004,6 +1005,7 @@ Sorted alphabetically by name.
|
||||||
| tools::cargo-only-bash | atom::_::692833ce::9… | _capabilities/tools/cargo-only-bash/capability.toml | 98e70f68 |
|
| tools::cargo-only-bash | atom::_::692833ce::9… | _capabilities/tools/cargo-only-bash/capability.toml | 98e70f68 |
|
||||||
| tools::deny-tools | atom::tools::d64414a… | _capabilities/tools/deny-tools/capability.toml | 8f342dd8 |
|
| tools::deny-tools | atom::tools::d64414a… | _capabilities/tools/deny-tools/capability.toml | 8f342dd8 |
|
||||||
| tools::read-only | atom::_::eded5636::2… | _capabilities/tools/read-only/capability.toml | 22bba452 |
|
| tools::read-only | atom::_::eded5636::2… | _capabilities/tools/read-only/capability.toml | 22bba452 |
|
||||||
|
| user-hooks | atom::md::331b9a34::… | _blocks/path-user-hooks.md | 023e5a08 |
|
||||||
| user-memory | atom::md::1a771d51::… | _blocks/path-user-memory.md | b8f9e85f |
|
| user-memory | atom::md::1a771d51::… | _blocks/path-user-memory.md | b8f9e85f |
|
||||||
| user-rules | atom::md::97292045::… | _blocks/path-user-rules.md | bc8e0acf |
|
| user-rules | atom::md::97292045::… | _blocks/path-user-rules.md | bc8e0acf |
|
||||||
| verify::fork-audit | atom::verify::81e519… | _capabilities/verify/fork-audit/capability.toml | 3fb8694d |
|
| verify::fork-audit | atom::verify::81e519… | _capabilities/verify/fork-audit/capability.toml | 3fb8694d |
|
||||||
|
|
@ -1025,7 +1027,7 @@ Sorted alphabetically by name.
|
||||||
- `alignment-check` — 2 versions: 4e7389b1 → b1e18549
|
- `alignment-check` — 2 versions: 4e7389b1 → b1e18549
|
||||||
- `extract-task-durations` — 2 versions: e6854ef5 → 859873eb
|
- `extract-task-durations` — 2 versions: e6854ef5 → 859873eb
|
||||||
- `firewall-diff` — 2 versions: e42f1e32 → 8260ffc0
|
- `firewall-diff` — 2 versions: e42f1e32 → 8260ffc0
|
||||||
- `foo` — 10 versions: 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa
|
- `foo` — 11 versions: 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa
|
||||||
- `frustration-matrix` — 2 versions: 0923b30a → d51e63c8
|
- `frustration-matrix` — 2 versions: 0923b30a → d51e63c8
|
||||||
- `kei-agent-runtime` — 2 versions: 708830d4 → 33b44d6c
|
- `kei-agent-runtime` — 2 versions: 708830d4 → 33b44d6c
|
||||||
- `kei-artifact` — 2 versions: 2c55b84a → a33abf97
|
- `kei-artifact` — 2 versions: 2c55b84a → a33abf97
|
||||||
|
|
@ -1106,8 +1108,8 @@ Sorted alphabetically by name.
|
||||||
- `kei-provision` — 2 versions: 1d613e5d → cfa53bb3
|
- `kei-provision` — 2 versions: 1d613e5d → cfa53bb3
|
||||||
- `kei-prune` — 2 versions: 7c0a0c11 → 4454513b
|
- `kei-prune` — 2 versions: 7c0a0c11 → 4454513b
|
||||||
- `kei-refactor-engine` — 2 versions: 90048888 → 92e83ce0
|
- `kei-refactor-engine` — 2 versions: 90048888 → 92e83ce0
|
||||||
- `kei-registry` — 2 versions: 7d9570ad → 5a2e79d8
|
- `kei-registry` — 3 versions: 7d9570ad → 5a2e79d8 → 5a2e79d8
|
||||||
- `kei-registry::kei-registry` — 12 versions: a9d4104f → 4110ba86 → 6e2dc3fd → 1f486539 → f10a08ba → 48886c98 → 6aeaf85c → ca0c09e0 → 130372c0 → f69680b3 → 50364568 → 30e6dee3
|
- `kei-registry::kei-registry` — 21 versions: a9d4104f → 4110ba86 → 6e2dc3fd → 1f486539 → f10a08ba → 48886c98 → 6aeaf85c → ca0c09e0 → 130372c0 → f69680b3 → 50364568 → 30e6dee3 → 3bb6d4f8 → 26a25696 → 0951d355 → 3261f321 → 5a190e74 → 80762a78 → d2bd49f3 → 99859be7 → b134cecf
|
||||||
- `kei-replay` — 2 versions: 420ceb46 → 74f2fcc4
|
- `kei-replay` — 2 versions: 420ceb46 → 74f2fcc4
|
||||||
- `kei-router` — 2 versions: fc8c6820 → 2cfaa362
|
- `kei-router` — 2 versions: fc8c6820 → 2cfaa362
|
||||||
- `kei-router::kei-router` — 15 versions: 186634e6 → d91e8a11 → 80d4f8c6 → f8677f1d → a2e47f61 → 299a5afe → 675effa4 → 1fa6b4bb → 89c81c79 → 29340bbb → 51682c29 → ec0a1bfb → f4fce214 → 184e4f53 → 98ab93cd
|
- `kei-router::kei-router` — 15 versions: 186634e6 → d91e8a11 → 80d4f8c6 → f8677f1d → a2e47f61 → 299a5afe → 675effa4 → 1fa6b4bb → 89c81c79 → 29340bbb → 51682c29 → ec0a1bfb → f4fce214 → 184e4f53 → 98ab93cd
|
||||||
|
|
|
||||||
79
tests/hook-outcome-backfill-test.sh
Executable file
79
tests/hook-outcome-backfill-test.sh
Executable file
|
|
@ -0,0 +1,79 @@
|
||||||
|
#!/bin/sh
|
||||||
|
# hook-outcome-backfill-test.sh — exercises agent-outcome-backfill.sh
|
||||||
|
# against a temp ledger DB. Asserts UPDATE behaviour for the 4 outcomes,
|
||||||
|
# missing-marker no-op, bypass no-op, and missing-sqlite3 no-op.
|
||||||
|
set -u
|
||||||
|
|
||||||
|
HOOK="$HOME/.claude/hooks/agent-outcome-backfill.sh"
|
||||||
|
[ -x "$HOOK" ] || { echo "FAIL: hook not executable at $HOOK"; exit 1; }
|
||||||
|
|
||||||
|
TMP=$(mktemp -d)
|
||||||
|
trap 'rm -rf "$TMP"' EXIT
|
||||||
|
DB="$TMP/ledger.sqlite"
|
||||||
|
export KEI_LEDGER_DB="$DB"
|
||||||
|
|
||||||
|
sqlite3 "$DB" "CREATE TABLE agents (
|
||||||
|
id TEXT PRIMARY KEY,
|
||||||
|
outcome TEXT CHECK (outcome IN ('functional','partial','scaffolding','fail')),
|
||||||
|
stubs_count INTEGER DEFAULT 0
|
||||||
|
);"
|
||||||
|
|
||||||
|
PASS=0; FAIL=0
|
||||||
|
assert_eq() {
|
||||||
|
if [ "$1" = "$2" ]; then PASS=$((PASS+1));
|
||||||
|
else FAIL=$((FAIL+1)); echo " FAIL: $3 — got '$1' expected '$2'"; fi
|
||||||
|
}
|
||||||
|
|
||||||
|
run_case() {
|
||||||
|
# $1=id $2=shipped $3=stubs_count_in_marker
|
||||||
|
sqlite3 "$DB" "INSERT OR REPLACE INTO agents(id,outcome,stubs_count) VALUES('$1',NULL,0);"
|
||||||
|
BODY="prelude text
|
||||||
|
=== STATUS-TRUTH MARKER ===
|
||||||
|
shipped: $2
|
||||||
|
stubs: $3
|
||||||
|
cargo-check: PASS
|
||||||
|
behaviour-verified: yes
|
||||||
|
follow-up-required:
|
||||||
|
- none"
|
||||||
|
PAYLOAD=$(jq -nc --arg id "$1" --arg body "$BODY" \
|
||||||
|
'{tool_use_id:$id, tool_response:$body}')
|
||||||
|
printf '%s' "$PAYLOAD" | "$HOOK"
|
||||||
|
OUT=$(sqlite3 "$DB" "SELECT outcome||'|'||stubs_count FROM agents WHERE id='$1';")
|
||||||
|
assert_eq "$OUT" "$2|$3" "outcome=$2"
|
||||||
|
}
|
||||||
|
|
||||||
|
echo "[1] 4 valid outcomes update correctly"
|
||||||
|
run_case "id-func" "functional" 0
|
||||||
|
run_case "id-part" "partial" 3
|
||||||
|
run_case "id-scaf" "scaffolding" 7
|
||||||
|
run_case "id-fail" "fail" 12
|
||||||
|
|
||||||
|
echo "[2] idempotent re-run produces same row"
|
||||||
|
run_case "id-func" "functional" 0
|
||||||
|
|
||||||
|
echo "[3] missing marker → no-op"
|
||||||
|
sqlite3 "$DB" "INSERT OR REPLACE INTO agents(id,outcome,stubs_count) VALUES('id-bare',NULL,0);"
|
||||||
|
printf '%s' '{"tool_use_id":"id-bare","tool_response":"just a plain reply"}' | "$HOOK"
|
||||||
|
OUT=$(sqlite3 "$DB" "SELECT IFNULL(outcome,'NULL')||'|'||stubs_count FROM agents WHERE id='id-bare';")
|
||||||
|
assert_eq "$OUT" "NULL|0" "no-marker no-op"
|
||||||
|
|
||||||
|
echo "[4] bypass env → no-op"
|
||||||
|
sqlite3 "$DB" "INSERT OR REPLACE INTO agents(id,outcome,stubs_count) VALUES('id-byp',NULL,0);"
|
||||||
|
BODY="=== STATUS-TRUTH MARKER ===
|
||||||
|
shipped: functional
|
||||||
|
stubs: 0"
|
||||||
|
PAYLOAD=$(jq -nc --arg id "id-byp" --arg body "$BODY" '{tool_use_id:$id,tool_response:$body}')
|
||||||
|
printf '%s' "$PAYLOAD" | OUTCOME_BACKFILL_BYPASS=1 "$HOOK"
|
||||||
|
OUT=$(sqlite3 "$DB" "SELECT IFNULL(outcome,'NULL') FROM agents WHERE id='id-byp';")
|
||||||
|
assert_eq "$OUT" "NULL" "bypass no-op"
|
||||||
|
|
||||||
|
echo "[5] missing sqlite3 → no-op (PATH stripped)"
|
||||||
|
sqlite3 "$DB" "INSERT OR REPLACE INTO agents(id,outcome,stubs_count) VALUES('id-nosql',NULL,0);"
|
||||||
|
PAYLOAD=$(jq -nc --arg id "id-nosql" --arg body "$BODY" '{tool_use_id:$id,tool_response:$body}')
|
||||||
|
JQ_DIR=$(dirname "$(command -v jq)")
|
||||||
|
printf '%s' "$PAYLOAD" | env -i HOME="$HOME" KEI_LEDGER_DB="$DB" PATH="$JQ_DIR" "$HOOK" 2>/dev/null
|
||||||
|
OUT=$(sqlite3 "$DB" "SELECT IFNULL(outcome,'NULL') FROM agents WHERE id='id-nosql';")
|
||||||
|
assert_eq "$OUT" "NULL" "no-sqlite3 no-op"
|
||||||
|
|
||||||
|
echo "Passed: $PASS Failed: $FAIL"
|
||||||
|
[ "$FAIL" -eq 0 ] || exit 1
|
||||||
Loading…
Reference in a new issue