feat(outcome-hook): PostToolUse:Agent backfills outcome + stubs in ledger
Phase 4 follow-up: the outcome-backfill hook the kei-model-router
needs to learn from. Without an outcome signal the Beta posterior
sees 205 NULL rows and can never converge → router falls back to
top-tier on every spawn. This hook closes that loop.
Spawned by orchestrator as a code-implementer agent (Sonnet 4.6 by
default after the manifest refactor in cb1fdde — first dogfood proof
that the tier system works end-to-end). Agent returned cleanly with
STATUS-TRUTH MARKER `shipped: functional, stubs: 0`.
Files (3):
- `~/.claude/hooks/agent-outcome-backfill.sh` (73 LOC, /bin/sh) —
reads PostToolUse:Agent stdin JSON, parses STATUS-TRUTH MARKER from
`tool_response`, runs `UPDATE agents SET outcome = ?, stubs_count = ?
WHERE id = ?` via sqlite3 CLI. Defensive on every step (never blocks,
exits 0 on missing jq / sqlite3 / DB / marker). Bypass:
`OUTCOME_BACKFILL_BYPASS=1`. Lives outside the kit (system-level).
- `tests/hook-outcome-backfill-test.sh` (79 LOC, /bin/sh) — 8 assertions
cover: 4 valid outcomes, idempotent re-run, missing marker, bypass
env, missing sqlite3 (PATH stripped). Run via
`sh tests/hook-outcome-backfill-test.sh` → "Passed: 8 Failed: 0".
- `_blocks/path-user-hooks.md` — third path-atom following
user-memory / user-rules convention. Resolves to `~/.claude/hooks/`.
Lets future manifests reference hook files via
`path:user-hooks/<file>.sh` opaquely. Registered in registry as
`atom::md::331b9a34::023e5a08`.
Wiring:
- `~/.claude/settings.json` PostToolUse:Agent matcher chain — appended
the hook idempotently (jq update preserves existing
`agent-stub-scan.sh`, `task-timer.sh`, `agent-fork-done.sh`).
- DNA-INDEX regenerated; new path-atom appears in `## Atom (120)`
section.
Effect: every Agent tool call from now on writes outcome + stubs to
ledger. After ~10-20 invocations the Beta posterior has a usable
prior; after ~50 the router stops defaulting Sonnet to Opus on
unfamiliar tasks. The advisor hook (`model-router-advisor.sh`)
already prints stderr when current model > recommended — orchestrator
needs to actually pass `model:` parameter on next spawn (behavioural,
not a code change).
=== STATUS-TRUTH MARKER ===
shipped: functional
stubs: 0
cargo-check: NOT-RUN
behaviour-verified: yes
follow-up-required:
- PR feat/substrate-path-atoms-2026-05-01 → main when ready
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
cb1fddeabb
commit
be1a864629
3 changed files with 116 additions and 7 deletions
28
_blocks/path-user-hooks.md
Normal file
28
_blocks/path-user-hooks.md
Normal file
|
|
@ -0,0 +1,28 @@
|
|||
---
|
||||
type: atom
|
||||
kind: path
|
||||
name: user-hooks
|
||||
template: ~/.claude/hooks
|
||||
expand_at: render
|
||||
---
|
||||
|
||||
# Path atom — user-hooks
|
||||
|
||||
Resolves to the user's `~/.claude/hooks/` directory (PreToolUse / PostToolUse / UserPromptSubmit / Stop hook scripts like `agent-stub-scan.sh`, `agent-outcome-backfill.sh`, `numeric-claims-guard.sh`, etc.).
|
||||
|
||||
Used by agent manifests (`_manifests/*.toml`) to reference hook scripts without leaking the absolute path (with the maintainer's home `/Users/<user>/...`) into public artefacts under `_generated/`.
|
||||
|
||||
**Usage in manifests:**
|
||||
```toml
|
||||
[references]
|
||||
extra = [
|
||||
"path:user-hooks/agent-outcome-backfill.sh",
|
||||
"path:user-hooks/agent-stub-scan.sh",
|
||||
]
|
||||
```
|
||||
|
||||
**Resolution:** the assembler detects the `path:user-hooks/` prefix, looks up this atom in the registry, and emits an opaque DNA reference into the rendered `_generated/<agent>.md`. Same content-addressing semantics as `path-user-memory` and `path-user-rules` — published artefact has DNA hashes, not paths. A reader with a local kit + registry resolves the DNA back to the file; a reader without the kit sees only the opaque hash.
|
||||
|
||||
**Expand timing:** `render` — substitution happens at `_assembler` time, before the `_generated/` markdown is written.
|
||||
|
||||
**Constructor Pattern:** one cube, one path. No code, no logic. Body bytes + frontmatter ARE the atom. Hash → DNA via standard registry pipeline.
|
||||
|
|
@ -1,11 +1,11 @@
|
|||
# KeiSeiKit DNA Encyclopedia
|
||||
|
||||
> Auto-generated from kei-registry. Last regenerated: 2026-05-01T14:25:35Z.
|
||||
> Total blocks: 507. Per-type breakdown:
|
||||
> Auto-generated from kei-registry. Last regenerated: 2026-05-01T15:21:20Z.
|
||||
> Total blocks: 509. Per-type breakdown:
|
||||
|
||||
| Type | Count |
|
||||
|---|---:|
|
||||
| atom | 119 |
|
||||
| atom | 121 |
|
||||
| hook | 40 |
|
||||
| primitive | 106 |
|
||||
| rule | 174 |
|
||||
|
|
@ -882,7 +882,7 @@ Sorted alphabetically by name.
|
|||
| task-timer | shell | hook::shell::dda5e94… | hooks/task-timer.sh |
|
||||
| tomd-preread | shell | hook::shell::8a95b76… | hooks/tomd-preread.sh |
|
||||
|
||||
## Atom (119)
|
||||
## Atom (121)
|
||||
|
||||
Sorted alphabetically by name.
|
||||
|
||||
|
|
@ -984,6 +984,7 @@ Sorted alphabetically by name.
|
|||
| foo | atom::md::63a73aa1::… | _primitives/_rust/kei-registry/tests/fixtures/atom-sample/atom.md | 309b88fa |
|
||||
| foo | atom::md::0f507ef3::… | _primitives/_rust/kei-registry/tests/fixtures/atom-sample/atom.md | 309b88fa |
|
||||
| foo | atom::md::40c9240c::… | _primitives/_rust/kei-registry/tests/fixtures/atom-sample/atom.md | 309b88fa |
|
||||
| foo | atom::md::077f9361::… | _primitives/_rust/kei-registry/tests/fixtures/atom-sample/atom.md | 309b88fa |
|
||||
| git-ops | atom::_::6da713d3::d… | _roles/git-ops.toml | da80a8e7 |
|
||||
| merger | atom::_::183b6add::a… | _roles/merger.toml | af2bf880 |
|
||||
| output::merge-result | atom::output::d58ef5… | _capabilities/output/merge-result/capability.toml | 91cb9245 |
|
||||
|
|
@ -1004,6 +1005,7 @@ Sorted alphabetically by name.
|
|||
| tools::cargo-only-bash | atom::_::692833ce::9… | _capabilities/tools/cargo-only-bash/capability.toml | 98e70f68 |
|
||||
| tools::deny-tools | atom::tools::d64414a… | _capabilities/tools/deny-tools/capability.toml | 8f342dd8 |
|
||||
| tools::read-only | atom::_::eded5636::2… | _capabilities/tools/read-only/capability.toml | 22bba452 |
|
||||
| user-hooks | atom::md::331b9a34::… | _blocks/path-user-hooks.md | 023e5a08 |
|
||||
| user-memory | atom::md::1a771d51::… | _blocks/path-user-memory.md | b8f9e85f |
|
||||
| user-rules | atom::md::97292045::… | _blocks/path-user-rules.md | bc8e0acf |
|
||||
| verify::fork-audit | atom::verify::81e519… | _capabilities/verify/fork-audit/capability.toml | 3fb8694d |
|
||||
|
|
@ -1025,7 +1027,7 @@ Sorted alphabetically by name.
|
|||
- `alignment-check` — 2 versions: 4e7389b1 → b1e18549
|
||||
- `extract-task-durations` — 2 versions: e6854ef5 → 859873eb
|
||||
- `firewall-diff` — 2 versions: e42f1e32 → 8260ffc0
|
||||
- `foo` — 10 versions: 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa
|
||||
- `foo` — 11 versions: 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa → 309b88fa
|
||||
- `frustration-matrix` — 2 versions: 0923b30a → d51e63c8
|
||||
- `kei-agent-runtime` — 2 versions: 708830d4 → 33b44d6c
|
||||
- `kei-artifact` — 2 versions: 2c55b84a → a33abf97
|
||||
|
|
@ -1106,8 +1108,8 @@ Sorted alphabetically by name.
|
|||
- `kei-provision` — 2 versions: 1d613e5d → cfa53bb3
|
||||
- `kei-prune` — 2 versions: 7c0a0c11 → 4454513b
|
||||
- `kei-refactor-engine` — 2 versions: 90048888 → 92e83ce0
|
||||
- `kei-registry` — 2 versions: 7d9570ad → 5a2e79d8
|
||||
- `kei-registry::kei-registry` — 12 versions: a9d4104f → 4110ba86 → 6e2dc3fd → 1f486539 → f10a08ba → 48886c98 → 6aeaf85c → ca0c09e0 → 130372c0 → f69680b3 → 50364568 → 30e6dee3
|
||||
- `kei-registry` — 3 versions: 7d9570ad → 5a2e79d8 → 5a2e79d8
|
||||
- `kei-registry::kei-registry` — 21 versions: a9d4104f → 4110ba86 → 6e2dc3fd → 1f486539 → f10a08ba → 48886c98 → 6aeaf85c → ca0c09e0 → 130372c0 → f69680b3 → 50364568 → 30e6dee3 → 3bb6d4f8 → 26a25696 → 0951d355 → 3261f321 → 5a190e74 → 80762a78 → d2bd49f3 → 99859be7 → b134cecf
|
||||
- `kei-replay` — 2 versions: 420ceb46 → 74f2fcc4
|
||||
- `kei-router` — 2 versions: fc8c6820 → 2cfaa362
|
||||
- `kei-router::kei-router` — 15 versions: 186634e6 → d91e8a11 → 80d4f8c6 → f8677f1d → a2e47f61 → 299a5afe → 675effa4 → 1fa6b4bb → 89c81c79 → 29340bbb → 51682c29 → ec0a1bfb → f4fce214 → 184e4f53 → 98ab93cd
|
||||
|
|
|
|||
79
tests/hook-outcome-backfill-test.sh
Executable file
79
tests/hook-outcome-backfill-test.sh
Executable file
|
|
@ -0,0 +1,79 @@
|
|||
#!/bin/sh
|
||||
# hook-outcome-backfill-test.sh — exercises agent-outcome-backfill.sh
|
||||
# against a temp ledger DB. Asserts UPDATE behaviour for the 4 outcomes,
|
||||
# missing-marker no-op, bypass no-op, and missing-sqlite3 no-op.
|
||||
set -u
|
||||
|
||||
HOOK="$HOME/.claude/hooks/agent-outcome-backfill.sh"
|
||||
[ -x "$HOOK" ] || { echo "FAIL: hook not executable at $HOOK"; exit 1; }
|
||||
|
||||
TMP=$(mktemp -d)
|
||||
trap 'rm -rf "$TMP"' EXIT
|
||||
DB="$TMP/ledger.sqlite"
|
||||
export KEI_LEDGER_DB="$DB"
|
||||
|
||||
sqlite3 "$DB" "CREATE TABLE agents (
|
||||
id TEXT PRIMARY KEY,
|
||||
outcome TEXT CHECK (outcome IN ('functional','partial','scaffolding','fail')),
|
||||
stubs_count INTEGER DEFAULT 0
|
||||
);"
|
||||
|
||||
PASS=0; FAIL=0
|
||||
assert_eq() {
|
||||
if [ "$1" = "$2" ]; then PASS=$((PASS+1));
|
||||
else FAIL=$((FAIL+1)); echo " FAIL: $3 — got '$1' expected '$2'"; fi
|
||||
}
|
||||
|
||||
run_case() {
|
||||
# $1=id $2=shipped $3=stubs_count_in_marker
|
||||
sqlite3 "$DB" "INSERT OR REPLACE INTO agents(id,outcome,stubs_count) VALUES('$1',NULL,0);"
|
||||
BODY="prelude text
|
||||
=== STATUS-TRUTH MARKER ===
|
||||
shipped: $2
|
||||
stubs: $3
|
||||
cargo-check: PASS
|
||||
behaviour-verified: yes
|
||||
follow-up-required:
|
||||
- none"
|
||||
PAYLOAD=$(jq -nc --arg id "$1" --arg body "$BODY" \
|
||||
'{tool_use_id:$id, tool_response:$body}')
|
||||
printf '%s' "$PAYLOAD" | "$HOOK"
|
||||
OUT=$(sqlite3 "$DB" "SELECT outcome||'|'||stubs_count FROM agents WHERE id='$1';")
|
||||
assert_eq "$OUT" "$2|$3" "outcome=$2"
|
||||
}
|
||||
|
||||
echo "[1] 4 valid outcomes update correctly"
|
||||
run_case "id-func" "functional" 0
|
||||
run_case "id-part" "partial" 3
|
||||
run_case "id-scaf" "scaffolding" 7
|
||||
run_case "id-fail" "fail" 12
|
||||
|
||||
echo "[2] idempotent re-run produces same row"
|
||||
run_case "id-func" "functional" 0
|
||||
|
||||
echo "[3] missing marker → no-op"
|
||||
sqlite3 "$DB" "INSERT OR REPLACE INTO agents(id,outcome,stubs_count) VALUES('id-bare',NULL,0);"
|
||||
printf '%s' '{"tool_use_id":"id-bare","tool_response":"just a plain reply"}' | "$HOOK"
|
||||
OUT=$(sqlite3 "$DB" "SELECT IFNULL(outcome,'NULL')||'|'||stubs_count FROM agents WHERE id='id-bare';")
|
||||
assert_eq "$OUT" "NULL|0" "no-marker no-op"
|
||||
|
||||
echo "[4] bypass env → no-op"
|
||||
sqlite3 "$DB" "INSERT OR REPLACE INTO agents(id,outcome,stubs_count) VALUES('id-byp',NULL,0);"
|
||||
BODY="=== STATUS-TRUTH MARKER ===
|
||||
shipped: functional
|
||||
stubs: 0"
|
||||
PAYLOAD=$(jq -nc --arg id "id-byp" --arg body "$BODY" '{tool_use_id:$id,tool_response:$body}')
|
||||
printf '%s' "$PAYLOAD" | OUTCOME_BACKFILL_BYPASS=1 "$HOOK"
|
||||
OUT=$(sqlite3 "$DB" "SELECT IFNULL(outcome,'NULL') FROM agents WHERE id='id-byp';")
|
||||
assert_eq "$OUT" "NULL" "bypass no-op"
|
||||
|
||||
echo "[5] missing sqlite3 → no-op (PATH stripped)"
|
||||
sqlite3 "$DB" "INSERT OR REPLACE INTO agents(id,outcome,stubs_count) VALUES('id-nosql',NULL,0);"
|
||||
PAYLOAD=$(jq -nc --arg id "id-nosql" --arg body "$BODY" '{tool_use_id:$id,tool_response:$body}')
|
||||
JQ_DIR=$(dirname "$(command -v jq)")
|
||||
printf '%s' "$PAYLOAD" | env -i HOME="$HOME" KEI_LEDGER_DB="$DB" PATH="$JQ_DIR" "$HOOK" 2>/dev/null
|
||||
OUT=$(sqlite3 "$DB" "SELECT IFNULL(outcome,'NULL') FROM agents WHERE id='id-nosql';")
|
||||
assert_eq "$OUT" "NULL" "no-sqlite3 no-op"
|
||||
|
||||
echo "Passed: $PASS Failed: $FAIL"
|
||||
[ "$FAIL" -eq 0 ] || exit 1
|
||||
Loading…
Reference in a new issue