Closes the loop on "without full tracking the system can't make decisions"
(user pushback on partial coverage). Three gaps that left the inference
layer blind are now wired:
GAP #1 — agent toolStats / token counts / cache hits captured
================================================================
`agent-outcome-backfill.sh` now appends one JSONL row per spawn to
`~/.claude/memory/time-metrics/agent-toolstats.jsonl` with:
agent_id, outcome, stubs, ts,
tool_use_count, duration_ms, tool_stats {Read:N, Bash:M, ...},
tokens_in, tokens_out, cache_read, cache_write
Sidecar journal (no schema migration). Production payload's
.tool_response.totalToolUseCount / totalDurationMs / toolStats / usage
fields land directly. Smoke-tested with synthetic spawn — row written.
GAP #2 — skill_invocations table actually receives writes
================================================================
The `skill_invocations` table (schema v8) had 0 rows because no caller
existed for `skill_metrics::record_invocation`. Added two pieces:
(a) `kei-ledger record-skill <name> --success {0|1}` CLI subcommand
Mirrors record-cost; same dispatch shape. Optional `--agent-id`,
`--trajectory-id`, `--duration-ms`, `--db`. Validates non-empty
name + duration ≥ 0. Outputs `{"ok":true,"skill":"...","ts":N}`.
(b) `hooks/skill-record.sh` — PostToolUse:Skill hook. 50 LOC POSIX.
Detects Skill tool calls, derives success heuristic from
tool_response (exit_code / status / content non-empty), shells
out to `kei-ledger record-skill`. Bypass via SKILL_RECORD_BYPASS=1.
83 kei-ledger tests pass (16 unit + 67 integration). Smoke-tested
end-to-end: `kei-ledger record-skill test-skill --success 1` inserts
a row with correct fields.
Phase D nightly skill-metrics decisions (archive if unused N days,
re-extract if success<60% over M days, validated if >20 calls + >90%
success) now have data to consume.
GAP #3 — numeric-claims.jsonl receives every evidence-tagged claim
================================================================
RULE 0.18 mandated three markers `[REAL:]` / `[FROM-JOURNAL:]` /
`[ESTIMATE-HTC:]` on every numeric/duration/cost claim, but no hook
appended valid claims to the journal — the calibration data RULE 0.18
promised never accumulated.
`hooks/numeric-claims-record.sh` — Stop hook, 140 LOC POSIX. Reads
transcript_path from stdin, locates the last assistant message via
recursive flatten (same pattern as agent-outcome-backfill.sh after
the production-payload-shape fix), regex-extracts every `<phrase>
[<TIER>: <pointer>]` triple, appends one JSONL row per claim.
Idempotent within 1-second window to avoid double-recording on
repeat Stop fires. Bypass via NUMERIC_CLAIMS_RECORD_BYPASS=1.
Smoke test: synthetic transcript with 3 markers (REAL + ESTIMATE-HTC
+ FROM-JOURNAL) produced exactly 3 well-formed JSONL rows.
Settings.json
================================================================
- PostToolUse:Skill matcher created (or augmented if already
present) with skill-record.sh.
- Stop:* matcher gains numeric-claims-record.sh after the existing
chain (stop-verify, task-timer, session-end-dump, extract-task-
durations, chat-numeric-postflag, affect-threshold-check,
enrich-from-jsonl).
What this does NOT do (deferred):
- Backfill `skill_invocations` from past traces (history started
today; Phase D cohort builds forward from now).
- Migrate the agent toolStats sidecar JSONL into a proper ledger
column. Append-only file is fine for the current scale.
- Refactor main.rs (now 233 LOC, was 212; pre-existing CP debt
flagged by skill-record agent — separate cleanup PR).
=== STATUS-TRUTH MARKER ===
shipped: functional
stubs: 0
cargo-check: PASS
behaviour-verified: yes
follow-up-required:
- kei-ledger main.rs Constructor Pattern split (212→233 LOC)
- Verify in next session: skill_invocations gets rows from real
Skill tool use; numeric-claims.jsonl gets rows from real assistant
messages with markers
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hook never fired in production despite passing unit tests. Diagnosed
via debug-log + payload dump: real Claude Code PostToolUse:Agent sends
`tool_response` as an OBJECT (not string, not array), with the agent's
reply at `tool_response.content[0].text` — keys: agentId / agentType /
content / prompt / status / toolStats / totalDurationMs / totalTokens
/ totalToolUseCount / usage.
Original jq filter handled string + object (`$r.content // $r.text`)
but `$r.content` returns the array verbatim; `jq -r` then dumps the
JSON literal which has `\n` as escape sequences, defeating the
`grep -m1 '^shipped:'` line-anchor.
Fix: recursive `flatten` jq function:
string → as-is
array of any → recurse, join "\n"
object with .text → return .text
object with .content → recurse into content
anything else → ""
Verified end-to-end: latest 4 code-implementer spawns now write
outcome=functional to ledger correctly. Beta posterior in
kei-model-router begins receiving signal.
Production cleanup:
- Removed verbose debug-log + payload-dump diagnostic. Toggle via
`AGENT_OUTCOME_DEBUG=1` env if hook stops firing in some future
Claude Code version.
- Hook source committed to `hooks/agent-outcome-backfill.sh` so
`install.sh` deploys it on fresh installs (was only in user-home
previously — gap from `feat/substrate-path-atoms` agent run).
=== STATUS-TRUTH MARKER ===
shipped: functional
stubs: 0
cargo-check: NOT-RUN
behaviour-verified: yes
follow-up-required:
- none
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two parallel agents (both Sonnet 4.6 via the just-activated tier system)
extended the substrate-unified-registry. First end-to-end proof that the
Phase 4 router refactor saves money: no Opus spawns this round.
PART 1 — `kei-registry secrets` subcommand (Agent A — code-implementer)
Reads env-var NAMES from `~/.claude/secrets/.env` (RULE 0.8 SSoT) and
per-project `secrets/*.env`, greps the kit tree for usages, reports
orphans (defined but unreferenced). Live run on this kit found 26 keys,
11 ORPHAN — actionable cleanup candidates incl. GitHub OAuth client
creds, Godaddy keys, KeiGit admin creds, KEI_MEMORY_TOKEN.
Files:
- `_primitives/_rust/kei-registry/src/secrets.rs` (152 LOC) — pure
read-side cube. SecretsReport + KeyRow types, env-file parser
(KEY=value lines, validates `^[A-Z][A-Z0-9_]*$`), walkdir-based
scanner with skips (target/ node_modules/ .git/ _generated/),
word-boundary regex per key. ASCII + JSON render.
- `_primitives/_rust/kei-registry/src/secrets_tests.rs` (125 LOC) —
5 unit tests covering env parse, scan correctness, word-boundary
regression (`MY_KEY` ≠ `MY_KEY_EXTRA`), JSON roundtrip, ORPHAN marker.
- `_primitives/_rust/kei-registry/src/secrets_handler.rs` (58 LOC) —
CLI dispatch handler.
- `cli.rs`, `handlers.rs`, `lib.rs` extended with Secrets variant.
Resolves the asymmetry called out in the design discussion: paths got
atomization (commit 3422bdc), keys get a query-layer instead. Reason:
env-var NAMES are already public and stable; opaque atom-DNA over them
adds zero security and full overhead. Orphan detection is the unique
value, and a 30-LOC subcommand delivers it without a per-key atom file.
PART 2 — kei-model catalog extension (Agent B — fal-ai-runner)
Adds 10 generation-model entries with VERIFIED pricing per RULE 0.4:
- google: gemini-3-1-flash-image, gemini-3-pro-image
- fal.ai: flux-2-pro, flux-pro-1-1, kling-o3, veo-3, ideogram-v3, recraft-v3
- elevenlabs: elevenlabs-v3, elevenlabs-multilingual-v2
Pricing sourced from each provider's public pricing page (URLs cited
per row in `notes` + `source_url` fields); 8/10 verified, 2 marked
needs-verification (gemini-3-pro-image price not found on public page).
Schema additions to `_primitives/_rust/kei-model/src/model.rs` to
support the new entries without `provider = "local"` placeholder:
- Provider enum + 3 variants: Google, Fal, Elevenlabs (with as_str
+ parse impls).
- Capability enum + 9 variants: image-gen, text-to-image, image-edit,
video-gen, text-to-video, image-to-video, voice-gen, text-to-speech,
voice-clone (with serde rename + as_str + parse).
Pricing struct unchanged: per-image / per-second / per-1k-chars unit
costs ride existing `output_per_mtok_micro` field with the unit
documented in `notes` (e.g. "Per-image cost. 1 unit = 1 image."). A
proper Pricing.unit field is a follow-up.
Files:
- `_primitives/_rust/kei-model/src/model.rs` (+24 LOC enum extensions)
- `_primitives/_rust/kei-model/data/models.toml` (+216 LOC, 471 total)
`kei-model list` returns the full 21-model catalog incl. new providers.
Tests:
- kei-registry: 25 passed (existing + 5 secrets tests + 10 status)
- kei-model: 0 (no unit tests in crate, parser smoke via list)
- agent-assembler: 29 passed (no regressions)
Verification (cited):
- `./target/release/kei-registry secrets --env-file ~/.claude/secrets/.env`
emits real report 26/11 orphan.
- `./target/release/kei-model list` parses all 21 entries cleanly.
- `cargo build --release --workspace` clean.
What this does NOT do (deferred):
- Pricing.unit field (per-mtok / per-image / per-second / per-1k-chars
discriminator) — needs Rust struct refactor + cost-estimator update.
- `secrets` skip-list extension (worktrees, _ts_packages/node_modules
duplicate counts) — minor noise.
- gemini-3-pro-image pricing (no public page; vendor-specific quote
needed).
=== STATUS-TRUTH MARKER ===
shipped: functional
stubs: 0
cargo-check: PASS
behaviour-verified: yes
follow-up-required:
- Pricing.unit field for cost-estimator correctness on gen models
- secrets scan: skip .claude/worktrees/ to avoid duplicate counts
- gemini-3-pro-image price verification
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 4 follow-up: the outcome-backfill hook the kei-model-router
needs to learn from. Without an outcome signal the Beta posterior
sees 205 NULL rows and can never converge → router falls back to
top-tier on every spawn. This hook closes that loop.
Spawned by orchestrator as a code-implementer agent (Sonnet 4.6 by
default after the manifest refactor in cb1fdde — first dogfood proof
that the tier system works end-to-end). Agent returned cleanly with
STATUS-TRUTH MARKER `shipped: functional, stubs: 0`.
Files (3):
- `~/.claude/hooks/agent-outcome-backfill.sh` (73 LOC, /bin/sh) —
reads PostToolUse:Agent stdin JSON, parses STATUS-TRUTH MARKER from
`tool_response`, runs `UPDATE agents SET outcome = ?, stubs_count = ?
WHERE id = ?` via sqlite3 CLI. Defensive on every step (never blocks,
exits 0 on missing jq / sqlite3 / DB / marker). Bypass:
`OUTCOME_BACKFILL_BYPASS=1`. Lives outside the kit (system-level).
- `tests/hook-outcome-backfill-test.sh` (79 LOC, /bin/sh) — 8 assertions
cover: 4 valid outcomes, idempotent re-run, missing marker, bypass
env, missing sqlite3 (PATH stripped). Run via
`sh tests/hook-outcome-backfill-test.sh` → "Passed: 8 Failed: 0".
- `_blocks/path-user-hooks.md` — third path-atom following
user-memory / user-rules convention. Resolves to `~/.claude/hooks/`.
Lets future manifests reference hook files via
`path:user-hooks/<file>.sh` opaquely. Registered in registry as
`atom::md::331b9a34::023e5a08`.
Wiring:
- `~/.claude/settings.json` PostToolUse:Agent matcher chain — appended
the hook idempotently (jq update preserves existing
`agent-stub-scan.sh`, `task-timer.sh`, `agent-fork-done.sh`).
- DNA-INDEX regenerated; new path-atom appears in `## Atom (120)`
section.
Effect: every Agent tool call from now on writes outcome + stubs to
ledger. After ~10-20 invocations the Beta posterior has a usable
prior; after ~50 the router stops defaulting Sonnet to Opus on
unfamiliar tasks. The advisor hook (`model-router-advisor.sh`)
already prints stderr when current model > recommended — orchestrator
needs to actually pass `model:` parameter on next spawn (behavioural,
not a code change).
=== STATUS-TRUTH MARKER ===
shipped: functional
stubs: 0
cargo-check: NOT-RUN
behaviour-verified: yes
follow-up-required:
- PR feat/substrate-path-atoms-2026-05-01 → main when ready
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 1 of substrate-unified-registry: move all references to user
home memory/rules out of plain strings and into content-addressable
path atoms. Public artefacts now contain opaque `{path::NAME}/file.md`
references; the actual home prefix lives only in the path-atom file's
frontmatter, registered in the local kei-registry.
NEW path atoms (`_blocks/path-*.md`):
- `path-user-memory.md` → template `~/.claude/memory`
- `path-user-rules.md` → template `~/.claude/rules`
Both files use frontmatter `type: atom, kind: path, template: ..., expand_at: render`.
BlockMdScanner auto-registers them; DNA index shows them under their
unprefixed names (`user-memory`, `user-rules`) for human lookup, while
the body sha8 makes them content-addressable.
Resolver (`_assembler/src/registry_client.rs`):
- `is_path_atom(conn, name)` — checks DB by name + filename convention
(`_blocks/path-<name>.md`) + frontmatter `kind: path`. Defensive:
filename + frontmatter must BOTH agree.
- `frontmatter_has_kind_path(body)` — minimal YAML parser. Tolerates
CRLF, quoted values, rejects substring matches (`pathological` ≠ `path`).
- 5 unit tests cover positive + 4 negative cases.
Resolver wire-up (`_assembler/src/assembler.rs:147 write_references`):
- For each `references.extra` entry starting with `path:NAME/...`:
- Lookup `NAME` via `is_path_atom`.
- On success: emit `{path::NAME}/<suffix>` — opaque, kit-resolvable.
- On miss: stderr warn + passthrough. Never fatal.
- Non-`path:` refs pass through unchanged. Backward compatible.
- 2 unit tests cover passthrough paths.
Manifest migration (38 manifests touched):
- `~/.claude/rules/<file>` → `path:user-rules/<file>`
- `~/.claude/memory/<file>` → `path:user-memory/<file>`
- 96 references migrated; 1 prose-style reference in security-auditor
left as plain text (lives inside a domain_in description, not in
references.extra — out of scope for this resolver).
Regenerated 38 `_generated/*.md` + 1 new `frontend-validator.md`.
Regenerated `docs/DNA-INDEX.md` (now includes 2 path-atoms by name).
Verification (cited):
- `git ls-files | grep denisparfionovich` → 0 hits outside allowlist
(NOTICE/README byline + `.github/workflows/leak-check.yml` detection
rule).
- `_generated/` contains 99 occurrences of `{path::user-...}/`.
- assembler tests: 29 passed (5 new). kei-registry tests: 10 passed
(8 short_path from earlier commit + 2 unrelated).
- assembler resolver verified end-to-end: ml-implementer.md line
479-485 shows `{path::user-rules}/ml-protocol.md` etc.
What this does NOT do (deferred):
- No registry-DB schema change. Path atoms ride existing Atom block-
type via convention, not via new `BlockType::PathAtom` variant.
- No git-branch tracking (Phase 2 of plan).
- No `kei-registry status` cross-cutting CLI (Phase 3 of plan).
- No path-atom orphan detection CLI (Phase 4).
The path:user-memory and path:user-rules cover 100% of the username-
leak surface from the current manifest set; future categories
(kit-root, registry-db, sync-repo, secrets-env, project-root) can
land additively without architectural changes.
=== STATUS-TRUTH MARKER ===
shipped: functional
stubs: 0
cargo-check: PASS
behaviour-verified: yes
follow-up-required:
- Phase 2 (git-branch tracker hook)
- Phase 3 (kei-registry status subcommand)
- Phase 4 (orphan detection CLI)
- Sync user-side install: ~/.claude/agents/_manifests/ still has
pre-migration absolute paths; will pick up new format on next
`install.sh --add` (out of scope for this commit).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Root-cause of the username-path leak in DNA-INDEX.md (107 atom rows
in v0.17 — sed-patched in a23910d). The encyclopedia render's
short_path() prefix list omitted every top-level dir except
`_primitives/`, `skills/`, `hooks/`, `rules/` — so atom and capability
rows fell through to the absolute path stored in the registry DB,
leaking the maintainer's home prefix into the public encyclopedia.
Fix: add `_blocks/`, `_manifests/`, `_generated/`, `_atoms/`,
`_assembler/`, `_roles/`, `_capabilities/`, `agents/`, `docs/` to
the prefix list. 8 unit tests cover the new prefixes (fixtures use
CI-style paths like `/srv/ci/build/...` so the source file does not
contain a maintainer-shaped path that would itself trip the local
pre-commit hook + leak-check CI).
Verified: regenerated docs/DNA-INDEX.md has 0 absolute-path hits.
Source fix supersedes the sed hot-fix in a23910d — the next
`kei-registry encyclopedia` invocation will not regress.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Parfii-bot