User pushback: live-graph showed only "main" node, no pulses on agents.
Root cause: hook stdin doesn't carry parent_tool_use_id for sub-agent
tool calls — we only get the sub-agent's own session_id, which doesn't
link back to the spawn's tool_use_id.
Sequential heuristic via shared state file:
- agent-event-spawn.sh appends tool_use_id to /tmp/kei-active-children.tsv
- tool-use-event.sh reads the LAST line of that file → uses that
tool_use_id as agent_id for the emitted event
- agent-event-done.sh removes the spawn's line (grep -v + atomic mv)
Verified end-to-end: a code-implementer agent ran 5 Bash calls during
its lifetime — all 5 tool_use events were correctly attributed to the
spawn's tool_use_id. After agent_done, subsequent orchestrator-direct
tool calls correctly fall back to agent_id="main".
Limitation: parallel agents may misattribute. The "most recent live
spawn" heuristic works for single-agent-at-a-time which is the common
case. Parallel spawns share /tmp/kei-active-children.tsv and a sub-
agent's tool calls all attribute to whichever spawn appended last.
Acceptable for v1 demo; proper parent-tool-use-id propagation requires
Claude Code to expose it in sub-agent stdin (upstream change).
The `mv` after `grep -v` runs UNCONDITIONALLY (not gated on grep's
exit code) — grep -v returns 1 when ALL lines match, which would
otherwise leave the stale file in place.
Bypass: `KEI_EVENTS_BYPASS=1` (existing) covers all 3 hooks.
Override path: `KEI_ACTIVE_SPAWNS_FILE=/path/to/file`.
=== STATUS-TRUTH MARKER ===
shipped: functional
stubs: 0
cargo-check: NOT-RUN
behaviour-verified: yes
follow-up-required:
- Parallel-agent attribution would need parent_tool_use_id from
Claude Code sub-agent stdin (not currently exposed).
- Race condition window between spawn append and done remove is
millisecond-scale; observed clean in single-agent demo.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User pushback: "транслирует в онлайне какие агенты создаются? основное
окно агента, а дальше при запусках появляются новые ветки, мы показываем
в онлайне как агенты собираются и работают"
Earlier `kei-graph-export` rendered the static SUBSTRATE (all 581 atoms,
catalog-style). User wanted the LIFECYCLE: orchestrator at center, every
new agent as a fading-in branch, every tool call as a pulse, every
completion as a fade-out. TTL = until done; pure online, no history
accumulation per user direction.
Three-layer architecture, all conforming to schema /tmp/agent-events-schema.md:
LAYER 1 — Event emitters (4 hooks)
hooks/agent-event-spawn.sh PreToolUse:Agent → agent_spawn event
hooks/agent-event-done.sh PostToolUse:Agent → agent_done event
(parses STATUS-TRUTH MARKER for outcome,
computes cost_usd from token×pricing table)
hooks/tool-use-event.sh PreToolUse:Bash|Read|Edit|Write|Grep|Glob|NotebookEdit
→ tool_use event
hooks/skill-record.sh EXTENDED — second emit step writes skill_use
event in addition to existing kei-ledger
record-skill call
All 4 are POSIX /bin/sh, defensive (never block, exit 0), bypass via
KEI_EVENTS_BYPASS=1. Append-only JSONL to
~/.claude/memory/agent-events.jsonl.
Smoke: 4 synthetic invocations cover spawn/done/tool/filter cases.
LAYER 2 — kei-graph-stream Rust daemon
_primitives/_rust/kei-graph-stream/ (~480 LOC, 5 files + 1 test)
- Tails events.jsonl every 200ms (poll-based, no notify dep).
- Parses each event, updates AliveState (insert on spawn, remove on done).
- Broadcasts {"type":"event","data":<event>} to all WebSocket clients.
- On client connect: sends {"type":"snapshot","alive":[...]} first.
- Heartbeat: {"type":"ping"} every 30s.
- axum 0.7 + ws feature (already in Cargo.lock via kei-cortex).
- Bypass: KEI_GRAPH_STREAM_BYPASS=1.
Bound to 127.0.0.1:8201 (loopback only). Endpoints:
GET /stream → WebSocket upgrade
GET /health → "kei-graph-stream alive"
4 unit + 1 integration test. cargo build clean.
Installed binary: ~/.cargo/bin/kei-graph-stream
Launchd plist: io.keisei.graph-stream (RunAtLoad, KeepAlive)
Loaded as PID 52678, /health 200 OK verified.
LAYER 3 — live-graph.html (single-file frontend)
~/Projects/lbm-graph-viz/live-graph.html (~464 LOC, self-contained)
- SVG full-viewport, dark #0f172a, CSS grid background.
- Pinned center node "main" (orchestrator), gold #fbbf24, glowing.
- Agents radiate via D3 force-simulation; color-by-model
(sonnet=green, opus=red, haiku=blue, default=gray).
- On agent_spawn: fade-in 300ms, edge from main to new node.
- On tool_use: pulse on agent node (r 8→12→8 over 400ms) +
floating tool name label fades 800ms.
- On agent_done: outcome-color flash → fade-out 800ms → remove.
- WebSocket client: ws://127.0.0.1:8201/stream, exponential-backoff
reconnect (1s→30s).
- Top-right status badge: ● connected | ○ reconnecting | ✕ disconnected.
- Bottom counters: alive / spawned / tool calls / done / last event age.
- No build step. D3 v7 from CDN. Pure HTML+JS+CSS.
End-to-end smoke (this machine, just now):
- daemon health 200 OK
- hook injected agent_spawn → daemon broadcasts → AliveState=1
- hook injected agent_done → daemon broadcasts → AliveState=0
- frontend file syntax-checked clean
What this does NOT do (deferred, by user direction "это онлайн"):
- History persistence — agents who finished are GONE from the graph.
Per-session log remains in events.jsonl + sleep-sync if user wants
to consult later, but the live view is RIGHT NOW only.
- Sub-agent attribution beyond "main" — orchestrator-direct tool calls
show on the orchestrator node. Sub-agent's internal tool calls would
need session-id correlation; current schema has agent_id="main"
placeholder for non-Agent tool calls.
- Replay mode — no time-scrubber. Possible follow-up if useful.
- Auth on WebSocket — bound to 127.0.0.1 only. Local-only by design.
=== STATUS-TRUTH MARKER ===
shipped: functional
stubs: 0
cargo-check: PASS
behaviour-verified: yes
follow-up-required:
- Sub-agent tool-call attribution (correlate session_id chain)
- Replay mode with time scrubber (if user finds use)
- Tool aggregator nodes ("Bash bucket" with N) instead of per-agent pulses
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User pushback: "можно нашего Кейси подключить к обсидиан? будет в
онлайне строить граф из всех наших агентов?"
Closer-to-question architecture: don't build new Obsidian plugin —
re-use the legacy `~/Projects/lbm-graph-viz/` D3 viewer (lineage:
keicode → living-graph → lbm → lbm-graph-viz → keisei-graph). Strip
its Hebbian/co-change edges, replace with DNA-derived edges from the
kei-registry + kei-ledger. Open in any browser, file://...index.html.
NEW Rust crate `_primitives/_rust/kei-graph-export/` (~440 LOC, 5 files)
Reads:
~/.claude/registry.sqlite (730 active blocks)
~/.claude/agents/ledger.sqlite (6 agents post-cleanup)
_manifests/*.toml (38 agent manifests)
Emits 581-node, 291-edge graph. Edge types:
block_dep 171 manifest → atom (blocks=[])
path_ref 99 manifest → atom (path:NAME refs)
branch_lineage 11 parent_branch → branch
agent_uses_manifest 10 agent → manifest (slug from branch name)
Output formats:
--format spaces-fragment → `window.RUNTIME_SPACE = {...}` JS file
--format json → raw {nodes, links} for downstream tools
Block-name lookup is multi-resolution: each block is registered under
display name + lowercased + file-stem slug (from path basename) so
manifest references like `blocks = ["baseline"]` resolve to a registry
row whose `name` column holds "BASELINE — inherit from Main Claude".
Without this fix the graph had 0 block_dep edges; with it, 171.
NEW background updater `hooks/graph-export-watcher.sh` + launchd plist
template `_primitives/templates/io.keisei.graph-export.plist`
5-second loop:
while true; do
kei-graph-export --format spaces-fragment --output <viz>/data-runtime.js.tmp
mv <viz>/data-runtime.js.tmp <viz>/data-runtime.js # atomic
sleep 5
done
launchd plist substitutes `HOME_DIR` and `HOOKS_DIR` placeholders at
install time. RunAtLoad=true, KeepAlive=true. Logs to
~/.claude/memory/graph-export.log. Bypass: GRAPH_EXPORT_BYPASS=1.
Loaded into user-side launchd (PID 16474 confirmed running). File
mtime advances every 5s — live updates verified.
PATCH `~/Projects/lbm-graph-viz/index.html` (outside kit, surgical)
Three changes:
1. Add `<script src="data-runtime.js">` BEFORE `spaces.js` (window
global available when SPACES is defined).
2. After spaces.js: `if (window.RUNTIME_SPACE) SPACES.runtime = window.RUNTIME_SPACE;`
3. Auto-refresh setInterval(5s): fetch data-runtime.js, eval (re-
assigns window.RUNTIME_SPACE), hash-compare, re-render via
`rebuildGraph()` if currently viewing the runtime space.
window.RUNTIME_SPACE (not const RUNTIME_SPACE) avoids the
"const cannot be re-declared" error on subsequent eval() calls.
Effect: open file://~/Projects/lbm-graph-viz/index.html in any
browser, switch to "Runtime" space — full DNA graph of every agent /
atom / skill / branch / manifest / hook / primitive / rule, force-
laid-out by D3. Updates every 5 seconds without page reload.
What this does NOT do (deferred):
- Obsidian mirror — separate work, would emit .md per node into
~/Projects/KeiSeiVault/. Useful for backlinks navigation but
file-watcher latency similar to current 5s polling.
- Skill-invocation edges — table is empty until next Skill tool
use; will populate naturally.
- Scoped queries (orphan finder, hot-path PageRank). Out of scope
for v1; the JSON --format export feeds any downstream tool.
- `agent_uses_manifest` heuristic warns on unknown subagent slugs
(e.g. `physics-deriver` with no manifest yet). Non-fatal.
=== STATUS-TRUTH MARKER ===
shipped: functional
stubs: 0
cargo-check: PASS
behaviour-verified: yes
follow-up-required:
- Obsidian vault mirror (Phase C, separate work)
- Skill-edges populate from real Skill use (not blockered)
- Hot-path PageRank highlighting in viewer (cosmetic)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User pushback: "Агент должен делать осмысленные выводы! С утра должен
быть отчет и пусть он приходит куда-то! На телеграмм, например, лучше
сразу после фазы сна, бот есть"
Wires the @KeiSeiBot Telegram bot as the delivery channel for nightly
Phase B reports, with a Claude Sonnet 4.6 reasoning step in front to
distil the multi-section markdown into a single actionable brief.
NEW — `hooks/sleep-report-tg.sh` (130 LOC POSIX bash)
Pipeline:
1. Source ~/.claude/secrets/.env (umbrella SSoT — RULE 0.8)
2. POST report markdown to Claude API messages endpoint with a
system prompt mandating: TL;DR + numbers + 3-5 actionable
findings + rule-candidates if any cross-session pattern ≥3×.
Sonnet 4.6, max_tokens=1500, 120s timeout.
3. Send distilled summary via Telegram sendMessage to whitelisted
chat_id (defaults to TELEGRAM_ALLOWED_CHAT_ID env, falls back
to 86059912).
4. Cap message at 3900 chars (TG limit 4096).
5. Fallback if Markdown parse_mode fails (orphan * / [ in body) →
retry without parse_mode so the user still sees the report.
6. Defensive on every step: missing API key → send raw excerpt;
missing curl/jq → log + exit 0; HTTP failure → log + exit 0.
7. Bypass: SLEEP_REPORT_TG_BYPASS=1.
WIRE — `hooks/phase-b-rem.sh`
Step 7 (new) calls sleep-report-tg.sh after the existing commit/push
step. Failure of TG delivery never affects Phase B's exit code —
the local report + memory-repo push remain the source-of-truth;
TG is convenience.
CONFIG (already done outside this commit, documented for completeness)
- ~/.claude/secrets/.env now has TELEGRAM_BOT_TOKEN +
TELEGRAM_ALLOWED_CHAT_ID (single-user whitelist 86059912).
- ~/.claude/tg-webhook.py whitelist locked to {86059912}; group
chat (-1003758632751) and partner (10954083) removed per
user request "сделай боту только один вайт адрес". Blocked
senders land in /var/log/tg-webhook/blocked.jsonl, no auto-reply.
- ~/.claude/tg-contacts.json shrunk from 3 contacts to 1.
Smoke verified: today's sleep-2026-05-02.md → cloud agent emitted
TL;DR ("Opus burned $1239 across 117 runs with 100% unknown outcomes")
+ 5 findings + 3 rule-candidates → delivered to chat_id 86059912 as
msg_id 1129 (HTTP 200). Cost: 3955 in + 897 out tokens on Sonnet
≈ $0.025/run. At 1 run/night that is ~$0.75/month for full reasoning
on every nightly report.
What this does NOT yet do:
- No retry on Telegram rate-limit (429). Single nightly call
is well below the 30/sec limit, but if the system ever bursts
multiple reports it would lose them.
- No multi-day digest mode (each run is independent; future:
weekly Sunday recap aggregating 7 reports).
- Cloud agent prompt is hard-coded inline; future: extract to
a path-atom-style block (post-2026-05-02 substrate work).
=== STATUS-TRUTH MARKER ===
shipped: functional
stubs: 0
cargo-check: NOT-RUN (pure shell)
behaviour-verified: yes
follow-up-required:
- Phase B prompt template extracted to atom (low priority)
- Weekly recap mode (Sunday)
- 429 rate-limit retry (defensive)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User pushback: "что теперь делает сон? все связано?" — Sleep Phase B
was reading only `traces/`, ignoring the four tracking journals shipped
in the previous commit. Cloud agent had a partial view of what happened.
This commit closes the loop. Sleep now sees everything that's tracked.
PUSH SIDE — `kei-sleep-sync.sh` (called on every Stop event)
Now mirrors the full observability surface into the memory-repo:
~/.claude/memory/time-metrics/sessions.jsonl → time-metrics/
~/.claude/memory/time-metrics/tasks.jsonl → time-metrics/
~/.claude/memory/time-metrics/numeric-claims.jsonl → time-metrics/
~/.claude/memory/time-metrics/agent-toolstats.jsonl→ time-metrics/
~/.claude/agents/ledger.sqlite agents table → ledger/agents.jsonl
~/.claude/agents/ledger.sqlite skill_invocations → ledger/skill_invocations.jsonl
Format: JSONL (one row per object). The two ledger tables are dumped
via `sqlite3 + json_object()` so cloud agents can stream-parse into
pandas / duckdb without binary-file handling.
First sync moved 6 files / 638 rows from local to remote — verified
by `git show --stat` of the resulting `memory: session traces` commit.
CONSUME SIDE — `phase-b-rem.sh` REM-consolidation report
Each nightly `reports/sleep-YYYY-MM-DD.md` now ends with a "Tracking
observability (last 7 days)" section containing four jq-aggregated
digests:
1. Agent outcomes — per-model: n, functional/partial/scaffolding/fail
counts + total_cost_usd. Lets the agent see whether the model-tier
refactor (50c9e76) actually paid off and whether Sonnet success
rate justifies routing more task classes to it.
2. Skill success rates — per-skill: n, successes, rate_pct. Drives
Phase D nightly decisions (archive unused / re-extract failing /
mark validated). Empty until Skill tool is invoked in the next
session.
3. Numeric-claims tier breakdown — REAL / FROM-JOURNAL / ESTIMATE-HTC
counts. High ESTIMATE-HTC ratio = orchestrator under-calibrated.
Cloud agent's job: spot frequent ESTIMATE-HTC categories and
propose conversion to FROM-JOURNAL via measured runs.
4. Agent tool-call patterns — mean tool_use_count, mean duration_ms,
per-tool total calls. Lets the agent see "this code-implementer
spawn made 30 Read but 1 Edit — was tier-allocation correct?".
All four sections gracefully skip if the source JSONL is missing or
empty. jq is the only new dependency (already present per existing
phase-b checks).
What is NOT yet automated:
- The cloud agent's prompt template doesn't yet INSTRUCT it to act
on these digests. Currently the digest is data; whether the agent
proposes rule + hook codification based on it depends on the
free-text instructions in the schedule. Follow-up: codify a Phase B
instruction block that maps each digest to a recommendation pattern.
- Idempotency on `cp` for time-metrics: I use plain `cp` (not `cp -n`)
so the latest local state always overwrites remote. The journals are
append-only on the local side, so this is safe — but if two machines
ever share one memory-repo it would corrupt. Out of scope for
single-machine setup.
=== STATUS-TRUTH MARKER ===
shipped: functional
stubs: 0
cargo-check: NOT-RUN (pure shell)
behaviour-verified: yes
follow-up-required:
- Phase B prompt template — instruct cloud agent to act on the four
digests (codify recurring patterns, calibrate ESTIMATE-HTC).
- skill_invocations.jsonl will populate from next session onward.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the loop on "without full tracking the system can't make decisions"
(user pushback on partial coverage). Three gaps that left the inference
layer blind are now wired:
GAP #1 — agent toolStats / token counts / cache hits captured
================================================================
`agent-outcome-backfill.sh` now appends one JSONL row per spawn to
`~/.claude/memory/time-metrics/agent-toolstats.jsonl` with:
agent_id, outcome, stubs, ts,
tool_use_count, duration_ms, tool_stats {Read:N, Bash:M, ...},
tokens_in, tokens_out, cache_read, cache_write
Sidecar journal (no schema migration). Production payload's
.tool_response.totalToolUseCount / totalDurationMs / toolStats / usage
fields land directly. Smoke-tested with synthetic spawn — row written.
GAP #2 — skill_invocations table actually receives writes
================================================================
The `skill_invocations` table (schema v8) had 0 rows because no caller
existed for `skill_metrics::record_invocation`. Added two pieces:
(a) `kei-ledger record-skill <name> --success {0|1}` CLI subcommand
Mirrors record-cost; same dispatch shape. Optional `--agent-id`,
`--trajectory-id`, `--duration-ms`, `--db`. Validates non-empty
name + duration ≥ 0. Outputs `{"ok":true,"skill":"...","ts":N}`.
(b) `hooks/skill-record.sh` — PostToolUse:Skill hook. 50 LOC POSIX.
Detects Skill tool calls, derives success heuristic from
tool_response (exit_code / status / content non-empty), shells
out to `kei-ledger record-skill`. Bypass via SKILL_RECORD_BYPASS=1.
83 kei-ledger tests pass (16 unit + 67 integration). Smoke-tested
end-to-end: `kei-ledger record-skill test-skill --success 1` inserts
a row with correct fields.
Phase D nightly skill-metrics decisions (archive if unused N days,
re-extract if success<60% over M days, validated if >20 calls + >90%
success) now have data to consume.
GAP #3 — numeric-claims.jsonl receives every evidence-tagged claim
================================================================
RULE 0.18 mandated three markers `[REAL:]` / `[FROM-JOURNAL:]` /
`[ESTIMATE-HTC:]` on every numeric/duration/cost claim, but no hook
appended valid claims to the journal — the calibration data RULE 0.18
promised never accumulated.
`hooks/numeric-claims-record.sh` — Stop hook, 140 LOC POSIX. Reads
transcript_path from stdin, locates the last assistant message via
recursive flatten (same pattern as agent-outcome-backfill.sh after
the production-payload-shape fix), regex-extracts every `<phrase>
[<TIER>: <pointer>]` triple, appends one JSONL row per claim.
Idempotent within 1-second window to avoid double-recording on
repeat Stop fires. Bypass via NUMERIC_CLAIMS_RECORD_BYPASS=1.
Smoke test: synthetic transcript with 3 markers (REAL + ESTIMATE-HTC
+ FROM-JOURNAL) produced exactly 3 well-formed JSONL rows.
Settings.json
================================================================
- PostToolUse:Skill matcher created (or augmented if already
present) with skill-record.sh.
- Stop:* matcher gains numeric-claims-record.sh after the existing
chain (stop-verify, task-timer, session-end-dump, extract-task-
durations, chat-numeric-postflag, affect-threshold-check,
enrich-from-jsonl).
What this does NOT do (deferred):
- Backfill `skill_invocations` from past traces (history started
today; Phase D cohort builds forward from now).
- Migrate the agent toolStats sidecar JSONL into a proper ledger
column. Append-only file is fine for the current scale.
- Refactor main.rs (now 233 LOC, was 212; pre-existing CP debt
flagged by skill-record agent — separate cleanup PR).
=== STATUS-TRUTH MARKER ===
shipped: functional
stubs: 0
cargo-check: PASS
behaviour-verified: yes
follow-up-required:
- kei-ledger main.rs Constructor Pattern split (212→233 LOC)
- Verify in next session: skill_invocations gets rows from real
Skill tool use; numeric-claims.jsonl gets rows from real assistant
messages with markers
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hook never fired in production despite passing unit tests. Diagnosed
via debug-log + payload dump: real Claude Code PostToolUse:Agent sends
`tool_response` as an OBJECT (not string, not array), with the agent's
reply at `tool_response.content[0].text` — keys: agentId / agentType /
content / prompt / status / toolStats / totalDurationMs / totalTokens
/ totalToolUseCount / usage.
Original jq filter handled string + object (`$r.content // $r.text`)
but `$r.content` returns the array verbatim; `jq -r` then dumps the
JSON literal which has `\n` as escape sequences, defeating the
`grep -m1 '^shipped:'` line-anchor.
Fix: recursive `flatten` jq function:
string → as-is
array of any → recurse, join "\n"
object with .text → return .text
object with .content → recurse into content
anything else → ""
Verified end-to-end: latest 4 code-implementer spawns now write
outcome=functional to ledger correctly. Beta posterior in
kei-model-router begins receiving signal.
Production cleanup:
- Removed verbose debug-log + payload-dump diagnostic. Toggle via
`AGENT_OUTCOME_DEBUG=1` env if hook stops firing in some future
Claude Code version.
- Hook source committed to `hooks/agent-outcome-backfill.sh` so
`install.sh` deploys it on fresh installs (was only in user-home
previously — gap from `feat/substrate-path-atoms` agent run).
=== STATUS-TRUTH MARKER ===
shipped: functional
stubs: 0
cargo-check: NOT-RUN
behaviour-verified: yes
follow-up-required:
- none
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Parfii-bot