Commit graph

16 commits

Author SHA1 Message Date
Parfii-bot
6b66914f97 feat(install): outcome-only minimum profile
Reviewer suggested an evaluation footprint that lands "the smallest
substrate any caller-LLM can use", with 5 files and ~200 LOC ceiling
in $HOME. This commit ships that profile.

Files installed in $HOME by `./install.sh --profile=outcome-only`:
1. ~/.claude/hooks/agent-outcome-backfill.sh   (PostToolUse:Agent)
2. ~/.claude/hooks/error-spike-detector.sh     (PostToolUse:Bash, rolling 20-call window)
3. ~/.claude/agents/ledger.sqlite              (full v9 schema via kei-ledger init, or sqlite3-fallback DDL)
4. ~/.claude/CLAUDE.md                         (1-line STATUS-TRUTH MARKER instruction appended)
5. ~/.claude/settings.json                     (jq-merge of 2 hook entries)

Plus optional 6th: kei-model-router binary built from _primitives/_rust if
cargo on PATH; deferred otherwise (warning printed, install continues).

Files added to repo:
- install/lib-profile-outcome-only.sh (145 LOC) — profile orchestrator with
  --dry-run support; sources lib-log/lib-backup/lib-hooks helpers; exits
  before heavy install phases when --profile=outcome-only
- install/sql/outcome-only-schema.sql (69 LOC) — flattened v9-equivalent
  SQLite DDL (agents + skill_invocations + indexes), used by sqlite3
  fallback when kei-ledger CLI is unavailable
- docs/PROFILE-OUTCOME-ONLY.md (97 LOC) — reviewer-facing doc: 5-file
  install table, what is NOT installed, kei-model-router activation
  explanation, privacy posture (no telemetry), 4-line uninstall paste

Files modified:
- install.sh (+12 LOC) — sources outcome-only lib, adds short-circuit
  before menu when --profile=outcome-only, accepts in profile validator
- install/lib-args.sh (+9 LOC) — registers --dry-run flag (sets
  OUTCOME_DRY_RUN=1), adds outcome-only + --dry-run lines to --help
- README.md (+7 LOC) — adds Outcome-only Quick-start section pointing to
  PROFILE-OUTCOME-ONLY.md

Verification:
- bash -n clean on all 3 modified shell files
- Dry-run produces exactly 5 numbered $HOME paths (verified end-to-end:
  HOME=/tmp/kei-fake-home bash install.sh --profile=outcome-only --dry-run)
- Real install against fake $HOME succeeds (5 files present, ledger init
  via kei-ledger binary, router build correctly skipped on toolchain
  absence with warning)
- Ledger schema includes agents + skill_invocations tables + 3 indexes
  + 2 triggers via real migration path (not the SQL fallback)

[FROM-JOURNAL: end-to-end install dry-run + real-run measured at
~/.claude/memory/time-metrics/sessions.jsonl this session, both <2s wall]

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 16:59:53 +08:00
Parfii-bot
d2068cded7 docs: reviewer-response — honesty pass + portable format specs
External reviewer raised 7 overclaim/scope concerns. Agents verified each
against source; this commit applies all fixes that landed in docs.

Honesty pass:
- README:25-29 — Cortex daemon track listed as alpha (was beta); MCP server
  marked "alpha (unpublished) — install via local dist build"; Phase B
  noted "auto-codification not yet wired (manual via /escalate-recurrence)";
  keigit framed as author-operated mirror (KeiSei84 / private Forgejo),
  not neutral community service
- README:95-97 — Cortex CLI/daemon track downgraded beta→alpha
  with rationale (browser-app + VSCode-extension are concept-level)
- docs/ARCHITECTURE.md — added "Model router — current state (2026-05-03)"
  subsection: per-call fixed estimate routing, NO 100-row Bayesian threshold
  in current source (select.rs:74-124); reviewer suggestion deferred
- docs/SLEEP-LAYER.md — added Phase B scope clarification: morning report
  is read-only markdown, no auto-codification path
- docs/PUBLISHING.md — aligned framing with README:43 ("author-operated
  mirror" not "community registry"); added vendor-neutrality note that
  substrate works against any npm-compatible registry
- mcp-server/package.json — added "private": true and description note
  to prevent accidental publish before maturity gate

Portable format specs (reviewer asked for memory-repo agnosticism):
- docs/MEMORY-FORMAT.md (196 LOC) — JSONL schemas for traces / decisions /
  agent-events with jq/awk/pandas recipes, grounded in actual writers
- docs/DNA-FORMAT.md (159 LOC) — DNA wire format ("type::caps::sha8")
  with shell+python parsers
- docs/LEDGER-SCHEMA.md (199 LOC) — full SQLite DDL (agents +
  skill_invocations + indexes + triggers) with sample queries

Auto-regen artifact:
- docs/DNA-INDEX.md — kei-registry regenerated count 564→565

Verification:
- All claims traced to file:line in source by agent a52b29ae
- All new docs ≤200 LOC per Constructor Pattern
- Reality verification verdicts: README/MCP/Phase-B/Cortex VERIFIED;
  Bayesian-router PARTIAL (overclaim removed); keigit PARTIAL (framing
  fixed in this commit); memory-format VERIFIED-FALSE (spec added)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 16:59:25 +08:00
Parfii-bot
036bc6a52e docs: SKILL.md triggers + STATUS-TRUTH footer + phase placeholders
Group G — markdown tech-debt cleanup (post-audit 2026-05-02).

- 36 SKILL.md files: added "## When to use" section. Was missing across the
  catalog; orchestrator routing by keyword could not auto-dispatch.

- 20 code-implementer agent .md files: added Output Footer block prescribing
  RULE 0.16 STATUS-TRUTH MARKER schema in agent's final report. Previously only
  code-implementer-rust.md had it; other 27 language/role variants were silent
  about the marker, breaking RULE 0.16 §3 status-truth aggregation for non-Rust
  batches.

- skills/site-create/: added phase-5-preview.md and phase-6-deploy.md skeleton
  files. SKILL.md table-of-contents referenced 7 phases; only 5 existed on disk.

- skills/{ai-animation,rag-pipeline}/skill.md: added migration banner comment
  noting they should be SKILL.md (canonical filename). Case-rename via git is a
  separate orchestrator task (macOS APFS is case-insensitive; Linux deploy needs
  explicit rename).

- 3 deprecated skills (site-builder, competitor-analysis, design-inspiration):
  added concrete removed-after dates (was vague "before v2").

- docs/CONVERGENCE-PLAN.md:129: TBD on _blocks/evidence-grading.md duplicate
  resolved (file exists, not duplicated).

- docs/DNA-INDEX.md: count edits made then overwritten by auto-encyclopedia-refresh
  hook during agent run. The .kei-registry-ignore files in test fixtures (Group F)
  are the structural fix; kei-registry walker implementation is the follow-up.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 21:41:41 +08:00
Parfii-bot
52a02dfbff feat(live-graph): WebSocket activity stream — orchestrator-centric live view
User pushback: "транслирует в онлайне какие агенты создаются? основное
окно агента, а дальше при запусках появляются новые ветки, мы показываем
в онлайне как агенты собираются и работают"

Earlier `kei-graph-export` rendered the static SUBSTRATE (all 581 atoms,
catalog-style). User wanted the LIFECYCLE: orchestrator at center, every
new agent as a fading-in branch, every tool call as a pulse, every
completion as a fade-out. TTL = until done; pure online, no history
accumulation per user direction.

Three-layer architecture, all conforming to schema /tmp/agent-events-schema.md:

LAYER 1 — Event emitters (4 hooks)
  hooks/agent-event-spawn.sh   PreToolUse:Agent  → agent_spawn event
  hooks/agent-event-done.sh    PostToolUse:Agent → agent_done event
                               (parses STATUS-TRUTH MARKER for outcome,
                                computes cost_usd from token×pricing table)
  hooks/tool-use-event.sh      PreToolUse:Bash|Read|Edit|Write|Grep|Glob|NotebookEdit
                               → tool_use event
  hooks/skill-record.sh        EXTENDED — second emit step writes skill_use
                               event in addition to existing kei-ledger
                               record-skill call

  All 4 are POSIX /bin/sh, defensive (never block, exit 0), bypass via
  KEI_EVENTS_BYPASS=1. Append-only JSONL to
  ~/.claude/memory/agent-events.jsonl.

  Smoke: 4 synthetic invocations cover spawn/done/tool/filter cases.

LAYER 2 — kei-graph-stream Rust daemon
  _primitives/_rust/kei-graph-stream/  (~480 LOC, 5 files + 1 test)

  - Tails events.jsonl every 200ms (poll-based, no notify dep).
  - Parses each event, updates AliveState (insert on spawn, remove on done).
  - Broadcasts {"type":"event","data":<event>} to all WebSocket clients.
  - On client connect: sends {"type":"snapshot","alive":[...]} first.
  - Heartbeat: {"type":"ping"} every 30s.
  - axum 0.7 + ws feature (already in Cargo.lock via kei-cortex).
  - Bypass: KEI_GRAPH_STREAM_BYPASS=1.

  Bound to 127.0.0.1:8201 (loopback only). Endpoints:
    GET /stream  → WebSocket upgrade
    GET /health  → "kei-graph-stream alive"

  4 unit + 1 integration test. cargo build clean.

  Installed binary: ~/.cargo/bin/kei-graph-stream
  Launchd plist: io.keisei.graph-stream (RunAtLoad, KeepAlive)
  Loaded as PID 52678, /health 200 OK verified.

LAYER 3 — live-graph.html (single-file frontend)
  ~/Projects/lbm-graph-viz/live-graph.html  (~464 LOC, self-contained)

  - SVG full-viewport, dark #0f172a, CSS grid background.
  - Pinned center node "main" (orchestrator), gold #fbbf24, glowing.
  - Agents radiate via D3 force-simulation; color-by-model
    (sonnet=green, opus=red, haiku=blue, default=gray).
  - On agent_spawn: fade-in 300ms, edge from main to new node.
  - On tool_use: pulse on agent node (r 8→12→8 over 400ms) +
    floating tool name label fades 800ms.
  - On agent_done: outcome-color flash → fade-out 800ms → remove.
  - WebSocket client: ws://127.0.0.1:8201/stream, exponential-backoff
    reconnect (1s→30s).
  - Top-right status badge: ● connected | ○ reconnecting | ✕ disconnected.
  - Bottom counters: alive / spawned / tool calls / done / last event age.
  - No build step. D3 v7 from CDN. Pure HTML+JS+CSS.

End-to-end smoke (this machine, just now):
  - daemon health 200 OK
  - hook injected agent_spawn → daemon broadcasts → AliveState=1
  - hook injected agent_done  → daemon broadcasts → AliveState=0
  - frontend file syntax-checked clean

What this does NOT do (deferred, by user direction "это онлайн"):
  - History persistence — agents who finished are GONE from the graph.
    Per-session log remains in events.jsonl + sleep-sync if user wants
    to consult later, but the live view is RIGHT NOW only.
  - Sub-agent attribution beyond "main" — orchestrator-direct tool calls
    show on the orchestrator node. Sub-agent's internal tool calls would
    need session-id correlation; current schema has agent_id="main"
    placeholder for non-Agent tool calls.
  - Replay mode — no time-scrubber. Possible follow-up if useful.
  - Auth on WebSocket — bound to 127.0.0.1 only. Local-only by design.

=== STATUS-TRUTH MARKER ===
shipped: functional
stubs: 0
cargo-check: PASS
behaviour-verified: yes
follow-up-required:
  - Sub-agent tool-call attribution (correlate session_id chain)
  - Replay mode with time scrubber (if user finds use)
  - Tool aggregator nodes ("Bash bucket" with N) instead of per-agent pulses

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 13:30:24 +08:00
Parfii-bot
a31a056f61 feat(graph): live runtime DNA viewer — kei-graph-export + lbm-graph-viz adapter
User pushback: "можно нашего Кейси подключить к обсидиан? будет в
онлайне строить граф из всех наших агентов?"

Closer-to-question architecture: don't build new Obsidian plugin —
re-use the legacy `~/Projects/lbm-graph-viz/` D3 viewer (lineage:
keicode → living-graph → lbm → lbm-graph-viz → keisei-graph). Strip
its Hebbian/co-change edges, replace with DNA-derived edges from the
kei-registry + kei-ledger. Open in any browser, file://...index.html.

NEW Rust crate `_primitives/_rust/kei-graph-export/` (~440 LOC, 5 files)

Reads:
  ~/.claude/registry.sqlite       (730 active blocks)
  ~/.claude/agents/ledger.sqlite  (6 agents post-cleanup)
  _manifests/*.toml               (38 agent manifests)

Emits 581-node, 291-edge graph. Edge types:
  block_dep        171  manifest → atom (blocks=[])
  path_ref          99  manifest → atom (path:NAME refs)
  branch_lineage    11  parent_branch → branch
  agent_uses_manifest 10  agent → manifest (slug from branch name)

Output formats:
  --format spaces-fragment  →  `window.RUNTIME_SPACE = {...}` JS file
  --format json             →  raw {nodes, links} for downstream tools

Block-name lookup is multi-resolution: each block is registered under
display name + lowercased + file-stem slug (from path basename) so
manifest references like `blocks = ["baseline"]` resolve to a registry
row whose `name` column holds "BASELINE — inherit from Main Claude".
Without this fix the graph had 0 block_dep edges; with it, 171.

NEW background updater `hooks/graph-export-watcher.sh` + launchd plist
template `_primitives/templates/io.keisei.graph-export.plist`

5-second loop:
  while true; do
      kei-graph-export --format spaces-fragment --output <viz>/data-runtime.js.tmp
      mv <viz>/data-runtime.js.tmp <viz>/data-runtime.js  # atomic
      sleep 5
  done

launchd plist substitutes `HOME_DIR` and `HOOKS_DIR` placeholders at
install time. RunAtLoad=true, KeepAlive=true. Logs to
~/.claude/memory/graph-export.log. Bypass: GRAPH_EXPORT_BYPASS=1.

Loaded into user-side launchd (PID 16474 confirmed running). File
mtime advances every 5s — live updates verified.

PATCH `~/Projects/lbm-graph-viz/index.html` (outside kit, surgical)

Three changes:
  1. Add `<script src="data-runtime.js">` BEFORE `spaces.js` (window
     global available when SPACES is defined).
  2. After spaces.js: `if (window.RUNTIME_SPACE) SPACES.runtime = window.RUNTIME_SPACE;`
  3. Auto-refresh setInterval(5s): fetch data-runtime.js, eval (re-
     assigns window.RUNTIME_SPACE), hash-compare, re-render via
     `rebuildGraph()` if currently viewing the runtime space.

window.RUNTIME_SPACE (not const RUNTIME_SPACE) avoids the
"const cannot be re-declared" error on subsequent eval() calls.

Effect: open file://~/Projects/lbm-graph-viz/index.html in any
browser, switch to "Runtime" space — full DNA graph of every agent /
atom / skill / branch / manifest / hook / primitive / rule, force-
laid-out by D3. Updates every 5 seconds without page reload.

What this does NOT do (deferred):
  - Obsidian mirror — separate work, would emit .md per node into
    ~/Projects/KeiSeiVault/. Useful for backlinks navigation but
    file-watcher latency similar to current 5s polling.
  - Skill-invocation edges — table is empty until next Skill tool
    use; will populate naturally.
  - Scoped queries (orphan finder, hot-path PageRank). Out of scope
    for v1; the JSON --format export feeds any downstream tool.
  - `agent_uses_manifest` heuristic warns on unknown subagent slugs
    (e.g. `physics-deriver` with no manifest yet). Non-fatal.

=== STATUS-TRUTH MARKER ===
shipped: functional
stubs: 0
cargo-check: PASS
behaviour-verified: yes
follow-up-required:
  - Obsidian vault mirror (Phase C, separate work)
  - Skill-edges populate from real Skill use (not blockered)
  - Hot-path PageRank highlighting in viewer (cosmetic)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 13:07:21 +08:00
Parfii-bot
e073df6c98 feat(tracking): close 3 last observability gaps — toolStats + skill-record + numeric-claims journal
Closes the loop on "without full tracking the system can't make decisions"
(user pushback on partial coverage). Three gaps that left the inference
layer blind are now wired:

GAP #1 — agent toolStats / token counts / cache hits captured
================================================================
`agent-outcome-backfill.sh` now appends one JSONL row per spawn to
`~/.claude/memory/time-metrics/agent-toolstats.jsonl` with:
  agent_id, outcome, stubs, ts,
  tool_use_count, duration_ms, tool_stats {Read:N, Bash:M, ...},
  tokens_in, tokens_out, cache_read, cache_write
Sidecar journal (no schema migration). Production payload's
.tool_response.totalToolUseCount / totalDurationMs / toolStats / usage
fields land directly. Smoke-tested with synthetic spawn — row written.

GAP #2 — skill_invocations table actually receives writes
================================================================
The `skill_invocations` table (schema v8) had 0 rows because no caller
existed for `skill_metrics::record_invocation`. Added two pieces:

(a) `kei-ledger record-skill <name> --success {0|1}` CLI subcommand
    Mirrors record-cost; same dispatch shape. Optional `--agent-id`,
    `--trajectory-id`, `--duration-ms`, `--db`. Validates non-empty
    name + duration ≥ 0. Outputs `{"ok":true,"skill":"...","ts":N}`.

(b) `hooks/skill-record.sh` — PostToolUse:Skill hook. 50 LOC POSIX.
    Detects Skill tool calls, derives success heuristic from
    tool_response (exit_code / status / content non-empty), shells
    out to `kei-ledger record-skill`. Bypass via SKILL_RECORD_BYPASS=1.

83 kei-ledger tests pass (16 unit + 67 integration). Smoke-tested
end-to-end: `kei-ledger record-skill test-skill --success 1` inserts
a row with correct fields.

Phase D nightly skill-metrics decisions (archive if unused N days,
re-extract if success<60% over M days, validated if >20 calls + >90%
success) now have data to consume.

GAP #3 — numeric-claims.jsonl receives every evidence-tagged claim
================================================================
RULE 0.18 mandated three markers `[REAL:]` / `[FROM-JOURNAL:]` /
`[ESTIMATE-HTC:]` on every numeric/duration/cost claim, but no hook
appended valid claims to the journal — the calibration data RULE 0.18
promised never accumulated.

`hooks/numeric-claims-record.sh` — Stop hook, 140 LOC POSIX. Reads
transcript_path from stdin, locates the last assistant message via
recursive flatten (same pattern as agent-outcome-backfill.sh after
the production-payload-shape fix), regex-extracts every `<phrase>
[<TIER>: <pointer>]` triple, appends one JSONL row per claim.

Idempotent within 1-second window to avoid double-recording on
repeat Stop fires. Bypass via NUMERIC_CLAIMS_RECORD_BYPASS=1.

Smoke test: synthetic transcript with 3 markers (REAL + ESTIMATE-HTC
+ FROM-JOURNAL) produced exactly 3 well-formed JSONL rows.

Settings.json
================================================================
- PostToolUse:Skill matcher created (or augmented if already
  present) with skill-record.sh.
- Stop:* matcher gains numeric-claims-record.sh after the existing
  chain (stop-verify, task-timer, session-end-dump, extract-task-
  durations, chat-numeric-postflag, affect-threshold-check,
  enrich-from-jsonl).

What this does NOT do (deferred):
  - Backfill `skill_invocations` from past traces (history started
    today; Phase D cohort builds forward from now).
  - Migrate the agent toolStats sidecar JSONL into a proper ledger
    column. Append-only file is fine for the current scale.
  - Refactor main.rs (now 233 LOC, was 212; pre-existing CP debt
    flagged by skill-record agent — separate cleanup PR).

=== STATUS-TRUTH MARKER ===
shipped: functional
stubs: 0
cargo-check: PASS
behaviour-verified: yes
follow-up-required:
  - kei-ledger main.rs Constructor Pattern split (212→233 LOC)
  - Verify in next session: skill_invocations gets rows from real
    Skill tool use; numeric-claims.jsonl gets rows from real assistant
    messages with markers

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 03:42:09 +08:00
Parfii-bot
033b9efbad fix(outcome-hook): production payload uses object.content[*].text shape
Hook never fired in production despite passing unit tests. Diagnosed
via debug-log + payload dump: real Claude Code PostToolUse:Agent sends
`tool_response` as an OBJECT (not string, not array), with the agent's
reply at `tool_response.content[0].text` — keys: agentId / agentType /
content / prompt / status / toolStats / totalDurationMs / totalTokens
/ totalToolUseCount / usage.

Original jq filter handled string + object (`$r.content // $r.text`)
but `$r.content` returns the array verbatim; `jq -r` then dumps the
JSON literal which has `\n` as escape sequences, defeating the
`grep -m1 '^shipped:'` line-anchor.

Fix: recursive `flatten` jq function:
  string                     → as-is
  array of any               → recurse, join "\n"
  object with .text          → return .text
  object with .content       → recurse into content
  anything else              → ""

Verified end-to-end: latest 4 code-implementer spawns now write
outcome=functional to ledger correctly. Beta posterior in
kei-model-router begins receiving signal.

Production cleanup:
- Removed verbose debug-log + payload-dump diagnostic. Toggle via
  `AGENT_OUTCOME_DEBUG=1` env if hook stops firing in some future
  Claude Code version.
- Hook source committed to `hooks/agent-outcome-backfill.sh` so
  `install.sh` deploys it on fresh installs (was only in user-home
  previously — gap from `feat/substrate-path-atoms` agent run).

=== STATUS-TRUTH MARKER ===
shipped: functional
stubs: 0
cargo-check: NOT-RUN
behaviour-verified: yes
follow-up-required:
  - none

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 01:09:15 +08:00
Parfii-bot
af46684330 feat(secrets+catalog): orphan-detector for env vars + image/video/voice models
Two parallel agents (both Sonnet 4.6 via the just-activated tier system)
extended the substrate-unified-registry. First end-to-end proof that the
Phase 4 router refactor saves money: no Opus spawns this round.

PART 1 — `kei-registry secrets` subcommand (Agent A — code-implementer)

Reads env-var NAMES from `~/.claude/secrets/.env` (RULE 0.8 SSoT) and
per-project `secrets/*.env`, greps the kit tree for usages, reports
orphans (defined but unreferenced). Live run on this kit found 26 keys,
11 ORPHAN — actionable cleanup candidates incl. GitHub OAuth client
creds, Godaddy keys, KeiGit admin creds, KEI_MEMORY_TOKEN.

Files:
- `_primitives/_rust/kei-registry/src/secrets.rs` (152 LOC) — pure
  read-side cube. SecretsReport + KeyRow types, env-file parser
  (KEY=value lines, validates `^[A-Z][A-Z0-9_]*$`), walkdir-based
  scanner with skips (target/ node_modules/ .git/ _generated/),
  word-boundary regex per key. ASCII + JSON render.
- `_primitives/_rust/kei-registry/src/secrets_tests.rs` (125 LOC) —
  5 unit tests covering env parse, scan correctness, word-boundary
  regression (`MY_KEY` ≠ `MY_KEY_EXTRA`), JSON roundtrip, ORPHAN marker.
- `_primitives/_rust/kei-registry/src/secrets_handler.rs` (58 LOC) —
  CLI dispatch handler.
- `cli.rs`, `handlers.rs`, `lib.rs` extended with Secrets variant.

Resolves the asymmetry called out in the design discussion: paths got
atomization (commit 3422bdc), keys get a query-layer instead. Reason:
env-var NAMES are already public and stable; opaque atom-DNA over them
adds zero security and full overhead. Orphan detection is the unique
value, and a 30-LOC subcommand delivers it without a per-key atom file.

PART 2 — kei-model catalog extension (Agent B — fal-ai-runner)

Adds 10 generation-model entries with VERIFIED pricing per RULE 0.4:
- google: gemini-3-1-flash-image, gemini-3-pro-image
- fal.ai: flux-2-pro, flux-pro-1-1, kling-o3, veo-3, ideogram-v3, recraft-v3
- elevenlabs: elevenlabs-v3, elevenlabs-multilingual-v2

Pricing sourced from each provider's public pricing page (URLs cited
per row in `notes` + `source_url` fields); 8/10 verified, 2 marked
needs-verification (gemini-3-pro-image price not found on public page).

Schema additions to `_primitives/_rust/kei-model/src/model.rs` to
support the new entries without `provider = "local"` placeholder:
- Provider enum + 3 variants: Google, Fal, Elevenlabs (with as_str
  + parse impls).
- Capability enum + 9 variants: image-gen, text-to-image, image-edit,
  video-gen, text-to-video, image-to-video, voice-gen, text-to-speech,
  voice-clone (with serde rename + as_str + parse).

Pricing struct unchanged: per-image / per-second / per-1k-chars unit
costs ride existing `output_per_mtok_micro` field with the unit
documented in `notes` (e.g. "Per-image cost. 1 unit = 1 image."). A
proper Pricing.unit field is a follow-up.

Files:
- `_primitives/_rust/kei-model/src/model.rs` (+24 LOC enum extensions)
- `_primitives/_rust/kei-model/data/models.toml` (+216 LOC, 471 total)

`kei-model list` returns the full 21-model catalog incl. new providers.

Tests:
- kei-registry: 25 passed (existing + 5 secrets tests + 10 status)
- kei-model: 0 (no unit tests in crate, parser smoke via list)
- agent-assembler: 29 passed (no regressions)

Verification (cited):
- `./target/release/kei-registry secrets --env-file ~/.claude/secrets/.env`
  emits real report 26/11 orphan.
- `./target/release/kei-model list` parses all 21 entries cleanly.
- `cargo build --release --workspace` clean.

What this does NOT do (deferred):
- Pricing.unit field (per-mtok / per-image / per-second / per-1k-chars
  discriminator) — needs Rust struct refactor + cost-estimator update.
- `secrets` skip-list extension (worktrees, _ts_packages/node_modules
  duplicate counts) — minor noise.
- gemini-3-pro-image pricing (no public page; vendor-specific quote
  needed).

=== STATUS-TRUTH MARKER ===
shipped: functional
stubs: 0
cargo-check: PASS
behaviour-verified: yes
follow-up-required:
  - Pricing.unit field for cost-estimator correctness on gen models
  - secrets scan: skip .claude/worktrees/ to avoid duplicate counts
  - gemini-3-pro-image price verification

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 00:06:16 +08:00
Parfii-bot
be1a864629 feat(outcome-hook): PostToolUse:Agent backfills outcome + stubs in ledger
Phase 4 follow-up: the outcome-backfill hook the kei-model-router
needs to learn from. Without an outcome signal the Beta posterior
sees 205 NULL rows and can never converge → router falls back to
top-tier on every spawn. This hook closes that loop.

Spawned by orchestrator as a code-implementer agent (Sonnet 4.6 by
default after the manifest refactor in cb1fdde — first dogfood proof
that the tier system works end-to-end). Agent returned cleanly with
STATUS-TRUTH MARKER `shipped: functional, stubs: 0`.

Files (3):
- `~/.claude/hooks/agent-outcome-backfill.sh` (73 LOC, /bin/sh) —
  reads PostToolUse:Agent stdin JSON, parses STATUS-TRUTH MARKER from
  `tool_response`, runs `UPDATE agents SET outcome = ?, stubs_count = ?
  WHERE id = ?` via sqlite3 CLI. Defensive on every step (never blocks,
  exits 0 on missing jq / sqlite3 / DB / marker). Bypass:
  `OUTCOME_BACKFILL_BYPASS=1`. Lives outside the kit (system-level).
- `tests/hook-outcome-backfill-test.sh` (79 LOC, /bin/sh) — 8 assertions
  cover: 4 valid outcomes, idempotent re-run, missing marker, bypass
  env, missing sqlite3 (PATH stripped). Run via
  `sh tests/hook-outcome-backfill-test.sh` → "Passed: 8 Failed: 0".
- `_blocks/path-user-hooks.md` — third path-atom following
  user-memory / user-rules convention. Resolves to `~/.claude/hooks/`.
  Lets future manifests reference hook files via
  `path:user-hooks/<file>.sh` opaquely. Registered in registry as
  `atom::md::331b9a34::023e5a08`.

Wiring:
- `~/.claude/settings.json` PostToolUse:Agent matcher chain — appended
  the hook idempotently (jq update preserves existing
  `agent-stub-scan.sh`, `task-timer.sh`, `agent-fork-done.sh`).
- DNA-INDEX regenerated; new path-atom appears in `## Atom (120)`
  section.

Effect: every Agent tool call from now on writes outcome + stubs to
ledger. After ~10-20 invocations the Beta posterior has a usable
prior; after ~50 the router stops defaulting Sonnet to Opus on
unfamiliar tasks. The advisor hook (`model-router-advisor.sh`)
already prints stderr when current model > recommended — orchestrator
needs to actually pass `model:` parameter on next spawn (behavioural,
not a code change).

=== STATUS-TRUTH MARKER ===
shipped: functional
stubs: 0
cargo-check: NOT-RUN
behaviour-verified: yes
follow-up-required:
  - PR feat/substrate-path-atoms-2026-05-01 → main when ready

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 23:24:02 +08:00
Parfii-bot
3422bdc8c3 feat(path-atoms): atomize ~/.claude memory + rules path references
Phase 1 of substrate-unified-registry: move all references to user
home memory/rules out of plain strings and into content-addressable
path atoms. Public artefacts now contain opaque `{path::NAME}/file.md`
references; the actual home prefix lives only in the path-atom file's
frontmatter, registered in the local kei-registry.

NEW path atoms (`_blocks/path-*.md`):
- `path-user-memory.md` → template `~/.claude/memory`
- `path-user-rules.md`  → template `~/.claude/rules`

Both files use frontmatter `type: atom, kind: path, template: ..., expand_at: render`.
BlockMdScanner auto-registers them; DNA index shows them under their
unprefixed names (`user-memory`, `user-rules`) for human lookup, while
the body sha8 makes them content-addressable.

Resolver (`_assembler/src/registry_client.rs`):
- `is_path_atom(conn, name)` — checks DB by name + filename convention
  (`_blocks/path-<name>.md`) + frontmatter `kind: path`. Defensive:
  filename + frontmatter must BOTH agree.
- `frontmatter_has_kind_path(body)` — minimal YAML parser. Tolerates
  CRLF, quoted values, rejects substring matches (`pathological` ≠ `path`).
- 5 unit tests cover positive + 4 negative cases.

Resolver wire-up (`_assembler/src/assembler.rs:147 write_references`):
- For each `references.extra` entry starting with `path:NAME/...`:
  - Lookup `NAME` via `is_path_atom`.
  - On success: emit `{path::NAME}/<suffix>` — opaque, kit-resolvable.
  - On miss: stderr warn + passthrough. Never fatal.
- Non-`path:` refs pass through unchanged. Backward compatible.
- 2 unit tests cover passthrough paths.

Manifest migration (38 manifests touched):
- `~/.claude/rules/<file>` → `path:user-rules/<file>`
- `~/.claude/memory/<file>` → `path:user-memory/<file>`
- 96 references migrated; 1 prose-style reference in security-auditor
  left as plain text (lives inside a domain_in description, not in
  references.extra — out of scope for this resolver).

Regenerated 38 `_generated/*.md` + 1 new `frontend-validator.md`.
Regenerated `docs/DNA-INDEX.md` (now includes 2 path-atoms by name).

Verification (cited):
- `git ls-files | grep denisparfionovich` → 0 hits outside allowlist
  (NOTICE/README byline + `.github/workflows/leak-check.yml` detection
  rule).
- `_generated/` contains 99 occurrences of `{path::user-...}/`.
- assembler tests: 29 passed (5 new). kei-registry tests: 10 passed
  (8 short_path from earlier commit + 2 unrelated).
- assembler resolver verified end-to-end: ml-implementer.md line
  479-485 shows `{path::user-rules}/ml-protocol.md` etc.

What this does NOT do (deferred):
- No registry-DB schema change. Path atoms ride existing Atom block-
  type via convention, not via new `BlockType::PathAtom` variant.
- No git-branch tracking (Phase 2 of plan).
- No `kei-registry status` cross-cutting CLI (Phase 3 of plan).
- No path-atom orphan detection CLI (Phase 4).

The path:user-memory and path:user-rules cover 100% of the username-
leak surface from the current manifest set; future categories
(kit-root, registry-db, sync-repo, secrets-env, project-root) can
land additively without architectural changes.

=== STATUS-TRUTH MARKER ===
shipped: functional
stubs: 0
cargo-check: PASS
behaviour-verified: yes
follow-up-required:
  - Phase 2 (git-branch tracker hook)
  - Phase 3 (kei-registry status subcommand)
  - Phase 4 (orphan detection CLI)
  - Sync user-side install: ~/.claude/agents/_manifests/ still has
    pre-migration absolute paths; will pick up new format on next
    `install.sh --add` (out of scope for this commit).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 22:29:50 +08:00
Parfii-bot
22ae9d1de5 fix(kei-registry): short_path strips _blocks/_manifests/_atoms/_roles/_caps/agents
Root-cause of the username-path leak in DNA-INDEX.md (107 atom rows
in v0.17 — sed-patched in a23910d). The encyclopedia render's
short_path() prefix list omitted every top-level dir except
`_primitives/`, `skills/`, `hooks/`, `rules/` — so atom and capability
rows fell through to the absolute path stored in the registry DB,
leaking the maintainer's home prefix into the public encyclopedia.

Fix: add `_blocks/`, `_manifests/`, `_generated/`, `_atoms/`,
`_assembler/`, `_roles/`, `_capabilities/`, `agents/`, `docs/` to
the prefix list. 8 unit tests cover the new prefixes (fixtures use
CI-style paths like `/srv/ci/build/...` so the source file does not
contain a maintainer-shaped path that would itself trip the local
pre-commit hook + leak-check CI).

Verified: regenerated docs/DNA-INDEX.md has 0 absolute-path hits.
Source fix supersedes the sed hot-fix in a23910d — the next
`kei-registry encyclopedia` invocation will not regress.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 21:09:15 +08:00
Parfii-bot
a23910d445 chore(security): scrub username paths from public artefacts + leak-check CI
Public repo had absolute paths revealing username:
- 5 _manifests/*.toml — companion_memory_files had author-time hardcoded
  ~/.claude/projects/-Users-<user>/memory/... paths
- 5 _generated/*.md — same paths rendered through to public output
- docs/DNA-INDEX.md — 107 absolute paths (kei-dna-index emits absolute
  for atoms but relative for primitives — generator inconsistency)
- skills/escalate-recurrence/SKILL.md — 2 instructional path examples

Substitution:
  ~/.claude/projects/-Users-<user>/memory/  ->  ~/.claude/memory/
  /Users/<user>/Projects/KeiSeiKit-public/  ->  <relative>

Defence-in-depth:
- .github/workflows/leak-check.yml — CI gate (PR + push to main)
- (local) .git/hooks/pre-commit — maintainer-side guard with allowlist
  for legitimate detection-rule files (the hook + the workflow itself)

NOTICE + README byline allowlisted (intentional copyright).
No secrets exposed — only metadata (username + private-memory filenames).
DNA-INDEX root-cause fix in kei-dna-index Rust binary tracked as TODO.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 19:59:29 +08:00
Parfii-bot
740889c422 chore: regenerate DNA-INDEX after kit changes (kei-db-contract + Wave B follow-up)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 17:59:15 +08:00
Parfii-bot
f3f5f79760 feat(frontend-loop): kei-db-contract primitive + frontend-validator agent + auto-dev-guard hook
Frontend continuous-quality loop landed. Three composable cubes:

Wave 1 — kei-db-contract primitive (~870 LOC, 7 cubes per Constructor Pattern):
- Diffs SQL CREATE TABLE migrations against TypeScript type/interface declarations
- 4 drift modes: ORPHAN-SQL, ORPHAN-TS, TYPE-MISMATCH, NULL-MISMATCH
- Reuses sqlparser-rs (Apache 2.0) + regex + walkdir + serde_json + clap
- CLI: kei-db-contract <project-root> [--output json|text] [--strict]
- 5/5 integration tests pass (cargo check + cargo test green)
- Smoke-tested on keisei-marketplace: drift_count=266 across 30 tables
  (expected — marketplace uses raw better-sqlite3 without explicit row types)

Wave 2 — frontend-validator agent + dev-guard skill extension:
- New _manifests/frontend-validator.toml (substrate_role: edit-local, tools: Bash+Read+Glob+Grep)
- Agent runs: stack detect → tsc --noEmit → eslint → kei-db-contract → playwright (optional)
- Severity rules: TYPE_CHECK FAIL = block, DB_CONTRACT drift > 0 = block, lint = advisory
- skills/dev-guard/SKILL.md extended: 4th agent triggered on .tsx/.ts/.dart edits or DB-layer touches
- adaptive-depth table extended with frontend + DB-layer rows

Wave 3 — auto-dev-guard.sh hook (PostToolUse:Edit|Write):
- Trivial-edit gate: skip if delta < 30 LOC (avoid spawn fatigue)
- File-pattern match: *.tsx|*.ts|*.svelte|*.vue|*.dart OR migrations/*.sql OR src/db/** OR src/types/** OR prisma/schema.prisma OR drizzle.config.*
- Auto-runs kei-db-contract for DB-layer edits if binary on PATH
- Stderr advisory only (exit 0 always — never blocks)
- Bypass: KEI_DISABLED_HOOKS or KEI_HOOK_PROFILE in {advisory-off, minimal, off}
- Smoke-tested with synthetic Edit input (39 LOC delta on .tsx → emits advisory)
- Registered in hooks/hooks.json under PostToolUse:Write|Edit chain

Reusability map (Constructor Pattern compose):
  shared cubes: detect-stack, tsc, eslint, kei-db-contract, kei-visual-snapshot (deferred)
  orchestrators: /dev-start (pre), /dev-guard (during, NOW with frontend-validator),
                 /dev-ship (final), /site-create (init)

Verify-before-commit (RULE 0.13):
- cargo check -p kei-db-contract: PASS
- cargo test -p kei-db-contract: 5 passed
- jq . hooks/hooks.json: valid
- bash hooks/auto-dev-guard.sh < synthetic-input: works (frontend-relevant edit detected, exit 0)

=== STATUS-TRUTH MARKER ===
shipped: functional
stubs: 0
cargo-check: PASS
cargo-test: PASS (5 tests, 0 failures)
behaviour-verified: yes
follow-up-required:
  - kei-visual-snapshot primitive (Playwright wrap) — Wave 4, deferred
  - /dev-start frontend-contract-designer agent + /dev-ship frontend-final-gate — Wave 5, after Wave 1-3 obkatka
  - install.sh wiring for kei-db-contract binary
  - hermes-style emit-on-drift advisory mode

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 15:34:39 +08:00
Parfii-bot
902fb3e81a feat(kei-memory): functional schema fix + 4-wave architecture refactor
Wave A — Functional ingest fix (root cause of empty Sleep reports):
- Rewrote TraceLine struct to match real Claude Code trace JSONL:
  type (was kind), timestamp ISO8601 (was epoch ts), message Object,
  cwd / gitBranch / parentUuid / uuid / subtype / toolUseID / toolUseResult
- New src/extract.rs: extract_tool_uses + extract_tool_result walks
  message.content[] for nested tool_use / tool_result blocks
- New src/classifier.rs: explicit table classifier (tool_error, user_correction,
  retry_loop, permission_denied, tool_use:<name>, ...) replaces shallow heuristic
- New src/error.rs: KeiMemoryError enum (IO/Parse/Db) replaces semantic
  mismatch where IO error was wrapped as rusqlite::InvalidParameterName
- New src/trace_line.rs: TraceLine + helpers (cube extraction)
- Schema migration v3: events.cwd column + 3 hot-query indices
  (events.tool, events.file_path, events.ts) + UNIQUE on patterns
- New tests/ingest_real_trace.rs: synth-fixture asserts tool/file/cwd/class extraction

Wave B — Lib crate split:
- Cargo.toml: [lib] target added alongside existing [[bin]]
- src/lib.rs: pub re-export of all 18 modules
- src/main.rs: 11 mod declarations replaced by single use kei_memory::{…}
- tests/integration.rs: #[path] hack replaced by use kei_memory::{…}

Wave C — TF-IDF dedup + single-JOIN + filter_map fix:
- Schema migration v2: tokens.idf_dirty column + flag-based dedup
- index_document no longer triggers per-call recompute_idf rebuild
- top_similar uses single JOIN via vectors_for_overlapping_sessions helper
  (was N round-trips, one session_vector per candidate)
- All filter_map(|r| r.ok()) row-error swallowing replaced with ? propagation
- New tests/tfidf_idf_dedup.rs: 4 tests covering dedup behaviour, IDF emptiness,
  JOIN-pruning, empty-query safety

Wave D — Commands split + nits:
- New src/dump.rs (43 LOC) + src/stats.rs (33 LOC):
  CLI renderers extracted from commands.rs (was inline SQL + format)
- src/commands.rs: thin wrappers, -42 LOC
- src/injection_guard.rs: inline tests removed (-26 LOC), file under 200 LOC threshold
- tests/injection_guard_unit.rs (new): 4 tests in proper integration crate
- src/patterns.rs: INSERT replaced with INSERT...ON CONFLICT...DO UPDATE
  (idempotent re-ingest, uses Wave A's UNIQUE index)
- src/analyze.rs + src/coaccess.rs: filter_map row-error fixes
- src/coaccess.rs: misleading PK comment rewritten

Verify-before-commit (RULE 0.13 §"Verify-before-commit"):
- cargo check --all-targets: PASS (1 unrelated dead-code warning)
- cargo test: 42 passed, 0 failed across 9 test binaries
- STATUS-TRUTH markers aggregated at .claude/agents/_merge/kei-memory-2026-05-01/

Architect-spotted ARCH-MAJOR + ARCH-MINOR + ARCH-NIT findings addressed:
- ARCH-MAJOR Cargo.toml binary-only (Wave B)
- ARCH-MAJOR schema missing indices (Wave A v3)
- ARCH-MAJOR ingest_jsonl choke point (Wave A — extract.rs + classifier.rs)
- ARCH-MAJOR idf O(N·V) per-call rebuild (Wave C)
- ARCH-MINOR patterns no UPSERT (Wave D)
- ARCH-MINOR commands.rs houses dump+stats (Wave D)
- ARCH-MINOR classifier silent contract (Wave A)
- ARCH-MINOR IO error wrapped as rusqlite (Wave A)
- ARCH-MINOR injection_guard inline tests (Wave D)
- ARCH-MINOR tfidf top_similar N round-trips (Wave C)
- ARCH-NIT 3× filter_map(|r| r.ok()) sites (Wave C + D)
- ARCH-NIT coaccess misleading comment (Wave D)

=== STATUS-TRUTH MARKER ===
shipped: functional
stubs: 0
cargo-check: PASS
cargo-test: PASS (42 tests, 0 failures)
behaviour-verified: yes
follow-up-required:
  - tests/ingest_guard_tests.rs + tests/guard_test_corpus.rs still on #[path] hack (Wave B follow-up note, ~5 LOC)
  - dead_code warning Severity::Warn unused (pre-existing, not blocking)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 14:10:06 +08:00
Parfii-bot
0be354a920 KeiSeiKit-public — clean state
Single-commit clean baseline after security scrub of niche-tells,
project codenames, internal jargon, and contributor-email leaks.

Contents:
- 100 Rust crates (_primitives/_rust/)
- 37 agent manifests (_manifests/) + generated specs (_generated/)
- 67 user-invocable skills (skills/)
- 33 hooks (hooks/)
- Composition blocks (_blocks/)
- Documentation (docs/, README.md)
- TS adapter packages (_ts_packages/)
- Assembler (_assembler/)
- Roles (_roles/)
- Templates (_templates/)
- Forgejo CI (.forgejo/)

Author: Denis Parfionovich <info@greendragon.info>

License: see LICENSE.
2026-05-01 12:09:03 +08:00