Commit graph

71 commits

Author SHA1 Message Date
Parfii-bot
e073df6c98 feat(tracking): close 3 last observability gaps — toolStats + skill-record + numeric-claims journal
Closes the loop on "without full tracking the system can't make decisions"
(user pushback on partial coverage). Three gaps that left the inference
layer blind are now wired:

GAP #1 — agent toolStats / token counts / cache hits captured
================================================================
`agent-outcome-backfill.sh` now appends one JSONL row per spawn to
`~/.claude/memory/time-metrics/agent-toolstats.jsonl` with:
  agent_id, outcome, stubs, ts,
  tool_use_count, duration_ms, tool_stats {Read:N, Bash:M, ...},
  tokens_in, tokens_out, cache_read, cache_write
Sidecar journal (no schema migration). Production payload's
.tool_response.totalToolUseCount / totalDurationMs / toolStats / usage
fields land directly. Smoke-tested with synthetic spawn — row written.

GAP #2 — skill_invocations table actually receives writes
================================================================
The `skill_invocations` table (schema v8) had 0 rows because no caller
existed for `skill_metrics::record_invocation`. Added two pieces:

(a) `kei-ledger record-skill <name> --success {0|1}` CLI subcommand
    Mirrors record-cost; same dispatch shape. Optional `--agent-id`,
    `--trajectory-id`, `--duration-ms`, `--db`. Validates non-empty
    name + duration ≥ 0. Outputs `{"ok":true,"skill":"...","ts":N}`.

(b) `hooks/skill-record.sh` — PostToolUse:Skill hook. 50 LOC POSIX.
    Detects Skill tool calls, derives success heuristic from
    tool_response (exit_code / status / content non-empty), shells
    out to `kei-ledger record-skill`. Bypass via SKILL_RECORD_BYPASS=1.

83 kei-ledger tests pass (16 unit + 67 integration). Smoke-tested
end-to-end: `kei-ledger record-skill test-skill --success 1` inserts
a row with correct fields.

Phase D nightly skill-metrics decisions (archive if unused N days,
re-extract if success<60% over M days, validated if >20 calls + >90%
success) now have data to consume.

GAP #3 — numeric-claims.jsonl receives every evidence-tagged claim
================================================================
RULE 0.18 mandated three markers `[REAL:]` / `[FROM-JOURNAL:]` /
`[ESTIMATE-HTC:]` on every numeric/duration/cost claim, but no hook
appended valid claims to the journal — the calibration data RULE 0.18
promised never accumulated.

`hooks/numeric-claims-record.sh` — Stop hook, 140 LOC POSIX. Reads
transcript_path from stdin, locates the last assistant message via
recursive flatten (same pattern as agent-outcome-backfill.sh after
the production-payload-shape fix), regex-extracts every `<phrase>
[<TIER>: <pointer>]` triple, appends one JSONL row per claim.

Idempotent within 1-second window to avoid double-recording on
repeat Stop fires. Bypass via NUMERIC_CLAIMS_RECORD_BYPASS=1.

Smoke test: synthetic transcript with 3 markers (REAL + ESTIMATE-HTC
+ FROM-JOURNAL) produced exactly 3 well-formed JSONL rows.

Settings.json
================================================================
- PostToolUse:Skill matcher created (or augmented if already
  present) with skill-record.sh.
- Stop:* matcher gains numeric-claims-record.sh after the existing
  chain (stop-verify, task-timer, session-end-dump, extract-task-
  durations, chat-numeric-postflag, affect-threshold-check,
  enrich-from-jsonl).

What this does NOT do (deferred):
  - Backfill `skill_invocations` from past traces (history started
    today; Phase D cohort builds forward from now).
  - Migrate the agent toolStats sidecar JSONL into a proper ledger
    column. Append-only file is fine for the current scale.
  - Refactor main.rs (now 233 LOC, was 212; pre-existing CP debt
    flagged by skill-record agent — separate cleanup PR).

=== STATUS-TRUTH MARKER ===
shipped: functional
stubs: 0
cargo-check: PASS
behaviour-verified: yes
follow-up-required:
  - kei-ledger main.rs Constructor Pattern split (212→233 LOC)
  - Verify in next session: skill_invocations gets rows from real
    Skill tool use; numeric-claims.jsonl gets rows from real assistant
    messages with markers

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 03:42:09 +08:00
Parfii-bot
033b9efbad fix(outcome-hook): production payload uses object.content[*].text shape
Hook never fired in production despite passing unit tests. Diagnosed
via debug-log + payload dump: real Claude Code PostToolUse:Agent sends
`tool_response` as an OBJECT (not string, not array), with the agent's
reply at `tool_response.content[0].text` — keys: agentId / agentType /
content / prompt / status / toolStats / totalDurationMs / totalTokens
/ totalToolUseCount / usage.

Original jq filter handled string + object (`$r.content // $r.text`)
but `$r.content` returns the array verbatim; `jq -r` then dumps the
JSON literal which has `\n` as escape sequences, defeating the
`grep -m1 '^shipped:'` line-anchor.

Fix: recursive `flatten` jq function:
  string                     → as-is
  array of any               → recurse, join "\n"
  object with .text          → return .text
  object with .content       → recurse into content
  anything else              → ""

Verified end-to-end: latest 4 code-implementer spawns now write
outcome=functional to ledger correctly. Beta posterior in
kei-model-router begins receiving signal.

Production cleanup:
- Removed verbose debug-log + payload-dump diagnostic. Toggle via
  `AGENT_OUTCOME_DEBUG=1` env if hook stops firing in some future
  Claude Code version.
- Hook source committed to `hooks/agent-outcome-backfill.sh` so
  `install.sh` deploys it on fresh installs (was only in user-home
  previously — gap from `feat/substrate-path-atoms` agent run).

=== STATUS-TRUTH MARKER ===
shipped: functional
stubs: 0
cargo-check: NOT-RUN
behaviour-verified: yes
follow-up-required:
  - none

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 01:09:15 +08:00
Parfii-bot
3b9133a311 Merge feat/substrate-path-atoms-2026-05-01 → main
Substrate-unified-registry — 5-commit feature branch consolidating
the leak-scrub root cause + visibility layer over the existing kit
artefact graph.

Commits in chronological order:
  3422bdc  feat(path-atoms): atomize ~/.claude memory + rules path references
  d6dbdee  feat(kei-registry): status subcommand — cross-cutting substrate dashboard
  cb1fdde  feat(model-tier+branch-dna): activate cost router + give branches DNA
  be1a864  feat(outcome-hook): PostToolUse:Agent backfills outcome + stubs in ledger
  af46684  feat(secrets+catalog): orphan-detector for env vars + image/video/voice models

Pre-merge verify (RULE 0.16):
  cargo check --workspace      → exit 0
  cargo test --no-run --workspace → exit 0
  agent-assembler test --no-run → exit 0

Coverage:
  120 atoms (incl 3 path: user-memory, user-rules, user-hooks)
  40 hooks, 106 primitives, 174 rules, 68 skills
  21 model catalog entries (8 providers; was 11/5)
  26 secrets indexed (11 orphan flagged for cleanup)
  205 agent forks + 17 branches with DNA (100%)

Net effect:
  - 0 username paths in tracked files (allowlisted: NOTICE, README, leak-check.yml)
  - All 38 agent manifests refactored to model = "sonnet" (4 stay opus)
  - kei-model-router learning loop closed via outcome-backfill hook
  - kei-registry status / secrets / list — three queries cover the substrate

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 00:11:12 +08:00
Parfii-bot
af46684330 feat(secrets+catalog): orphan-detector for env vars + image/video/voice models
Two parallel agents (both Sonnet 4.6 via the just-activated tier system)
extended the substrate-unified-registry. First end-to-end proof that the
Phase 4 router refactor saves money: no Opus spawns this round.

PART 1 — `kei-registry secrets` subcommand (Agent A — code-implementer)

Reads env-var NAMES from `~/.claude/secrets/.env` (RULE 0.8 SSoT) and
per-project `secrets/*.env`, greps the kit tree for usages, reports
orphans (defined but unreferenced). Live run on this kit found 26 keys,
11 ORPHAN — actionable cleanup candidates incl. GitHub OAuth client
creds, Godaddy keys, KeiGit admin creds, KEI_MEMORY_TOKEN.

Files:
- `_primitives/_rust/kei-registry/src/secrets.rs` (152 LOC) — pure
  read-side cube. SecretsReport + KeyRow types, env-file parser
  (KEY=value lines, validates `^[A-Z][A-Z0-9_]*$`), walkdir-based
  scanner with skips (target/ node_modules/ .git/ _generated/),
  word-boundary regex per key. ASCII + JSON render.
- `_primitives/_rust/kei-registry/src/secrets_tests.rs` (125 LOC) —
  5 unit tests covering env parse, scan correctness, word-boundary
  regression (`MY_KEY` ≠ `MY_KEY_EXTRA`), JSON roundtrip, ORPHAN marker.
- `_primitives/_rust/kei-registry/src/secrets_handler.rs` (58 LOC) —
  CLI dispatch handler.
- `cli.rs`, `handlers.rs`, `lib.rs` extended with Secrets variant.

Resolves the asymmetry called out in the design discussion: paths got
atomization (commit 3422bdc), keys get a query-layer instead. Reason:
env-var NAMES are already public and stable; opaque atom-DNA over them
adds zero security and full overhead. Orphan detection is the unique
value, and a 30-LOC subcommand delivers it without a per-key atom file.

PART 2 — kei-model catalog extension (Agent B — fal-ai-runner)

Adds 10 generation-model entries with VERIFIED pricing per RULE 0.4:
- google: gemini-3-1-flash-image, gemini-3-pro-image
- fal.ai: flux-2-pro, flux-pro-1-1, kling-o3, veo-3, ideogram-v3, recraft-v3
- elevenlabs: elevenlabs-v3, elevenlabs-multilingual-v2

Pricing sourced from each provider's public pricing page (URLs cited
per row in `notes` + `source_url` fields); 8/10 verified, 2 marked
needs-verification (gemini-3-pro-image price not found on public page).

Schema additions to `_primitives/_rust/kei-model/src/model.rs` to
support the new entries without `provider = "local"` placeholder:
- Provider enum + 3 variants: Google, Fal, Elevenlabs (with as_str
  + parse impls).
- Capability enum + 9 variants: image-gen, text-to-image, image-edit,
  video-gen, text-to-video, image-to-video, voice-gen, text-to-speech,
  voice-clone (with serde rename + as_str + parse).

Pricing struct unchanged: per-image / per-second / per-1k-chars unit
costs ride existing `output_per_mtok_micro` field with the unit
documented in `notes` (e.g. "Per-image cost. 1 unit = 1 image."). A
proper Pricing.unit field is a follow-up.

Files:
- `_primitives/_rust/kei-model/src/model.rs` (+24 LOC enum extensions)
- `_primitives/_rust/kei-model/data/models.toml` (+216 LOC, 471 total)

`kei-model list` returns the full 21-model catalog incl. new providers.

Tests:
- kei-registry: 25 passed (existing + 5 secrets tests + 10 status)
- kei-model: 0 (no unit tests in crate, parser smoke via list)
- agent-assembler: 29 passed (no regressions)

Verification (cited):
- `./target/release/kei-registry secrets --env-file ~/.claude/secrets/.env`
  emits real report 26/11 orphan.
- `./target/release/kei-model list` parses all 21 entries cleanly.
- `cargo build --release --workspace` clean.

What this does NOT do (deferred):
- Pricing.unit field (per-mtok / per-image / per-second / per-1k-chars
  discriminator) — needs Rust struct refactor + cost-estimator update.
- `secrets` skip-list extension (worktrees, _ts_packages/node_modules
  duplicate counts) — minor noise.
- gemini-3-pro-image pricing (no public page; vendor-specific quote
  needed).

=== STATUS-TRUTH MARKER ===
shipped: functional
stubs: 0
cargo-check: PASS
behaviour-verified: yes
follow-up-required:
  - Pricing.unit field for cost-estimator correctness on gen models
  - secrets scan: skip .claude/worktrees/ to avoid duplicate counts
  - gemini-3-pro-image price verification

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 00:06:16 +08:00
Parfii-bot
be1a864629 feat(outcome-hook): PostToolUse:Agent backfills outcome + stubs in ledger
Phase 4 follow-up: the outcome-backfill hook the kei-model-router
needs to learn from. Without an outcome signal the Beta posterior
sees 205 NULL rows and can never converge → router falls back to
top-tier on every spawn. This hook closes that loop.

Spawned by orchestrator as a code-implementer agent (Sonnet 4.6 by
default after the manifest refactor in cb1fdde — first dogfood proof
that the tier system works end-to-end). Agent returned cleanly with
STATUS-TRUTH MARKER `shipped: functional, stubs: 0`.

Files (3):
- `~/.claude/hooks/agent-outcome-backfill.sh` (73 LOC, /bin/sh) —
  reads PostToolUse:Agent stdin JSON, parses STATUS-TRUTH MARKER from
  `tool_response`, runs `UPDATE agents SET outcome = ?, stubs_count = ?
  WHERE id = ?` via sqlite3 CLI. Defensive on every step (never blocks,
  exits 0 on missing jq / sqlite3 / DB / marker). Bypass:
  `OUTCOME_BACKFILL_BYPASS=1`. Lives outside the kit (system-level).
- `tests/hook-outcome-backfill-test.sh` (79 LOC, /bin/sh) — 8 assertions
  cover: 4 valid outcomes, idempotent re-run, missing marker, bypass
  env, missing sqlite3 (PATH stripped). Run via
  `sh tests/hook-outcome-backfill-test.sh` → "Passed: 8 Failed: 0".
- `_blocks/path-user-hooks.md` — third path-atom following
  user-memory / user-rules convention. Resolves to `~/.claude/hooks/`.
  Lets future manifests reference hook files via
  `path:user-hooks/<file>.sh` opaquely. Registered in registry as
  `atom::md::331b9a34::023e5a08`.

Wiring:
- `~/.claude/settings.json` PostToolUse:Agent matcher chain — appended
  the hook idempotently (jq update preserves existing
  `agent-stub-scan.sh`, `task-timer.sh`, `agent-fork-done.sh`).
- DNA-INDEX regenerated; new path-atom appears in `## Atom (120)`
  section.

Effect: every Agent tool call from now on writes outcome + stubs to
ledger. After ~10-20 invocations the Beta posterior has a usable
prior; after ~50 the router stops defaulting Sonnet to Opus on
unfamiliar tasks. The advisor hook (`model-router-advisor.sh`)
already prints stderr when current model > recommended — orchestrator
needs to actually pass `model:` parameter on next spawn (behavioural,
not a code change).

=== STATUS-TRUTH MARKER ===
shipped: functional
stubs: 0
cargo-check: NOT-RUN
behaviour-verified: yes
follow-up-required:
  - PR feat/substrate-path-atoms-2026-05-01 → main when ready

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 23:24:02 +08:00
Parfii-bot
cb1fddeabb feat(model-tier+branch-dna): activate cost router + give branches DNA
Phase 4 of substrate-unified-registry: turn on the existing
kei-model-router by changing manifest defaults from `model = "opus"`
to `model = "sonnet"` for routine agents, and give every git branch
a deterministic DNA in the kei-status dashboard.

The model-tier system was BUILT (`_primitives/_rust/kei-model-router/`
crate with Beta posterior, complexity τ-estimator, escalate ladder,
calibrate subcommand) and the advisor hook
(`~/.claude/hooks/model-router-advisor.sh`) was REGISTERED. But every
ledger row from this session ran on Opus because:
  1. All 38 manifests hard-coded `model = "opus"` → no chance for the
     router to recommend cheaper.
  2. The orchestrator (me) ignored the stderr advisory.

This commit closes (1). (2) is a behavioural change tracked separately.

Manifest reclassification (4 Opus + 34 Sonnet):
  Opus (hard reasoning):
    - architect            (system-design synthesis)
    - ml-implementer       (Math-First paradigm)
    - ml-researcher        (literature analysis)
    - security-auditor     (deep risk synthesis)
  Sonnet (everything else):
    - 8 code-implementer-* + code-implementer
    - 5 critic-* + critic
    - 6 infra-implementer-* + infra-implementer
    - 4 researcher-* + researcher
    - 6 validator-* + validator
    - 3 security-auditor-{differential,supply-chain,variant}
    - cost-guardian, fal-ai-runner, frontend-validator, modal-runner

Regenerated all 38 `_generated/*.md` so the YAML frontmatter `model:`
field matches the manifest.

Branch DNA (kei-registry status):
  - New `compute_branch_dna(name, commit_sha)` in `status.rs`. Format
    `branch::git::<sha8(name)>::<sha8(commit)>`, mirrors kei-shared
    DNA wire layout `<role>::<caps>::<scope_sha8>::<body_sha8>`.
  - Deterministic — same `(name, commit)` → same DNA. Changes when
    either changes. No DB persistence: the underlying truth lives in
    `.git/refs/heads/<name>`.
  - 3 new unit tests cover format, determinism, name-change, commit-
    change. `cargo test status::tests` → 10 passed.

`kei-registry status` output now shows DNA prefix per branch alongside
ahead/behind, last commit. Combined with existing per-block DNA in the
[Blocks] and [Path Atoms] sections + `dna` column on `agents` table in
kei-ledger, every artefact in the dashboard has an identifier:

  Atoms (incl path-atoms)  → atom::<caps>::<scope>::<body>     (registry)
  Skills/Rules/Hooks/Prim  → <role>::<caps>::<scope>::<body>   (registry)
  Agent forks              → row.dna in agents table           (ledger)
  Local branches           → branch::git::<sha8>::<sha8>       (computed)

What this does NOT do:
- No outcome backfill — the 205 NULL outcomes in ledger still prevent
  the Beta posterior from learning. Router falls back to top-tier
  until ≥1 datapoint per (task_class, model) accumulates. Tracked as
  follow-up.
- No post-checkout hook to auto-register branches in kei-ledger. Live
  shell-out to `git for-each-ref` is fast enough for the dashboard;
  persistence buys nothing the .git tree doesn't already give.

=== STATUS-TRUTH MARKER ===
shipped: functional
stubs: 0
cargo-check: PASS
behaviour-verified: yes
follow-up-required:
  - Outcome backfill hook (writes outcome to ledger after agent done)
  - User /model claude-sonnet-4-6 for current session (5x cheaper)
  - Push the orchestrator (me) to read advisor stderr in real-time

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 23:05:07 +08:00
Parfii-bot
d6dbdee870 feat(kei-registry): status subcommand — cross-cutting substrate dashboard
Phase 3 of substrate-unified-registry: a single command shows every
live artefact across the three sources without merging stores.

`kei-registry status` joins:
1. `blocks` table (kei-registry SQLite) — active counts per BlockType,
   plus the registered path-atoms with DNA prefix + body sha8.
2. `git for-each-ref refs/heads` (shell-out, no DB persistence) — local
   branches, current marker, ahead/behind via `upstream:track,nobracket`.
3. `agents` table (kei-ledger SQLite) — fork counts per status
   (running/done/failed/merged/rejected). Missing ledger DB → section
   skipped, never an error.

Output: ASCII multi-section table by default; `--format json` for
machine consumption.

Files:
- `_primitives/_rust/kei-registry/src/status.rs` — new module, ~270
  LOC. Pure read-side per Constructor Pattern. 7 unit tests cover
  `parse_track` (in sync / ahead / behind / both / "gone"), DNA prefix
  rendering, and empty-status section presence.
- `_primitives/_rust/kei-registry/src/cli.rs` — new `Status` variant
  with `--db`, `--git-repo`, `--ledger-db`, `--format` flags.
- `_primitives/_rust/kei-registry/src/handlers.rs` — `handle_status`
  dispatcher, ASCII/JSON branching.
- `_primitives/_rust/kei-registry/src/lib.rs` — module export.

End-to-end run from kit root shows the prior gap: 17 local branches
(many `worktree-agent-*` orphans), kei-ledger summary 4 running /
158 done / 35 failed / 7 merged / 0 rejected — visibility the user
asked for ("в каждой сессии видеть, чтобы не бегать по диску в
поисках несмерженных").

What this does NOT do (Phase 4):
- No orphan detection (`kei-status orphans`) — counts only.
- No auto-registration of branches into kei-ledger (Phase 2). Branches
  come from live `git for-each-ref` shell-out; if the repo moves or
  is deleted the row vanishes from the dashboard. Acceptable for v1.

=== STATUS-TRUTH MARKER ===
shipped: functional
stubs: 0
cargo-check: PASS
behaviour-verified: yes
follow-up-required:
  - Phase 2 (post-checkout hook → kei-ledger auto-register)
  - Phase 4 (orphan detection: branches with no commits in N days,
    path-atoms with no consumers, agent forks stuck running)
  - --filter flags (--type, --status) for targeted queries

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 22:46:53 +08:00
Parfii-bot
3422bdc8c3 feat(path-atoms): atomize ~/.claude memory + rules path references
Phase 1 of substrate-unified-registry: move all references to user
home memory/rules out of plain strings and into content-addressable
path atoms. Public artefacts now contain opaque `{path::NAME}/file.md`
references; the actual home prefix lives only in the path-atom file's
frontmatter, registered in the local kei-registry.

NEW path atoms (`_blocks/path-*.md`):
- `path-user-memory.md` → template `~/.claude/memory`
- `path-user-rules.md`  → template `~/.claude/rules`

Both files use frontmatter `type: atom, kind: path, template: ..., expand_at: render`.
BlockMdScanner auto-registers them; DNA index shows them under their
unprefixed names (`user-memory`, `user-rules`) for human lookup, while
the body sha8 makes them content-addressable.

Resolver (`_assembler/src/registry_client.rs`):
- `is_path_atom(conn, name)` — checks DB by name + filename convention
  (`_blocks/path-<name>.md`) + frontmatter `kind: path`. Defensive:
  filename + frontmatter must BOTH agree.
- `frontmatter_has_kind_path(body)` — minimal YAML parser. Tolerates
  CRLF, quoted values, rejects substring matches (`pathological` ≠ `path`).
- 5 unit tests cover positive + 4 negative cases.

Resolver wire-up (`_assembler/src/assembler.rs:147 write_references`):
- For each `references.extra` entry starting with `path:NAME/...`:
  - Lookup `NAME` via `is_path_atom`.
  - On success: emit `{path::NAME}/<suffix>` — opaque, kit-resolvable.
  - On miss: stderr warn + passthrough. Never fatal.
- Non-`path:` refs pass through unchanged. Backward compatible.
- 2 unit tests cover passthrough paths.

Manifest migration (38 manifests touched):
- `~/.claude/rules/<file>` → `path:user-rules/<file>`
- `~/.claude/memory/<file>` → `path:user-memory/<file>`
- 96 references migrated; 1 prose-style reference in security-auditor
  left as plain text (lives inside a domain_in description, not in
  references.extra — out of scope for this resolver).

Regenerated 38 `_generated/*.md` + 1 new `frontend-validator.md`.
Regenerated `docs/DNA-INDEX.md` (now includes 2 path-atoms by name).

Verification (cited):
- `git ls-files | grep denisparfionovich` → 0 hits outside allowlist
  (NOTICE/README byline + `.github/workflows/leak-check.yml` detection
  rule).
- `_generated/` contains 99 occurrences of `{path::user-...}/`.
- assembler tests: 29 passed (5 new). kei-registry tests: 10 passed
  (8 short_path from earlier commit + 2 unrelated).
- assembler resolver verified end-to-end: ml-implementer.md line
  479-485 shows `{path::user-rules}/ml-protocol.md` etc.

What this does NOT do (deferred):
- No registry-DB schema change. Path atoms ride existing Atom block-
  type via convention, not via new `BlockType::PathAtom` variant.
- No git-branch tracking (Phase 2 of plan).
- No `kei-registry status` cross-cutting CLI (Phase 3 of plan).
- No path-atom orphan detection CLI (Phase 4).

The path:user-memory and path:user-rules cover 100% of the username-
leak surface from the current manifest set; future categories
(kit-root, registry-db, sync-repo, secrets-env, project-root) can
land additively without architectural changes.

=== STATUS-TRUTH MARKER ===
shipped: functional
stubs: 0
cargo-check: PASS
behaviour-verified: yes
follow-up-required:
  - Phase 2 (git-branch tracker hook)
  - Phase 3 (kei-registry status subcommand)
  - Phase 4 (orphan detection CLI)
  - Sync user-side install: ~/.claude/agents/_manifests/ still has
    pre-migration absolute paths; will pick up new format on next
    `install.sh --add` (out of scope for this commit).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 22:29:50 +08:00
Parfii-bot
3043188da2 fix(ci): leak-check uses awk instead of sed (shellcheck SC2001)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 21:15:09 +08:00
Parfii-bot
22ae9d1de5 fix(kei-registry): short_path strips _blocks/_manifests/_atoms/_roles/_caps/agents
Root-cause of the username-path leak in DNA-INDEX.md (107 atom rows
in v0.17 — sed-patched in a23910d). The encyclopedia render's
short_path() prefix list omitted every top-level dir except
`_primitives/`, `skills/`, `hooks/`, `rules/` — so atom and capability
rows fell through to the absolute path stored in the registry DB,
leaking the maintainer's home prefix into the public encyclopedia.

Fix: add `_blocks/`, `_manifests/`, `_generated/`, `_atoms/`,
`_assembler/`, `_roles/`, `_capabilities/`, `agents/`, `docs/` to
the prefix list. 8 unit tests cover the new prefixes (fixtures use
CI-style paths like `/srv/ci/build/...` so the source file does not
contain a maintainer-shaped path that would itself trip the local
pre-commit hook + leak-check CI).

Verified: regenerated docs/DNA-INDEX.md has 0 absolute-path hits.
Source fix supersedes the sed hot-fix in a23910d — the next
`kei-registry encyclopedia` invocation will not regress.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 21:09:15 +08:00
Parfii-bot
ce3deb43fa fix(ci): leak-check allowlists itself (workflow contains pattern as detection rule)
leak-check.yml has the pattern 'denisparfionovich' as a literal in its grep.
On first run after install, it flags itself. Same fix as the local
.git/hooks/pre-commit — allowlist the workflow file alongside NOTICE/README.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 20:04:36 +08:00
Parfii-bot
a23910d445 chore(security): scrub username paths from public artefacts + leak-check CI
Public repo had absolute paths revealing username:
- 5 _manifests/*.toml — companion_memory_files had author-time hardcoded
  ~/.claude/projects/-Users-<user>/memory/... paths
- 5 _generated/*.md — same paths rendered through to public output
- docs/DNA-INDEX.md — 107 absolute paths (kei-dna-index emits absolute
  for atoms but relative for primitives — generator inconsistency)
- skills/escalate-recurrence/SKILL.md — 2 instructional path examples

Substitution:
  ~/.claude/projects/-Users-<user>/memory/  ->  ~/.claude/memory/
  /Users/<user>/Projects/KeiSeiKit-public/  ->  <relative>

Defence-in-depth:
- .github/workflows/leak-check.yml — CI gate (PR + push to main)
- (local) .git/hooks/pre-commit — maintainer-side guard with allowlist
  for legitimate detection-rule files (the hook + the workflow itself)

NOTICE + README byline allowlisted (intentional copyright).
No secrets exposed — only metadata (username + private-memory filenames).
DNA-INDEX root-cause fix in kei-dna-index Rust binary tracked as TODO.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 19:59:29 +08:00
Parfii-bot
72d257e602 feat(install): Rust binary acquisition for fresh-clone installs (Path A + Path B)
Pre-fix: install.sh on a fresh clone has no binaries — target/ is gitignored,
copy_prebuilt_substrate_binaries() skips silently with "no pre-built found",
end users get zero kei-fork / kei-ledger / kei-cortex / etc.

New module: install/lib-rust-prebuild.sh (~120 LOC, Constructor Pattern).

  ensure_rust_binaries() — main entry, idempotent
    1. has_prebuilt_substrate_binaries() — quorum check (5+ kit binaries already in
       target/release/ → no-op).
    2. Else Path A: download keisei-${TARGET}.tar.gz from
       https://github.com/KeiSei84/KeiSeiKit-1.0/releases/latest/download/
       Detects target via uname (x86_64/aarch64 × darwin/linux), verifies
       sha256, extracts into target/release/.
    3. Path A fail (404, network, sha mismatch) → Path B fallback:
       cargo build --release --workspace (slow first time, requires Rust).
    4. Path B fail (no cargo) → say + return non-zero.

  Bypass: KEI_SKIP_RUST=1 — skip both paths (markdown-only install).

Wired in install/lib-rust.sh — ensure_rust_binaries() called BEFORE
copy_prebuilt_substrate_binaries() in regenerate_rust_workspace.

Path A activates ONLY after a v* tag is pushed and release.yml uploads
tarballs to github releases. Until then, Path A 404s and Path B kicks in.
This commit lays the wire — release tag is a separate user-driven action.

Verify:
- bash -n install.sh: OK
- bash -n install/lib-rust-prebuild.sh: OK
- detect_rust_target on this host: aarch64-apple-darwin
- has_prebuilt_substrate_binaries: correctly returns false on partial dev
  target (only kei-memory + kei-db-contract built locally today)

Out of scope (not done in this commit):
- Tag v0.X release to populate github tarballs (deploy step, deferred)
- Update keiseikit.dev/install.sh redirect target (downstream wiring)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 19:07:55 +08:00
Parfii-bot
825a5576e9 fix(ci): regenerate insta golden snapshots after Wave A scrub
After session-wide kit scrub (CfC/Born-like/KILL GUARD/etc → generic),
_assembler/tests/snapshots/{code-implementer,researcher}.snap held stale
expectations. CI failed on golden_code_implementer + golden_researcher
across 9 consecutive runs.

Regenerated via cargo-insta accept. Snapshot diffs are exactly the term
substitutions Wave A applied to manifests:
  - "ML/RL/CfC-adjacent" → "ML/RL specialized-node"
  - similar 1-line shifts in code-implementer.snap

cargo test --release now green across all 11 _assembler test binaries
(22+3+3+3+1+2+2+7+4+6 = 53 tests passing).

=== STATUS-TRUTH MARKER ===
shipped: functional
stubs: 0
cargo-check: PASS
cargo-test: PASS (53 tests, 0 failures)
behaviour-verified: yes
follow-up-required:
  - none — CI should turn green on next push

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 18:28:28 +08:00
Parfii-bot
49c7bb406b chore(security): scrub residual leaks — NOTICE email + HERMES patent section
NOTICE: parfionovich@keilab.ioinfo@greendragon.info
  Earlier mass-scrub missed NOTICE; now consistent with all Cargo.toml + package.json.

HERMES-MIGRATION-PLAN.md: removed 'Patent / IP considerations' section.
  Original text revealed: internal prior-art search activity, plan to file
  defensive provisional before public release of P3, reference to
  keipatent-project-specialist (private agent, not in public kit).
  Replaced with generic 'Licensing' block (MIT/Apache attribution only).

Audit confirmed remaining 'patent' mentions are legitimate:
  - LICENSE/NOTICE Apache 2.0 boilerplate (patent grant clause required)
  - README license-choice rationale
  - kei-leak-matrix source — IT IS the secret-scanner with Category::PatentIp
  - _generated/{researcher,ml-researcher,validator}.md — reference public agent 'patent-researcher'
  - kei-store/github.rs — explain RULE 0.1 rationale to kit users
  - skills/ci-scaffold — security-feature wording
  - docs/AGENT-ROLES + skills/compose-solution — 'prior-art search' as generic researcher capability

No real-secret findings (sk-/ghp_/AKIA/etc) in tracked files.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 18:19:14 +08:00
Parfii-bot
740889c422 chore: regenerate DNA-INDEX after kit changes (kei-db-contract + Wave B follow-up)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 17:59:15 +08:00
Parfii-bot
c35e1ae9ca chore(kit): wire kei-db-contract into installer + drop final #[path] hack
A1 — install.sh wiring for kei-db-contract:
- install/lib-substrate.sh substrate_core_binaries(): add kei-db-contract
  to always-copy list. End users now get the binary in ~/.cargo/bin/
  immediately after install (no manual cargo install --path needed).

A2 — Wave B follow-up: drop #[path] hack from guard_test_corpus.rs
- tests/guard_test_corpus.rs: #[path = "../src/injection_*"] mod ...
  → use kei_memory::injection_guard::scan
- Now uses Wave B's [lib] target like tests/integration.rs already does.
- 4 tests still pass.

Verified via cargo test: 18 lib + 4 corpus + 3 ingest_guard + 1 injection_unit
+ 4 dedup + 8 integration + 4 ingest_real_trace = 42 tests, all green.

=== STATUS-TRUTH MARKER ===
shipped: functional
stubs: 0
cargo-check: PASS
cargo-test: PASS (42 tests, 0 failures)
behaviour-verified: yes
follow-up-required:
  - tests/ingest_guard_tests.rs already migrated (Wave A's earlier work)
  - kei-db-contract still requires kit user to have run install.sh; existing
    installs need re-run. Kit ledger-validate should add post-install probe.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 17:55:52 +08:00
Parfii-bot
6304d1b4a6 feat(frontend-loop): /visual-loop skill + dev-ship frontend-final-gate (Wave 4+5a)
Wave 4 — /visual-loop skill (replaces deferred kei-visual-snapshot Rust primitive):
- One-command Playwright scaffolding per project (config + 2 spec templates + npm scripts + axe-core)
- Auto-discovers routes (Next.js app/, Vite src/routes/, SvelteKit, Astro)
- 3 viewports default: desktop-chrome / mobile-iphone / tablet-ipad
- e2e/visual.spec.ts: route × viewport screenshot matrix vs baseline
- e2e/a11y.spec.ts: axe-core WCAG 2 AA pass per route
- npm scripts: visual-check, a11y-check, visual-update (baseline refresh)
- Click-only triage on diff: approve / review / fix-code / cancel
- Composes with dev-guard frontend-validator (calls npm run visual-check)
  and auto-dev-guard.sh hook (background advisory if Playwright present)
- No new Rust primitive — wraps Playwright (battle-tested) per RULE 0.10 (don't reinvent)

Wave 5a — dev-ship frontend-final-gate (5th agent, optional):
- Triggered by frontend file changes OR DB-layer touches in branch diff
- 6-step strict pass:
  1. Production build (npm run build) — zero errors+warnings
  2. tsc --noEmit --strict — force strict regardless of project tsconfig
  3. kei-db-contract --strict — drift_count must be 0
  4. visual-check FULL — across all routes × all viewports
  5. a11y-check FULL — zero WCAG 2 AA violations
  6. Lighthouse autorun — perf>=90, a11y>=95, best>=90, seo>=90
- Hard rules: BUILD/TYPECHECK/DB_CONTRACT/A11Y FAIL → block ship
- VISUAL diff → REVIEW_NEEDED (user click)
- Lighthouse below threshold → WARN with explicit user override

Wave 5b (dev-start frontend-contract-designer) — SKIPPED.
  Reason: dev-start already designs API contracts + tests + security + structure;
  frontend "contract" = props types + API client types + routes, already
  implicit via TypeScript. Adding dedicated agent = duplication, not value.

frontend-validator.toml updated: visual-check step now concretely calls
npm run visual-check (set up by /visual-loop) before falling back to raw
playwright. A11y quick step added (npm run a11y-check if available).

Verify-before-commit:
- visual-loop SKILL.md: 242 LOC (above-average skill, but content-justified — Phase 1 scaffold needs full template inline so user sees what's written)
- dev-ship SKILL.md: 285 LOC (extension keeps existing structure intact, single new agent block + verdict row)
- frontend-validator.toml: format matches existing critic-perf.toml; assembler-compatible

=== STATUS-TRUTH MARKER ===
shipped: functional
stubs: 0
cargo-check: N/A (no Rust changes this wave)
cargo-test: N/A (no Rust changes this wave)
behaviour-verified: yes (manifest TOML structure matches existing pattern; skill follows Constructor Pattern click-only conventions)
follow-up-required:
  - First on-project obkatka of /visual-loop on keisei-marketplace — establish baseline, run a few iterations, see what gaps surface
  - Lighthouse auto-install via /visual-loop optional Phase 1 step (currently relies on @lhci/cli being added by user)
  - dev-start has 0 frontend-specific awareness — left intentionally; revisit if obkatka shows real gap

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 15:43:05 +08:00
Parfii-bot
f3f5f79760 feat(frontend-loop): kei-db-contract primitive + frontend-validator agent + auto-dev-guard hook
Frontend continuous-quality loop landed. Three composable cubes:

Wave 1 — kei-db-contract primitive (~870 LOC, 7 cubes per Constructor Pattern):
- Diffs SQL CREATE TABLE migrations against TypeScript type/interface declarations
- 4 drift modes: ORPHAN-SQL, ORPHAN-TS, TYPE-MISMATCH, NULL-MISMATCH
- Reuses sqlparser-rs (Apache 2.0) + regex + walkdir + serde_json + clap
- CLI: kei-db-contract <project-root> [--output json|text] [--strict]
- 5/5 integration tests pass (cargo check + cargo test green)
- Smoke-tested on keisei-marketplace: drift_count=266 across 30 tables
  (expected — marketplace uses raw better-sqlite3 without explicit row types)

Wave 2 — frontend-validator agent + dev-guard skill extension:
- New _manifests/frontend-validator.toml (substrate_role: edit-local, tools: Bash+Read+Glob+Grep)
- Agent runs: stack detect → tsc --noEmit → eslint → kei-db-contract → playwright (optional)
- Severity rules: TYPE_CHECK FAIL = block, DB_CONTRACT drift > 0 = block, lint = advisory
- skills/dev-guard/SKILL.md extended: 4th agent triggered on .tsx/.ts/.dart edits or DB-layer touches
- adaptive-depth table extended with frontend + DB-layer rows

Wave 3 — auto-dev-guard.sh hook (PostToolUse:Edit|Write):
- Trivial-edit gate: skip if delta < 30 LOC (avoid spawn fatigue)
- File-pattern match: *.tsx|*.ts|*.svelte|*.vue|*.dart OR migrations/*.sql OR src/db/** OR src/types/** OR prisma/schema.prisma OR drizzle.config.*
- Auto-runs kei-db-contract for DB-layer edits if binary on PATH
- Stderr advisory only (exit 0 always — never blocks)
- Bypass: KEI_DISABLED_HOOKS or KEI_HOOK_PROFILE in {advisory-off, minimal, off}
- Smoke-tested with synthetic Edit input (39 LOC delta on .tsx → emits advisory)
- Registered in hooks/hooks.json under PostToolUse:Write|Edit chain

Reusability map (Constructor Pattern compose):
  shared cubes: detect-stack, tsc, eslint, kei-db-contract, kei-visual-snapshot (deferred)
  orchestrators: /dev-start (pre), /dev-guard (during, NOW with frontend-validator),
                 /dev-ship (final), /site-create (init)

Verify-before-commit (RULE 0.13):
- cargo check -p kei-db-contract: PASS
- cargo test -p kei-db-contract: 5 passed
- jq . hooks/hooks.json: valid
- bash hooks/auto-dev-guard.sh < synthetic-input: works (frontend-relevant edit detected, exit 0)

=== STATUS-TRUTH MARKER ===
shipped: functional
stubs: 0
cargo-check: PASS
cargo-test: PASS (5 tests, 0 failures)
behaviour-verified: yes
follow-up-required:
  - kei-visual-snapshot primitive (Playwright wrap) — Wave 4, deferred
  - /dev-start frontend-contract-designer agent + /dev-ship frontend-final-gate — Wave 5, after Wave 1-3 obkatka
  - install.sh wiring for kei-db-contract binary
  - hermes-style emit-on-drift advisory mode

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 15:34:39 +08:00
Parfii-bot
902fb3e81a feat(kei-memory): functional schema fix + 4-wave architecture refactor
Wave A — Functional ingest fix (root cause of empty Sleep reports):
- Rewrote TraceLine struct to match real Claude Code trace JSONL:
  type (was kind), timestamp ISO8601 (was epoch ts), message Object,
  cwd / gitBranch / parentUuid / uuid / subtype / toolUseID / toolUseResult
- New src/extract.rs: extract_tool_uses + extract_tool_result walks
  message.content[] for nested tool_use / tool_result blocks
- New src/classifier.rs: explicit table classifier (tool_error, user_correction,
  retry_loop, permission_denied, tool_use:<name>, ...) replaces shallow heuristic
- New src/error.rs: KeiMemoryError enum (IO/Parse/Db) replaces semantic
  mismatch where IO error was wrapped as rusqlite::InvalidParameterName
- New src/trace_line.rs: TraceLine + helpers (cube extraction)
- Schema migration v3: events.cwd column + 3 hot-query indices
  (events.tool, events.file_path, events.ts) + UNIQUE on patterns
- New tests/ingest_real_trace.rs: synth-fixture asserts tool/file/cwd/class extraction

Wave B — Lib crate split:
- Cargo.toml: [lib] target added alongside existing [[bin]]
- src/lib.rs: pub re-export of all 18 modules
- src/main.rs: 11 mod declarations replaced by single use kei_memory::{…}
- tests/integration.rs: #[path] hack replaced by use kei_memory::{…}

Wave C — TF-IDF dedup + single-JOIN + filter_map fix:
- Schema migration v2: tokens.idf_dirty column + flag-based dedup
- index_document no longer triggers per-call recompute_idf rebuild
- top_similar uses single JOIN via vectors_for_overlapping_sessions helper
  (was N round-trips, one session_vector per candidate)
- All filter_map(|r| r.ok()) row-error swallowing replaced with ? propagation
- New tests/tfidf_idf_dedup.rs: 4 tests covering dedup behaviour, IDF emptiness,
  JOIN-pruning, empty-query safety

Wave D — Commands split + nits:
- New src/dump.rs (43 LOC) + src/stats.rs (33 LOC):
  CLI renderers extracted from commands.rs (was inline SQL + format)
- src/commands.rs: thin wrappers, -42 LOC
- src/injection_guard.rs: inline tests removed (-26 LOC), file under 200 LOC threshold
- tests/injection_guard_unit.rs (new): 4 tests in proper integration crate
- src/patterns.rs: INSERT replaced with INSERT...ON CONFLICT...DO UPDATE
  (idempotent re-ingest, uses Wave A's UNIQUE index)
- src/analyze.rs + src/coaccess.rs: filter_map row-error fixes
- src/coaccess.rs: misleading PK comment rewritten

Verify-before-commit (RULE 0.13 §"Verify-before-commit"):
- cargo check --all-targets: PASS (1 unrelated dead-code warning)
- cargo test: 42 passed, 0 failed across 9 test binaries
- STATUS-TRUTH markers aggregated at .claude/agents/_merge/kei-memory-2026-05-01/

Architect-spotted ARCH-MAJOR + ARCH-MINOR + ARCH-NIT findings addressed:
- ARCH-MAJOR Cargo.toml binary-only (Wave B)
- ARCH-MAJOR schema missing indices (Wave A v3)
- ARCH-MAJOR ingest_jsonl choke point (Wave A — extract.rs + classifier.rs)
- ARCH-MAJOR idf O(N·V) per-call rebuild (Wave C)
- ARCH-MINOR patterns no UPSERT (Wave D)
- ARCH-MINOR commands.rs houses dump+stats (Wave D)
- ARCH-MINOR classifier silent contract (Wave A)
- ARCH-MINOR IO error wrapped as rusqlite (Wave A)
- ARCH-MINOR injection_guard inline tests (Wave D)
- ARCH-MINOR tfidf top_similar N round-trips (Wave C)
- ARCH-NIT 3× filter_map(|r| r.ok()) sites (Wave C + D)
- ARCH-NIT coaccess misleading comment (Wave D)

=== STATUS-TRUTH MARKER ===
shipped: functional
stubs: 0
cargo-check: PASS
cargo-test: PASS (42 tests, 0 failures)
behaviour-verified: yes
follow-up-required:
  - tests/ingest_guard_tests.rs + tests/guard_test_corpus.rs still on #[path] hack (Wave B follow-up note, ~5 LOC)
  - dead_code warning Severity::Warn unused (pre-existing, not blocking)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 14:10:06 +08:00
Parfii-bot
0be354a920 KeiSeiKit-public — clean state
Single-commit clean baseline after security scrub of niche-tells,
project codenames, internal jargon, and contributor-email leaks.

Contents:
- 100 Rust crates (_primitives/_rust/)
- 37 agent manifests (_manifests/) + generated specs (_generated/)
- 67 user-invocable skills (skills/)
- 33 hooks (hooks/)
- Composition blocks (_blocks/)
- Documentation (docs/, README.md)
- TS adapter packages (_ts_packages/)
- Assembler (_assembler/)
- Roles (_roles/)
- Templates (_templates/)
- Forgejo CI (.forgejo/)

Author: Denis Parfionovich <info@greendragon.info>

License: see LICENSE.
2026-05-01 12:09:03 +08:00