KeiSeiKit-1.0/skills/observability-setup/phase-3-scrape-ship.md

Phase 3 — Scrape + ship wiring

Produce two concrete config artefacts in the target repo:

• config/prometheus.yml (or config/otel-collector.yaml if STACK == otel-vendor)
• config/log-ship.env — env-var bundle for _primitives/log-ship.sh

3a — Emit AskUserQuestion (one call)

{
  "questions": [
    {
      "question": "Scrape / collect topology?",
      "header": "Topology",
      "multiSelect": false,
      "options": [
        {"label": "Prometheus pulls /metrics",      "description": "Prom-native. App exposes 9090. Standard for prom-grafana."},
        {"label": "OTel Collector sidecar",         "description": "Per-host collector. App → collector → backend. Uniform for logs+metrics+traces."},
        {"label": "OTel Collector central gateway", "description": "One collector pool for the cluster. HA, scales, single ingress point."},
        {"label": "Vendor agent (Datadog / BS)",    "description": "Vendor-supplied agent does discovery + shipping. Lowest ops."}
      ]
    }
  ]
}

Store as TOPOLOGY.

3b — Generate scrape config

If TOPOLOGY == "Prometheus pulls /metrics" — write config/prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s
scrape_configs:
  - job_name: "$SERVICE"
    metrics_path: /metrics
    static_configs:
      - targets: ["${SERVICE_HOST:-localhost}:${METRICS_PORT:-9090}"]
  - job_name: "node"
    static_configs:
      - targets: ["${NODE_HOST:-localhost}:9100"]

Reference: _blocks/obs-metrics.md for label cardinality budget, naming conventions. Reference Prometheus config spec [VERIFIED: prometheus.io/docs/prometheus/latest/configuration/configuration/].

If TOPOLOGY is an OTel variant — write config/otel-collector.yaml:

receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }
processors:
  batch: {}
  memory_limiter: { check_interval: 1s, limit_mib: 512 }
exporters:
  prometheusremotewrite:
    endpoint: ${PROM_REMOTE_WRITE_URL}
  otlphttp/traces:
    endpoint: ${TRACES_BACKEND_URL}
service:
  pipelines:
    metrics: { receivers: [otlp], processors: [memory_limiter, batch], exporters: [prometheusremotewrite] }
    traces:  { receivers: [otlp], processors: [memory_limiter, batch], exporters: [otlphttp/traces] }
    logs:    { receivers: [otlp], processors: [memory_limiter, batch], exporters: [otlphttp/traces] }

Reference OTel Collector spec [VERIFIED: opentelemetry.io/docs/collector/configuration/].

If TOPOLOGY == "Vendor agent" — output the vendor install snippet (Datadog Agent, Better Stack Vector config, etc.) and skip to 3c.

3c — Generate log-ship invocation

Build config/log-ship.env referencing _primitives/log-ship.sh with fields from Phase 1's LOG_TARGET:

# config/log-ship.env — env bundle for _primitives/log-ship.sh
# Source before piping app stdout:
#   set -a && . config/log-ship.env && set +a
#   ./app 2>&1 | ~/.claude/agents/_primitives/log-ship.sh --target $LOG_SHIP_TARGET --endpoint "$LOG_SHIP_ENDPOINT" --label "job=$SERVICE"

LOG_SHIP_TARGET="${LOG_SHIP_TARGET:-stdout}"      # stdout | loki | datadog | http
LOG_SHIP_ENDPOINT="${LOG_SHIP_ENDPOINT:-}"        # e.g. http://loki:3100/loki/api/v1/push
# LOG_SHIP_DD_API_KEY=...   # ← put in ~/.claude/secrets/.env or service .env — NEVER in git
# LOG_SHIP_BEARER=...       # generic HTTP target bearer — same rule

Map Phase 1's LOG_TARGET → LOG_SHIP_TARGET:

• stdout-only → stdout (no endpoint)
• file → stdout (container runtime captures; skip shipping)
• ship-loki → loki + endpoint
• ship-datadog → datadog + endpoint + LOG_SHIP_DD_API_KEY via env
• ship-http → http + endpoint + optional LOG_SHIP_BEARER

3d — Verify scrape end-to-end

Before finishing the phase, invoke _primitives/metrics-scrape.sh against the freshly instrumented app:

~/.claude/agents/_primitives/metrics-scrape.sh \
  "http://${SERVICE_HOST:-localhost}:${METRICS_PORT:-9090}/metrics" --format table

If the output is empty or the curl fails — HALT, report to user (likely Phase 2 init-call mis-wired). Do NOT proceed to Phase 4 with a silent scraper.

Verify-criterion

• config/prometheus.yml OR config/otel-collector.yaml written.
• config/log-ship.env written (with # NEVER in git comment next to any secret-var placeholder — RULE 0.8).
• metrics-scrape.sh dry-run returns > 0 lines.
• TOPOLOGY stored for Phase 5's alert-rule scope.