KeiSeiKit-1.0/_blocks/stack-python-ml.md

STACK — Python ML (PyTorch / JAX)

Python is acceptable here because ML training > ~10M params is still the dominant ecosystem. Inference should still be Rust/C++/ONNX where possible.

Core: PyTorch ≥ 2.0 (compile, FlashAttn 2). pyproject.toml only — NO setup.py, NO requirements.txt as source of truth (lock via uv lock or pip-compile).

Tooling:

• ruff format + lint (replaces black / isort / flake8)
• mypy --strict on library modules; relaxed on training scripts
• pytest + pytest-asyncio for tests; synthetic-data smoke test that runs in < 5 s

Observability (non-negotiable — a silent long run with no output is a real incident we've hit):

• print(..., flush=True) on EVERY print in any script > 2 min wall-time.
• Progress every 250 steps OR every 30 s wall-time, whichever first.
• Launch via python3 -u or PYTHONUNBUFFERED=1.
• Format: [env/topo/seed] ep N: last100=X.X, time=Ys.

Reproducibility:

• Seeds fixed: torch.manual_seed(seed), np.random.seed(seed), random.seed(seed). Default [42, 137, 256] for multi-seed runs.
• Log ALL hyperparams at run start — exact param count (not "~7M"), batch, LR, seq-len, dataset hash.

Training on Modal: see deploy-modal.md. flush=True, vol.commit() after each write, checkpoints every 500 steps, .spawn() not .map(), retries=modal.Retries(max_retries=1), KILL GUARD (never stop a running job without explicit user confirmation).

Results logging: after EVERY run record in memory/{project}.md — architecture, dims, params (EXACT), data, steps, metric, time, hardware, status, cost, notes. DATA FIRST, analysis second.

Forbidden: print() without flush=True; "~7M" instead of exact param count; skipping result logging; LR schedule tuning before ablating what's unnecessary (Math-First); single-seed claims for anything that will be published or cited (need ≥ 5 seeds).