Initial release

docs/PLUGINS.md

@@ -0,0 +1,453 @@
+# Plugin Authoring Guide
+
+How to create custom solvers, schedulers, and guidance modes for the HOT-Step CPP engine.
+
+---
+
+## Quick Start
+
+1. Create a `.lua` file in the appropriate directory:
+   - `engine/plugins/solvers/` — ODE/SDE solvers
+   - `engine/plugins/schedulers/` — noise schedules
+   - `engine/plugins/guidance/` — CFG guidance modes
+2. Declare a metadata table (`solver`, `scheduler`, or `guidance`)
+3. Implement the required function (`step`, `schedule`, or `guide`)
+4. Restart the app — your plugin appears in the UI automatically
+
+No C++ rebuild required. The engine hot-loads all `.lua` files at startup.
+
+---
+
+## Plugin Types
+
+### Solver
+
+Solvers advance the latent state `xt` by one step along the ODE/SDE trajectory.
+
+**Metadata table:**
+
+```lua
+solver = {
+    name        = "my_solver",           -- unique internal ID (lowercase, underscores)
+    display     = "My Solver (2 NFE)",   -- name shown in UI dropdown
+    description = "A custom solver",     -- tooltip text
+    nfe         = 2,                     -- number of function evaluations per step
+    order       = 2,                     -- solver order (informational)
+    needs_model = true,                  -- true if step() needs model_fn callback
+    stateful    = false,                 -- true if solver carries state across steps
+    stochastic  = false,                 -- true if solver uses randomness (SDE)
+}
+```
+
+**Required function — single-eval solver:**
+
+```lua
+function step(xt, vt, t_curr, t_prev, n)
+    -- xt:     mutable FloatArray — current latent state (modify in-place)
+    -- vt:     read-only FloatArray — velocity at (xt, t_curr)
+    -- t_curr: float — current timestep
+    -- t_prev: float — next timestep (we step FROM t_curr TO t_prev)
+    -- n:      int — total elements in xt/vt
+
+    local dt = t_curr - t_prev
+    for i = 0, n - 1 do
+        xt[i] = xt[i] - vt[i] * dt
+    end
+end
+```
+
+**Required function — multi-eval solver** (when `needs_model = true`):
+
+```lua
+function step(xt, vt, t_curr, t_prev, n, model_fn, vt_buf)
+    -- Additional args when needs_model = true:
+    -- model_fn(xt_array, t_val): evaluates the model at (xt_array, t_val),
+    --                            writes velocity output to vt_buf
+    -- vt_buf:  mutable FloatArray — receives model_fn output
+
+    local dt = t_curr - t_prev
+    local t_mid = t_curr - 0.5 * dt
+
+    -- Save state
+    local k1 = {}
+    local xt_orig = {}
+    for i = 0, n - 1 do
+        k1[i] = vt[i]
+        xt_orig[i] = xt[i]
+    end
+
+    -- Evaluate at midpoint
+    for i = 0, n - 1 do
+        xt[i] = xt_orig[i] - 0.5 * k1[i] * dt
+    end
+    model_fn(xt, t_mid)  -- result appears in vt_buf
+
+    -- Final update using midpoint velocity
+    for i = 0, n - 1 do
+        xt[i] = xt_orig[i] - vt_buf[i] * dt
+    end
+end
+```
+
+---
+
+### Scheduler
+
+Schedulers produce a timestep sequence for the denoising trajectory.
+
+**Metadata table:**
+
+```lua
+scheduler = {
+    name        = "my_schedule",
+    display     = "My Schedule",
+    description = "Custom noise schedule",
+}
+```
+
+**Required function:**
+
+```lua
+function schedule(output, num_steps, shift)
+    -- output:    mutable FloatArray — write num_steps timestep values
+    -- num_steps: int — number of timesteps to generate
+    -- shift:     float — noise shift parameter from UI
+
+    for i = 0, num_steps - 1 do
+        output[i] = 1.0 - i / num_steps
+    end
+    apply_shift(output, num_steps, shift)
+end
+```
+
+Timesteps go from `1.0` (pure noise) to `~0.0` (clean signal). The engine appends a final `0.0` step automatically — your schedule should produce `num_steps` values, not `num_steps + 1`.
+
+**Common helper — shift warp:**
+
+```lua
+function apply_shift(ts, n, shift)
+    if shift == 1.0 then return end
+    for i = 0, n - 1 do
+        local t = ts[i]
+        ts[i] = shift * t / (1.0 + (shift - 1.0) * t)
+    end
+end
+```
+
+---
+
+### Guidance
+
+Guidance modes control how the conditional and unconditional model predictions are combined.
+
+**Metadata table:**
+
+```lua
+guidance = {
+    name        = "my_guidance",
+    display     = "My Guidance",
+    description = "Custom guidance mode",
+}
+```
+
+**Required function:**
+
+```lua
+function guide(pred_cond, pred_uncond, guidance_scale, result, Oc, T, norm_threshold)
+    -- pred_cond:      read-only FloatArray — conditional velocity prediction
+    -- pred_uncond:    read-only FloatArray — unconditional velocity prediction
+    -- guidance_scale: float — the guidance scale (w) from the UI
+    -- result:         mutable FloatArray — write the guided velocity here
+    -- Oc:             int — output channels per timestep frame
+    -- T:              int — number of timestep frames (n = Oc * T)
+    -- norm_threshold: float — APG norm threshold from the UI
+
+    -- Route through APG for stability (STRONGLY RECOMMENDED)
+    apg(pred_cond, pred_uncond, guidance_scale, result, Oc, T, norm_threshold)
+end
+```
+
+> **⚠️ Important:** Always route through `apg()` for the base guidance computation. Raw linear interpolation (`result = uncond + w * (cond - uncond)`) produces severe audio artifacts (static, underwater sound, frequency distortion). The `apg()` function provides momentum smoothing, perpendicular projection, and norm thresholding that are essential for stable audio output. If your guidance mode needs custom math, apply it as a correction on top of the APG result.
+
+**Available globals in guidance plugins:**
+
+| Global | Type | Description |
+|--------|------|-------------|
+| `step_idx` | int | Current step index (0-based) |
+| `total_steps` | int | Total number of denoising steps |
+| `dt` | float | Current timestep delta (t_curr - t_next) |
+| `t_curr` | float | Current timestep value |
+| `params` | table | User-configured parameter values |
+
+---
+
+## The `apg()` Bridge
+
+The `apg()` function is the native C++ APG (Analytical Perpendicular Guidance) implementation, exposed to Lua guidance plugins. It handles:
+
+1. **Perpendicular projection** — removes the component of `(cond - uncond)` parallel to `uncond`, keeping only the steering signal
+2. **Momentum smoothing** — exponential moving average across steps to prevent jitter
+3. **Norm thresholding** — caps per-channel magnitudes to prevent blowup
+
+**Signature:**
+
+```lua
+apg(pred_cond, pred_uncond, guidance_scale, result, Oc, T, norm_threshold)
+```
+
+All guidance plugins have access to this function. It is registered automatically on first use.
+
+---
+
+## Advanced: The `post_step()` Hook
+
+For guidance modes that need to run extra model evaluations *after* the solver step (e.g., manifold projection), guidance plugins can declare a `post_step()` function. The engine detects this at load time and calls it after each solver step.
+
+**When to use this:**
+- Your guidance technique requires iterative refinement of the latent state
+- You need to evaluate the model at positions different from the main solver trajectory
+- The technique calls the model with conditioning and unconditioning separately
+
+**Performance warning:** Each call to `eval_cond` or `eval_uncond` runs a full model forward pass. This is expensive — use sparingly.
+
+```lua
+function post_step(xt, t, n, eval_cond, eval_uncond, vt_cond, vt_uncond)
+    -- xt:          mutable FloatArray — latent state after solver step (modify in-place)
+    -- t:           float — timestep we just stepped TO (t_next)
+    -- n:           int — total elements in xt
+    -- eval_cond:   function(xt_arr, t) — runs model with conditioning, writes to vt_cond
+    -- eval_uncond: function(xt_arr, t) — runs model without conditioning, writes to vt_uncond
+    -- vt_cond:     mutable FloatArray — output buffer for conditional velocity
+    -- vt_uncond:   mutable FloatArray — output buffer for unconditional velocity
+
+    -- Example: one iteration of manifold projection
+    local a = math.abs(dt) / 2.0
+
+    eval_uncond(xt, t)                        -- fills vt_uncond
+    for i = 0, n - 1 do
+        xt[i] = xt[i] - a * vt_uncond[i]     -- push away from uncond manifold
+    end
+
+    eval_cond(xt, t)                          -- fills vt_cond
+    for i = 0, n - 1 do
+        xt[i] = xt[i] + a * vt_cond[i]       -- pull toward cond manifold
+    end
+end
+```
+
+The `post_step` hook has access to the same globals as `guide()` (`step_idx`, `total_steps`, `dt`, `t_curr`, `params`).
+
+The hook is **not called on the final step** (the latent is about to be decoded, so further projection is pointless).
+
+---
+
+## Parameter Schema
+
+Plugins can declare user-facing parameters that appear in the UI. Parameters are defined in the `params` array of the metadata table.
+
+### Slider
+
+```lua
+{ key = "strength", type = "slider", label = "Strength",
+  default = 0.5, min = 0.0, max = 1.0, step = 0.01,
+  hint = "Controls the effect intensity" }
+```
+
+### Select (Dropdown)
+
+```lua
+{ key = "mode", type = "select", label = "Mode",
+  default = "fast",
+  options = {
+      { value = "fast",    label = "Fast" },
+      { value = "quality", label = "Quality" },
+  },
+  hint = "Choose between speed and quality" }
+```
+
+### Toggle
+
+```lua
+{ key = "enabled", type = "toggle", label = "Enable Feature",
+  default = false,
+  hint = "Turn this feature on or off" }
+```
+
+### Conditional Visibility
+
+Parameters can be shown/hidden based on another parameter's value:
+
+```lua
+{ key = "sub_param", type = "slider", label = "Sub-Parameter",
+  default = 1.0, min = 0.0, max = 5.0, step = 0.1,
+  visible_when = { key = "mode", equals = "quality" },
+  hint = "Only visible when Mode is set to Quality" }
+```
+
+### Transform Expressions
+
+The `transform` field allows the UI to apply a mathematical transformation to the displayed value before sending it to the plugin. This is useful when the internal value differs from what the user sees:
+
+```lua
+{ key = "sigma", type = "slider", label = "Noise σ",
+  default = 5, min = 0, max = 100, step = 1,
+  transform = "value * 0.05",
+  hint = "Displayed as 0-100, sent to plugin as 0-5" }
+```
+
+### Reading Parameters
+
+Parameters are available via the `params` global table, keyed by their `key` field:
+
+```lua
+local strength = (params and params.strength) or 0.5
+local mode     = (params and params.mode) or "fast"
+local enabled  = (params and params.enabled) or false
+```
+
+Always provide a fallback default with `or` — `params` may be `nil` if no parameters have been set.
+
+---
+
+## FloatArray
+
+All array data passes between C++ and Lua via the `FloatArray` userdata type. This is a **zero-copy** bridge — Lua reads and writes the same memory that the C++ engine uses.
+
+**Indexing:** 0-based (matching C++ convention, not Lua's typical 1-based).
+
+```lua
+-- Read
+local val = xt[i]
+
+-- Write (only on mutable arrays)
+xt[i] = val
+
+-- Length
+local n = #xt
+```
+
+Read-only arrays (like `pred_cond` and `pred_uncond` in guidance) will raise an error if you try to write to them.
+
+---
+
+## Available Globals
+
+### Solver globals
+
+| Global | Type | Description |
+|--------|------|-------------|
+| `step_index` | int | Current step index |
+| `batch_n` | int | Number of batch elements |
+| `n_per` | int | Elements per batch element |
+| `params` | table | Plugin parameters |
+
+### Guidance globals
+
+| Global | Type | Description |
+|--------|------|-------------|
+| `step_idx` | int | Current step index (0-based) |
+| `total_steps` | int | Total denoising steps |
+| `dt` | float | Timestep delta |
+| `t_curr` | float | Current timestep |
+| `params` | table | Plugin parameters |
+
+---
+
+## Sandbox
+
+Each plugin runs in an isolated Lua 5.4 VM with:
+
+**Available:** `math`, `string`, `table`, `print`, `type`, `pairs`, `ipairs`, `tonumber`, `tostring`, `require` (for companion data files)
+
+**Blocked:** `os`, `io`, `debug`, `dofile`, `loadfile` — no filesystem access, no shell commands, no process control.
+
+The `require()` function works for loading companion Lua data files (e.g., precomputed constants in a separate `.lua` file in the same directory), but cannot load C modules.
+
+---
+
+## Complete Examples
+
+### Minimal Solver
+
+```lua
+-- my_solver.lua
+solver = {
+    name        = "my_solver",
+    display     = "My Solver",
+    description = "Simple Euler variant",
+    nfe         = 1,
+    order       = 1,
+    needs_model = false,
+}
+
+function step(xt, vt, t_curr, t_prev, n)
+    local dt = t_curr - t_prev
+    for i = 0, n - 1 do
+        xt[i] = xt[i] - vt[i] * dt
+    end
+end
+```
+
+### Scheduler with Custom Curve
+
+```lua
+-- my_schedule.lua
+scheduler = {
+    name        = "my_schedule",
+    display     = "Quadratic",
+    description = "Quadratic timestep spacing",
+    params      = {
+        { key = "power", type = "slider", label = "Power",
+          default = 2.0, min = 1.0, max = 4.0, step = 0.1 },
+    },
+}
+
+function schedule(output, num_steps, shift)
+    local p = (params and params.power) or 2.0
+    for i = 0, num_steps - 1 do
+        local frac = i / num_steps
+        output[i] = (1.0 - frac) ^ p
+    end
+    -- Apply shift warp
+    if shift ~= 1.0 then
+        for i = 0, num_steps - 1 do
+            local t = output[i]
+            output[i] = shift * t / (1.0 + (shift - 1.0) * t)
+        end
+    end
+end
+```
+
+### Guidance with APG + Custom Logic
+
+```lua
+-- my_guidance.lua
+guidance = {
+    name        = "my_guidance",
+    display     = "My Guidance",
+    description = "Warm-up guidance with linear ramp",
+    params      = {
+        { key = "warmup_steps", type = "slider", label = "Warm-Up Steps",
+          default = 3, min = 0, max = 10, step = 1 },
+    },
+}
+
+function guide(pred_cond, pred_uncond, guidance_scale, result, Oc, T, norm_threshold)
+    local warmup = (params and params.warmup_steps) or 3
+    local progress = math.min((step_idx or 0) / math.max(warmup, 1), 1.0)
+    local effective_scale = 1.0 + (guidance_scale - 1.0) * progress
+
+    apg(pred_cond, pred_uncond, effective_scale, result, Oc, T, norm_threshold)
+end
+```
+
+---
+
+## Tips
+
+- **Test with Euler + Linear first.** The simplest solver/scheduler combination isolates your plugin's behaviour.
+- **Use `print()` for debugging.** Output goes to the terminal panel in the app.
+- **Lua tables for scratch space.** If you need temporary arrays, use Lua tables: `local tmp = {}; for i = 0, n-1 do tmp[i] = 0 end`. They're slower than FloatArrays but work for intermediate calculations.
+- **Be careful with the loop range.** FloatArrays are 0-indexed: `for i = 0, n - 1 do ... end`.
+- **Guidance plugins: always use `apg()`.** Raw math without APG produces audio artifacts. Apply your custom logic as a delta on top.
+- **Stateful plugins** can use file-level `local` variables to carry state across steps (e.g., previous velocity buffers, error accumulators). These reset when the plugin is reloaded.

docs/RELEASING.md

@@ -0,0 +1,116 @@
+# Releasing HOT-Step CPP — agent runbook
+
+How to cut and publish a release, plus the non-obvious gotchas. Written for an
+agent (or human) driving the process with the `gh` CLI on Windows/Git-Bash.
+
+## TL;DR
+
+Releases are cut by **pushing a `vX.Y.Z` tag**. The `Release` workflow builds
+every platform and creates a **draft** GitHub Release; you review and publish it.
+A separate `Cache Warm` workflow keeps the engine build cache on `master` so
+release builds take ~10–15 min instead of ~1.5h for the CUDA jobs.
+
+## Prerequisites
+
+- `gh` authenticated (as `scragnog`).
+- On `master`, working tree clean, everything committed **and pushed**.
+- Pick a semver version **without a hyphen**: `vX.Y.Z` (hyphens are reserved for
+  test/pre-release tags — see gotchas).
+
+## 1. (Optional) Compile-test before releasing
+
+To verify CI compiles without cutting a real release, push a throwaway
+**hyphenated** tag — it triggers the same build pipeline but is ignored by the
+changelog logic:
+
+```bash
+git tag -a vX.Y.Z-CI-Test -m "compile test" && git push origin vX.Y.Z-CI-Test
+# ...watch it (section 3)... then delete when done:
+gh release delete vX.Y.Z-CI-Test --cleanup-tag --yes   # removes draft + remote tag
+git tag -d vX.Y.Z-CI-Test
+```
+
+Re-pushing the **same** `-CI-Test` name is free (delete remote+local, recreate,
+push). Tags cannot be renamed.
+
+## 2. Cut the release
+
+```bash
+git tag -a vX.Y.Z -m "vX.Y.Z — <one-line summary>"
+git push origin vX.Y.Z
+```
+
+This triggers `Release` → builds Windows (cuda13.1 / cuda12.8 / vulkan / cpu),
+Linux (same four), macOS (Metal) → creates a **draft** release with **22 assets**
+(11 archives + 11 `.sha256`).
+
+To change the commit or re-run: delete + re-push the tag (it rebuilds).
+
+## 3. Monitor
+
+```bash
+gh run list --limit 5
+gh run view <run-id>                 # per-job status + timings
+```
+
+To read a **failed/cancelled job's** log while the run is still in progress
+(`gh run view --log` won't show it yet), pull it from the API:
+
+```bash
+MSYS_NO_PATHCONV=1 gh api repos/scragnog/HOT-Step-CPP/actions/jobs/<job-id>/logs > log.txt
+```
+
+`MSYS_NO_PATHCONV=1` stops Git-Bash rewriting the leading-slash API path into a
+filesystem path.
+
+## 4. Publish
+
+The workflow leaves the release as a **draft**. Verify the asset count (18) and
+the `What's Changed` notes, then:
+
+```bash
+gh release view vX.Y.Z --json assets --jq '.assets | length'   # expect 22
+gh release edit vX.Y.Z --draft=false --latest
+```
+
+## 5. Cleanup
+
+Delete any leftover test tags and their drafts (see section 1).
+
+## Build caching — why releases are fast (and how it breaks)
+
+- **GitHub Actions caches are ref-scoped.** A cache saved by one tag run is NOT
+  visible to a different tag run — only **default-branch (`master`) caches** are
+  visible to every run, including release tags. So releases can only reuse a
+  cache that was created on `master`.
+- **`.github/workflows/cache-warm.yml`** builds the engine on `master` (when
+  `engine/ggml` or `engine/CMakeLists.txt` change, or via manual dispatch) and
+  saves the build dir under the **same cache keys** `release.yml` uses. Release
+  runs restore it and skip the CUDA compile (the long part).
+- **Timings:** cold (no master cache) CUDA jobs ≈ 1.5h each; warm ≈ 7–13 min.
+- **If CUDA suddenly rebuilds slow:** the master cache is missing/stale. Re-warm
+  it: GitHub → Actions → **Cache Warm** → *Run workflow* (on `master`). It also
+  auto-runs when `engine/ggml`/`CMakeLists.txt` change.
+- Cache reuse depends on git-restored source mtimes (incl. the **ggml submodule**
+  — its `.cu` files live in the submodule's own history, not the superproject).
+
+## Gotchas / lessons learned
+
+- **Windows runner is pinned to `windows-2022`.** Do NOT switch to
+  `windows-latest` — that's windows-2025, whose MSVC (`_MSC_VER >= 1950`) is
+  rejected by CUDA 12.8/13.1 `nvcc` (`host_config.h`: VS 2017–2022 only).
+- **Any pushed `v*` tag triggers the Release pipeline.** Use `vX.Y.Z` for
+  releases and `-CI-Test` (or other hyphenated) tags for throwaway checks; delete
+  them afterward. Don't push local feature tags that match `v*`.
+- **Changelog range** = commits since the previous **non-hyphenated** tag
+  (`git describe ... --exclude '*-*'`). This is why a stray `vX-CI-Test` tag must
+  not be treated as a release; the exclude guard handles it, but still clean up.
+- **The release is a draft** — it does not auto-publish. Review before going live.
+- **Cache key** = `cmake-<runner>-<variant>-<hash(engine/ggml, CMakeLists)>`.
+  Changing the runner image invalidates it (compiler abs-paths bake into
+  `CMakeCache.txt`); the key includes the runner image to prevent stale restores.
+
+## Reference
+
+- Workflows: [`.github/workflows/release.yml`](../.github/workflows/release.yml),
+  [`.github/workflows/cache-warm.yml`](../.github/workflows/cache-warm.yml)

docs/TRAINING.md

@@ -0,0 +1,160 @@
+# Training System — Architecture & Continuation Guide
+
+*The complete map of HOT-Step's native training system (Training Studio). Written as the handoff/continuation doc: everything an agent or contributor needs to keep working on this subsystem in a fresh session. Built 2026-07-27/28; all measurements from an RTX 5090 (32 GB, sm_120).*
+
+## What it is
+
+End-to-end adapter training **entirely in C++/GGML — no Python anywhere**:
+
+```
+Dataset creation → Tensor preprocessing → LM LoRA training (0.6B/1.7B/4B)
+                                        → DiT LoRA training (LoKR planned)
+                                        → Pure-LM audition (A/B, no DiT influence)
+```
+
+Side-Step (`D:\Ace-Step-Latest\Side-Step`, local) is the Python reference implementation this system reaches parity with; its dev sanctioned porting. Dataset/sidecar formats are byte-compatible with Side-Step; tensor caches deliberately are not (safetensors, not pickle).
+
+## Component map
+
+| Piece | Engine | Server | UI |
+|---|---|---|---|
+| Dataset creation | — (uses ace-server `/understand` optionally, legacy) | `routes/training.ts` + `services/training/{sidecarIO,datasetScan,labelStore,essentiaClient,enhanceService,captionPrompt,datasetBuilder,labelingQueue}.ts` | `training-studio/{DatasetList,NewDatasetWizard,SampleGrid,SampleDrawer,LabelPanel,EnhancePanel,BuildPanel}.tsx` |
+| Preprocess | `ace-train preprocess` (`engine/src/train/preprocess-*.h`, `st-write.h`) | `preprocessRunner.ts`, `preprocessStatus.ts`, `aceTrain.ts` | `Preprocess{Panel,OptionsForm,VariantCard}.tsx` |
+| LM training | `ace-train train-lm` (`lm-*.h`: graph/optim/ckpt/bf16/vram/data/extract/export/selftest/train-run) | `trainLmRunner.ts`, `trainLmStatus.ts` | `TrainPanel.tsx`, `TrainLmForm.tsx`, `TrainingChart.tsx`, `TrainingRunStats.tsx` |
+| DiT training | `ace-train train-dit` (`dit-*.h`: same family) | `trainDitRunner.ts`, `trainDitStatus.ts` | `TrainDitForm.tsx` (+ shared chart/stats) |
+| Audition | `POST /codes-decode` on **ace-server** (`hot-step-server.cpp`, ~174 lines) | `audition{Service,Runner,Store}.ts` | `AuditionCard.tsx`, `AuditionPlayer.tsx`, `LmAdapterPicker.tsx` |
+| Engine lifecycle | — | `services/aceEngineProcess.ts` (stop/restart with epoch-guarded respawn cancellation) | engine-paused banners |
+
+Frozen contracts (types, routes, JSONL event schemas, CLI) are duplicated verbatim between `server/src/services/training/types.ts` and `ui/src/services/trainingApi.ts` — **kept in sync by hand, deliberately**.
+
+Job model: one global promise-chain queue (`labelingQueue.ts`), SSE streams with replayable capped buffers, `_meta.json` persistence, `TrainingMetricEvent` for training numbers. GPU jobs stop the ace-server child first (`stopEngine`, default on) and restart it in a `finally`.
+
+## Key design facts (the ones that bite)
+
+**Data & labeling**
+- Sidecars (`<stem>.txt`) live **next to the audio**; `dataset.json` in the source folder — Side-Step-compatible on purpose. Studio-private state (audio_codes, provenance, raw analyzer results) lives in `server/data/training/datasets/<slug>/labels/`.
+- Labeling is engine-free: Essentia (BPM/key, cached by size+mtime) ∥ Genius lyrics (relaxed collab matching — "Electric Callboy & BABYMETAL" broke exact primary-artist equality) → Gemini captions **with the audio attached** (96 kbps MP3 inline; the local caption is *omitted* from the prompt to avoid anchoring). `ace-understand` is out of the default flow (weak captions, hallucinated lyrics, J-pop prior); still reachable via API `useUnderstand:true`.
+- Dataset language is **declared, not detected** (`default_language`, forced-write on every label pass).
+- Lyrics *absence* never writes `is_instrumental` — only lyrics presence writes `false`.
+
+**FSQ (critical correctness area)**
+- `engine/src/fsq-quant.h` is the single source of truth. The reference path is **ResidualFSQ `preserve_symmetry`**: soft clamp `c=1+1/(L−1)` → `tanh(z/c)·c` → hard clamp → `floor((L−1)(w+1)/2+0.5)`. NOT `FSQ.bound` (that branch never executes in vqp). Encode verified 13046/13046 vs the checkpoint's own tokenizer.
+- `verify-hooks.ps1` has 2 FSQ hooks — upstream syncs must not revert this.
+- Side-Step's stored `lm_codes.jsonl` are a **bf16 ceiling** (~85% identity); engine-F32 vs fp32-reference is the honest gate (99.0%).
+
+**LM trainer**
+- Hand-rolled loop (`lm-optim.h`): ggml-opt can't grad-clip and its grad accessor crashes with dynamic graphs. Persistent grad accumulators indexed by forward-node order, **in-graph** global-norm clip at 1.0, AdamW nodes, Side-Step-exact LR schedule (`lr(0)=0`, 5% warmup, cosine→0.1).
+- ggml gap catalogue: `out_prod` F32-only (the central constraint), no backward for flash-attn / `SET_ROWS` (KV cache) / fused swiglu; CE labels are dense one-hot; `CONCAT` has no backward (use `ACC`); rms_norm backward is wrong on **non-contiguous (permuted) inputs** — norm before permute.
+- 4B fits ~12 GB via per-layer checkpoint segments + a transient per-layer F32 window + chunked-CE head (BF16 embedding). Naive path preserved byte-identical for 0.6B/1.7B.
+- `--weights bf16` (experimental): backward surgery rewrites `out_prod(W_f32,gT)` → `mul_mat(cont(transpose(W_bf16)),g)`. **1.256× at 4B** measured; gradient cos ~0.9992 vs F32 (BF16 rounding compounds sub-linearly with depth; partly ggml-cuda's dst-BF16 rounding, only fixable by a vendor patch). Zero vendor patches shipped; `engine/ggml` submodule is clean upstream.
+- `--batch` intentionally **not built** — measured amortisable overhead below the bar where it matters (4B: 9.3% < 10%); flag exits 2 citing the numbers.
+- Defaults (Side-Step parity, from Rob's real runs): target loss **4.0** (not 0.4!), GA 2, epochs 75, milestone step 1.0, lr 1e-4.
+- Self-test: `ace-train train-lm --self-test` — T1–T13 (+T14–T17 bf16, report-not-gate). T5/T4 finite differences run **TF32-off in a child process** (cuBLAS TF32 noise defeats central differences; gradients were always right). Known pre-existing 4B rough edges: T11e/T12 slightly over bars (BF16-dtype-resolution class).
+
+**DiT trainer**
+- Trainable graph is **bit-identical** to the inference forward (17 debug-tensor diff = 0.0). Unfused load via `g_dit_load_no_fuse` (mirrors `g_qwen3_load_no_fuse`; the zero-stub-adapter trick is dead).
+- F32 mirror streamed from the GGUF via CPU-backend load (a GPU-copy mirror transiently costs +5–8 GB and OOMs cards the steady state fits).
+- Crop-window training (random window/step). No flash-attn backward ⇒ O(S²·Nh) retained softmax ⇒ full-song full-depth ≈ 68 GB: **impossible**; crop ~1000–1250 frames max at 24 GB full depth; `--layers` top-K ladder for smaller cards; <16 GB refused.
+- Product loss = flow_snr (window-normalized — batch-1 mean normalization is silently a no-op, the trap) + channel_balance from `channel_stats.json`; lr default **5e-4** (1e-4 measurably does not train).
+- Defaults: **LoRA** epochs 400, r128/α256, lr 5e-4, GA 4; **LoKR** epochs **250**, dim512/α512/factor 6, lr **2e-3**, GA **4**, target-loss 0.6, loss-weighting `none`. Genre ratio 30%, target-MLP **on** for both. DiT target-loss 0.4 requires Side-Step-length runs; epoch cap is the practical stop.
+- **LoKR lr and grad-accum move together** (2026-07-30 five-run A/B on gunship_unicorn). The old 1e-2 @ GA 20 reproduced Side-Step's effective batch of 20 by accumulation; 2e-3 @ GA 4 is the same effective LR per sample under linear scaling and measured *identical* epochs-to-target (227 vs 228) with better-behaved gradients (median grad-norm 0.062 vs 0.031 — the √5 a 5× smaller batch predicts — and no warmup spike, where GA 20 peaked at 13.5 on epoch 1). Setting one without the other is an untested config.
+- **The 250-epoch LoKR horizon is a schedule fix, not a shortcut.** Every 400-horizon run stopped with the cosine only halfway down (LR still ~50% of peak), so it never decayed *into* the target. 250 cut epochs-to-0.6 from 228 → 203 and audio-seconds-to-target by 14% — the only one of five knobs tested that reduced the work required rather than rearranging it. Don't go much lower: that run used 203 of its 250, and a horizon under the epochs actually needed strands the LR at its 10% floor with the tail unfinished.
+- **Time-to-target is set by audio-seconds seen, not epochs or update count.** ~85,000 audio-seconds to reach 0.6, stable to ±5% across a 5× change in update count and a 1.8× change in window length. Changing grad-accum or crop length only re-slices it; only the schedule shape (above) and engine throughput move it.
+- Adapter output: PEFT dir, loads through **both** `adapter-merge.h` and runtime paths (round-trip reproduces trainer loss; that check is a permanent gate). Exported **BF16** since 2026-07-30 (LyCORIS/Side-Step parity; halves a dim-512 census from 872 → 436 MB). LK5 gates the dtype — it compares alpha as a *bit pattern* against the writer's own rounding, so an alpha that is not BF16-exact still fails honestly.
+
+**LoKR `apply()` — never let the token axis reach `ne2` of a mul_mat** (2026-07-30, `docs/plans/2026-07-30-dit-trainer-step-profile.md`)
+- **The single largest perf defect found in this trainer: 9.7×.** LoKR training ran at 1902 ms/step where the same run with a LoRA took 116 ms — 16.4× — and the whole gap was `DitLokrAdapter::apply()`.
+- **Mechanism.** Both kron contractions have a 2-D trainable factor (`ne2 == 1`) as src0. If src1 carries the token count in `ne2`, ggml emits the weight gradient as `out_prod(src1, grad)` with `dst->ne[2] == S`, and ggml-cuda's `out_prod` takes its `dps2 > 1` **fallback**: one `cublasSgemm` per token (`out-prod.cu:96-108`). One `OUT_PROD [512,5,344,1]` node = 344 launches of a 512×5 GEMM; ×189 such nodes ≈ 65,000 micro-GEMMs per micro-step from that shape alone, plus a `repeat_back` to reduce the per-token gradient slabs. Op histogram: 80% `OUT_PROD` + 8% `REPEAT_BACK`.
+- **Fix (free).** `[in_n, in_m, S]` and `[in_n, in_m·S]` are the SAME BYTES, so folding the token axis into the column count is a pure `ggml_reshape_2d` — it leaves `ne2 == 1`, takes the strided-batched fast path in one call, and removes the `repeat_back` entirely. Restore 3-D only around the `permute`s, which genuinely need it.
+- **Generalise this.** Any adapter or side-path whose trainable factor is 2-D must keep tokens out of `ne2`, or it silently buys the per-token fallback. A LoRA never hit it because its A/B matmuls are already 2-D.
+- Gated by **LK3** (kron-matvec vs materialized-kron reference, 2.86e-08) and **LK4** (finite differences). A loss-level A/B is the *wrong* check here and misled once: the VRAM auto-fit picked crop 692 vs 694 between the two runs, so the losses differed for reasons unrelated to the change. Pin `--crop` **and** `--layers` if you ever want one.
+- **`--profile-step <n>` / `--profile-ops`** are the tools that found it, and they stay: per-micro-step buckets (assemble/upload/build/backward/alloc/compute/readback/free) plus graph node and scheduler split counts, and a per-node op/shape histogram for one warmed-up step. `dit_train_log.json` now carries `runtime.backend` / `graph_nodes` / `sched_splits` / `profile_ms`, so a finished run explains itself — before this, "was that run even on CUDA?" was unanswerable after the fact.
+
+**Muon optimizer** (2026-07-30, `--optimizer adamw|muon`, shared by both trainers via `lm-optim.h`)
+- **DEFAULT for the DiT** since the ear test. Measured on gunship_unicorn (LoKR dim512+MLP, 32 layers, crop/depth pinned): **161 epochs to ma5 0.6 vs AdamW's 227** — 1.41× more sample-efficient — and ~1.23× on wall-clock once bucketed. Rob's own run reached 0.6 in ~5 min and the adapter was judged perfect by ear. AdamW is unchanged and one dropdown away.
+- **Per PARAMETER, not global.** 2-D parameters with a short side ≥ `--muon-min-dim` (16) get orthogonalised-momentum (Newton-Schulz) updates; everything else stays on AdamW. A LoKR `w1` is `[4,5]` — orthogonalising a 4-dimensional subspace is vacuous — so on a DiT LoKR run the split is **448 Muon / 352 AdamW**. Every published Muon uses this hybrid.
+- **The LR does NOT mean AdamW's.** Muon's update is normalised by construction, so per element it is ~20× smaller; `--muon-lr-scale` multiplies the shared schedule for Muon parameters only. Sweep on the DiT: 5 undershoots, **20 matches**, 50 overshoots. `--muon-lr-scale` and the base `--lr` are not independent.
+- **Free VRAM:** one momentum buffer instead of two — 16315 → 15443 MB trainer-owned at LoKR dim512 (**872 MB**). `dit-vram.h` still charges for both, so the auto-fit is conservative, not wrong.
+- **Bucketing is what makes it affordable, and its knob is non-monotone.** Same-shape parameters are batched into one Newton-Schulz. Optimizer step, LoKR dim512+MLP: **298.6 ms** (bucket 1) → 136.1 (8) → **138.0 (16, default)** → 187.3 (64) → 201.2 (128), against AdamW's **19.2 ms**. Larger buckets are WORSE because the gather is a left-folded `ggml_concat`, quadratic in bucket size. ~55 ms of what remains is irreducible NS FLOPs (~2.8 TFLOP/step); killing the concat (allocate accumulators/momenta as slabs of one per-bucket tensor) is the remaining win.
+- **LM trainer: wired and functional, NOT yet defaulted.** All 392 LoRA parameters qualify at rank 16 (**the short side of a LoRA `A` is the rank** — a rank-8 adapter would fall entirely to AdamW; the startup line reports the split). 8-epoch smoke: loss 7.193 vs AdamW's 7.581 at +14% step time. That window is far too short to conclude anything — the DiT looked like parity at 10 epochs and was 1.41× ahead at 200. A run to target 4.0 is the honest test.
+- Gated by **MU1**: update vs a double-precision host reference (CPU 5.7e-05, bar 1e-4; CUDA reported not gated — cuBLAS TF32 floors five chained iterations at ~4e-3), semi-orthogonality of **the graph's own** O, and a bucket case with grads spread ×8/×1/×⅛ that catches a global-instead-of-per-slab Frobenius norm.
+
+**DiT micro-batching + gradient checkpointing** (2026-07-29, `docs/plans/2026-07-29-dit-batching-checkpointing.md`)
+- `--batch <n>` (default **1** = OFF, range 1-16): crops per micro-batch, drawn from that many DIFFERENT songs (Side-Step DataLoader parity — B same-song crops was never on the table). Clamped down to the dataset's own song count with a `warn` when the variant has fewer songs than requested; further clamped by ggml's CUDA `REPEAT_BACK` cap on `Nkv·max(S,enc_S)·B` (GQA head-expansion backward — **both** attentions expand, and cross-attention's token axis is the dataset's padded `enc_S`, not the crop), and by the C4 auto-fit's own VRAM order (crop shrinks first, then checkpoint segments rise, then B itself gives with a warn — **before** depth is ever touched, since depth is the quality axis). `--grad-accum` counts **micro-batches**, not samples — effective samples/optimizer-step is `batch × grad-accum`.
+- **Why the batch default is 1** (changed 2026-07-29 from 5, on the measurement below): batching is ~2.5× SLOWER at full depth on a 32 GB card and ~2.4× faster on shallow / partial-depth runs, so it ships off and is opt-in for the shallow case. The same default is mirrored in `DitTrainArgs`, the `ace-train` usage text, the `/api/training/train-dit` route fallback, `TrainDitOptions` (both mirror files) and `TRAIN_DIT_DEFAULTS` (which `TRAIN_DIT_LOKR_DEFAULTS` spreads, so LoKR inherits it). `--ckpt` stays at 1 (auto), and auto resolves to a single unsegmented run whenever full depth fits — `seg_candidates()` tries 1 first — so the default pair is the monolithic graph.
+- **Mixed-length batches mask the pad in ATTENTION, not just in the loss.** When any element of a micro-batch is padded (a song shorter than the crop), `dit_batch_assemble` switches the self-attention mask from the shared `[S,S]` broadcast to a per-element `[S,S,1,B]` pair: `sa` (sliding window + padded KV columns) for `layer_type 0` and `sa_pad` (padded KV columns only, no window) for `layer_type 1`, which otherwise takes no self-attention mask at all. Without this the pad reaches every valid query as an attention KEY and from there the adapter gradients — the design-B4 loss mask cannot stop it, since it only zeroes the padded frames' own loss contribution. Padded QUERY rows are deliberately left unmasked (a fully-masked softmax row is NaN); they are inert anyway, because a zero output gradient contributes exactly zero to dQ/dK/dV. Self-test rungs SB3/SC2 gate it by dirtying the padded INPUT and requiring loss and every adapter gradient to move by EXACTLY 0. The unpadded case keeps the `[S,S]` fast path, and B=1 can never pad.
+- **Grad-accum weighting is an element share.** A micro-batch's `t_lossgrad` is `elems_in_micro_batch / elems_in_window`, not `1/n_micro_batches`: the in-graph `t_lw` already normalises within a micro-batch, so `sum_mb (nb/wlen) * loss_mb` telescopes to Side-Step's global mean-of-per-sample-means over the whole optimizer window. The two agree while every micro-batch is full; a short tail of `nb < B` elements used to be over-weighted by `B/nb`. At B=1 the two forms are algebraically identical, so the `--batch 1 --ckpt 0` anchor is untouched.
+- `--ckpt <n>` (default **1** = auto): hand-rolled segmented-recompute checkpointing — `0` disables it (byte-identical to the pre-batching monolithic graph), `1` lets the VRAM fit pick a segment count, `2-32` pins it. Mechanism: one no-grad forward over the whole trained stack to capture segment-boundary activations, then segments rebuilt LAST→FIRST with grads, each feeding the next via a `spike-s3`-style surrogate loss (`sum(seg_out ⊙ G)`) seeded from the boundary gradient above it; adapter grads accumulate in-place across every segment's graph compute (`lm-optim.h` untouched — it never needed to change, since PARAM-flagged boundary tensors get their own grad slot through the existing `ggml_build_backward_expand` machinery). Bit-exact vs `--ckpt 0` at the same seed/batch (self-test SC1-SC3).
+- JSONL: `step.micro` counts MICRO-BATCHES (not samples — samples is `micro × batch`, modulo a short tail). `start` cannot know the resolved batch/segment count (it is emitted before the model loads), so it carries `batchRequested` / `ckptRequested`; the RESOLVED pair is `vram.batch` / `vram.ckptSegments` (+ `ckptSource`), emitted after the fit.
+- Server/UI: `TrainDitOptions.batch` / `.ckptSegments` (route validation mirrors the CLI ranges; `ckptSegments` is the CLI's `--ckpt` value verbatim — 0/1/2-32). TrainDitForm's Advanced drawer has "Batch size" and a "Checkpointing" select (Auto/Off/2/4/8/16).
+- VRAM model (`dit-vram.h`): the arena/KV-expand terms scale with a straight `×B`, validated (not re-derived — the pre-existing arena polynomial was already the fitted part) against a measured {B 1,5} × {ckpt 0,4} × {crop 375,750} grid at `--layers 8`: every cell landed within **+0.1% to +6.9%** of the trainer-owned high-water peak, always over- never under-predicting. The boundary-buffer term is exact arithmetic (matched every grid cell's reported `boundaryMb` to the byte). See the dated comment block in `dit-vram.h` for the full 8-cell table.
+- Throughput (RTX 5090, 32 GB, LoKR dim512+MLP, full available depth = 32 layers, `--mirror bf16`, 10 epochs, same seed, tiny 5-song smoke tensors): `--batch 1 --ckpt 0` (yesterday's behavior) auto-fit a 674-frame crop and ran ~13.5 audio-seconds of training content per wall-clock second; `--batch 5 --ckpt auto` was forced to 4 checkpoint segments to fit batch 5 at full depth at all (batch 5 does **not** fit full-depth on this card with `--ckpt 0` — the auto-fit correctly reduces B to 1 rather than touch depth, confirming the C4 order), auto-fit a smaller 524-frame crop, and ran ~5.3 audio-seconds/wall-second — **~2.5× SLOWER**, not the ≥3× hoped for. Layers=8 (shallow, LoRA rank-16, no MLP) tells the opposite story: the grid's own probe timings show batching there is ~2.4× **faster** per crop. Read together: batching's ROI is depth/shape-dependent — a already-compute-bound full-depth 32-layer graph on a fast card gets little from wider batches while paying checkpointing's ~30-40% extra-forward tax and a forced-smaller crop; a shallow/small graph that under-uses the GPU at batch 1 gets a real win. Neither Side-Step's 0.223 s/(60 s crop) anchor nor the ≥3× target is met at full depth on this card/config — reported honestly per the design's own instruction to do so; smaller-depth or larger-VRAM configurations are the likelier place batching pays off and are untested here.
+- bf16-mirror A/B (same-seed loss-curve diff, `--mirror f32` vs `--mirror bf16`) repeated at `--batch 5` (LoRA rank-16, `--layers 8`, `--ckpt 0`, 10 epochs): max abs loss delta **7.9e-5**, max relative delta **5.8e-5** — essentially unchanged from the original batch-1 measurement (7.6e-5), so amendment A2's CUDA GEMM-route change (batched-cublas at B>1) does not measurably widen the f32/bf16 drift.
+- Loss curves at the throughput bench's Side-Step-parity LR (0.01) do **not** cleanly descend over just 10 toy epochs on a 5-song dataset — the cosine schedule (tuned for hundreds of production epochs) compresses its whole warmup+decay into 10-20 steps, producing a noisy rise-then-plateau in both the batch-1 and batch-5 runs. A same-config re-run at LR 0.001 descends cleanly in both (ma5 1.27→1.04 at batch 1, 1.34→1.09 at batch 5) confirming the mechanism itself is sound — the non-descent at LR 0.01 is a toy-bench artifact (LR schedule too aggressive for a 10-epoch/5-song smoke run), not a regression.
+
+**`--bwd <outprod|mm>` — the mul_mat-backward reformulation** (2026-07-29, `engine/patches/mm-backward.patch`)
+- **The constraint it removes.** ggml emits the MUL_MAT *activation* gradient as `out_prod(W, transpose(grad))`, and ggml-cuda's `OUT_PROD` is F32-only (`cublasSgemm`). That forces the frozen weight to be F32, which in turn drags the **forward** `mul_mat` into `ggml_cuda_op_mul_mat_cublas`'s F32 branch → TF32 tensor cores. Upstream `ggml.c` has always carried the alternative, commented out: `mul_mat(cont(transpose(W)), grad)`. It is dtype-agnostic, so a BF16 weight rides real BF16 tensor cores in **both** directions with no dequant and no F32 window. `--bwd mm` turns it on.
+- **The two arms are provably the same op**, not an approximation: `out_prod(W[n,m,q1,r1], transpose(grad)[p,m,qq,rr]) -> [n,p,qq,rr]` and `mul_mat(cont(transpose(W))[m,n,q1,r1], grad[m,p,qq,rr]) -> [n,p,qq,rr]`, and `ggml_can_out_prod` / `ggml_can_mul_mat` reduce to the identical `b->ne[2] % a->ne[2] == 0, b->ne[3] % a->ne[3] == 0` broadcast pair. This is **not** a quality trade like `--weights bf16`, which genuinely changes the quantity computed — `--bwd` only changes how the same quantity is reached.
+- **Per-layer GEMM measurement** (`ace-train spike gemmbench`, RTX 5090, Qwen3-4B projection shapes, reproduced 2026-07-29): `bf16_pre` beats the shipped `cur_lowvr` mix by **1.67–1.78×** per layer per step (1.41–1.64× vs `cur_naive`). Gradient parity vs the TF32 `out_prod` reference: **cosine 0.999996, max relative ~3–4e-3**. `bf16_cont` ≈ `bf16_pre`, i.e. the per-use `cont(transpose(W))` is free at these shapes — so the implementation needs **no pre-transposed weight cache**, which is why it lives entirely in graph-construction code and stays backend-agnostic.
+- **End-to-end it is much smaller than that — 1.10–1.17×, and you must interleave to see it at all.** Same-seed `train-dit` A/B (LoKR dim512+MLP, `--mirror bf16 --layers 8 --batch 1 --ckpt 0 --order fixed`, 3 interleaved rounds per arm, min/median over epochs 2-6): crop 1250 → outprod median 4179 ms/epoch vs mm 3785 ms = **1.10×**; crop 375 → 1435 vs 1229 ms = **1.17×**. The gap vs the per-layer figure is Amdahl, not a broken lever: at `--layers 8` only 8 of 32 layers have a backward at all, and the step is dominated by the retained-softmax O(S²·Nh) attention (no flash-attn backward) plus LoKR kron reconstruction — which is exactly why the smaller crop, with less attention per projection, scores higher. **Do not trust a single non-interleaved A/B here**: run-to-run contention on a working machine is ±10%, the same size as the effect, and the first two one-shot runs gave 1.47× and 0.82× purely on which arm caught the interference.
+- **Loss drift is negligible and deterministic.** Over 10 same-seed epochs: max absolute delta **7.4e-5**, max relative **6.7e-5** (6-epoch runs: 4.3e-5 at crop 1250, 2.7e-5 at crop 375); epoch 1 is bit-identical to 9 decimals, and each arm reproduces its own curve exactly across 3 repeats. Same order as the `--mirror f32`/`bf16` A/B (7.9e-5) — this is GEMM-reassociation rounding, not a different training run.
+- **Self-test:** `ace-train train-dit --self-test` gives the identical **21/22** rung-for-rung with the env unset and with `GGML_BACKWARD_MM=1`, T9 (the known `E[v^2]` fingerprint drift) the only failure in both. The FD gate T4 reports the same max rel 2.3142e-04 on the same loss 1.659584 either way; SC1's graph grows 4386→4505 nodes under mm (the extra `cont` per converted site) while its loss and its segmented-vs-monolithic gradient delta stay bit-identical.
+- **Contiguity guard (found by the self-test, not by reading).** `grad` becomes `mul_mat`'s src1, and the CUDA mul_mat kernels require it row-contiguous — `ggml-cuda/mmf.cu:28` asserts `nb10 == ts_src1` and **aborts** otherwise, where `out_prod` happily takes an arbitrary transposed view. The patch therefore takes the mm arm only when `ggml_is_contiguous(grad)` and otherwise falls back to out_prod. Without the guard the LoKR rung SC3 hard-aborts. `ggml_cont` on grad is not the fix — grad is activation-sized, so copying it costs more than the GEMM saves.
+- **How it is wired.** `engine/ggml` is a submodule kept upstream-clean, so this ships as a vendored patch alongside `bf16-out-prod.patch` — disjoint files (`ggml.c` vs `ggml-cuda/*.cu`), applied by a `for p in engine/patches/*.patch` loop in CI's "Apply engine patches" step and guarded by **Hook 8** in `engine/verify-hooks.ps1`. The patch is **env-gated**: with `GGML_BACKWARD_MM` unset the emitted graph is byte-identical to upstream, so a stock build regresses nothing. `ace-train`'s `--bwd mm` sets the variable in `cmd_train_lm`/`cmd_train_dit` **before** `ggml_time_init()` and any model load — the patch latches it into a static on the first backward it builds, so it must be set before graph construction, and setting it there covers `--self-test` too. `--bwd outprod` deliberately does *not* clear an externally-set variable, so exporting `GGML_BACKWARD_MM=1` still A/Bs the whole self-test battery.
+- **`--weights bf16` (LM) and `--bwd mm` COLLIDE — refused, not coerced.** Lever A reaches the same mul_mat backward by a different route: it lets ggml build the out_prod form and then rewrites those nodes in place, asserting **exactly 7 rewrites / 0 skipped / 0 residual** per segment (its S18 tripwire). Under `--bwd mm` ggml emits mul_mat directly, the surgery finds nothing, and the tripwire `GGML_ABORT`s — correct behaviour, but mid-run, after the model load. `ace-train train-lm` now exits 2 on the pair and the route answers 400. There is no version of that pair worth building: on the `f32-window` path `--bwd mm` gains nothing either, because the weight it transposes is the **F32 window**, so the GEMM stays TF32 and the extra `cont` is pure cost. **The LM already solved this problem its own way; `--bwd mm` is a train-dit lever.**
+- **Defaults are deliberately split three ways.** The engine default is `outprod` (a bare `ace-train` invocation is unchanged). The **server** defaults **train-dit to `mm`** and **train-lm to `outprod`** — the LM's `weights` already defaults to `bf16`, so an LM default of `mm` would brick the *default* LM job on the collision above. `buildTrainLmArgs`/`buildTrainDitArgs` always emit `--bwd` so an older `ace-train.exe` rejects it loudly instead of silently running the slow path. Surfaced as "Backward GEMM" in both Advanced drawers; recorded in the `start` JSONL event and in `dit_train_log.json` / `lm_train_log.json`'s `config.bwd`.
+
+**Trigger words (embedded in the adapter)**
+- The tag was always trained in — `preprocess-run.h:192-204` bakes `custom_tag` into the caption before text encoding, and `lm-extract.h` re-applies it — but export used to drop it, leaving inference to guess the trigger from the filename (wrong for our adapters: the dir is `<name>-<size>`, not the tag).
+- Both trainers now write `hot_step_trigger`, `hot_step_trigger_position` and `modelspec.trigger_phrase` into the adapter's safetensors `__metadata__`. Tensors and `adapter_config.json` are untouched, so ComfyUI/PEFT/Side-Step load it exactly as before. **Not** `adapter_config.json` — PEFT does `LoraConfig(**json)` and unknown keys are a version-dependent TypeError.
+- Source of the value: `--trigger`/`--trigger-position` flags, else the variant's `preprocess_meta.json` (`custom_tag`/`tag_position`). The **variant meta, not the dataset row**, is authoritative — a dataset's tag can be edited after preprocessing, in which case the tensors (and the adapter) carry the old one. `tag_position: replace` embeds nothing: that path never puts the tag in the caption at all.
+- Generation-side resolution lives in `services/generation/triggerWords.ts` and runs **server-side** for every caller: per adapter, manual override → embedded → filename fallback → none. A stack can mix prepend and append. The gate widened from "a DiT adapter is loaded" to "any adapter", so planner-LM adapters now contribute their trigger too.
+- Legacy corpus: `server/scripts/stamp-adapter-triggers.mjs`, dry-run by default. Verified on a copy — 132 MB payload SHA-256 identical, header stays 8-byte aligned, `.bak` kept.
+
+**Adapter layout (per-base + per-run, 2026-07-28)**
+- `<adapters>/lm-06b|lm-17b|lm-4b/<artist>/<run>/` and `<adapters>/dit-<shorthand>/<artist>/<run>/`, where `<run>` = `YYYY-MM-DD_HH-MM-SS` (logs/ convention) — retraining an artist never overwrites an earlier adapter. Artist names carry **no** `-<size>` suffix; the parent folder says the base.
+- Single source of truth: `server/src/services/training/adapterLayout.ts` (size slugs, the confirmed DiT shorthand map `xl-thirds`/`xl-base-turbo`/`xl-sft-turbo`/…, run stamps, latest-run resolution). `migrate-adapter-layout.mjs` moves an old corpus; its shorthand map must stay in sync.
+- Writes go through `lmRunDirFor`/`ditRunDirFor` (fresh stamped dir); reads through `adapterDirFor`/`adapterDitDirFor` (newest run → unversioned artist dir → legacy flat `lm/<name>-4B` / root DiT dir). Two legacy forms are read-everywhere, written-never.
+- Scanners: `GET /api/adapters/lm` walks all lm-* roots + legacy `lm/` (entries carry `lmSize`, `run`, `trigger`) — this is what the global-bar planner list, Lyric Studio's preset picker and the Training Studio picker all consume. `POST /api/adapters/scan` descends into `dit-*` folders (and their run subdirs) only.
+- The audition's base-LM pinning now derives the size from the `lm-<size>` parent folder (suffix kept as legacy fallback) — do not reintroduce suffix-only derivation.
+
+**Audition (pure-LM preview)**
+- `POST /codes-decode`: codes → `detok_ggml_decode` → tail-call the existing VAE decode worker (layouts are byte-identical; zero changes to `/vae`). Deterministic; full song ≈ 3.1 s warm.
+- A/B = two `/lm` calls, same explicit `lm_seed`, base side **must** byte-match an adapterless run (V5 hard gate). Base LM auto-derived from the adapter's `-<size>` suffix (`pickLmFor`) — never let the engine's `resolve_name` fallback pick (sticky 0.6B met a 4B adapter: "36 layers but model has 28").
+- LM-echo sideband trap: never forward the `/lm` reply into another request — build requests fresh (see `generation-request-flow` skill).
+
+## Build & verification rules
+
+- `ace-train`-only changes: `cmake --build engine/build --config Release --target ace-train` — safe with the app running **if no ace-train.exe process is live** (training runs spawn it). New `train/*.h` headers need **no CMake change**.
+- `hot-step-server.cpp` (ace-server) changes: full `dev-rebuild.bat` cycle (app goes down; Rob restarts with `dev.bat`). Never `build.cmd` directly, never `--clean-first`.
+- Self-tests are the regression net: `ace-train train-lm --self-test`, `ace-train train-dit --self-test`, `ace-train spike …` (gemmbench, bf16layer, dit2 — the measurement harnesses that justified every design call; keep them).
+- GPU discipline for agents: `nvidia-smi` before runs, stay under ~29 GB total, never kill Rob's python/node/ace-server, bounded runs while he's working.
+- TypeScript: `server` `npx tsc --noEmit`; `ui` `npx tsc -p tsconfig.app.json --noEmit` (one pre-existing error in `globalParamsStore.ts:449` — `lmAdapter` vs `lmAdapters` in `types.ts:347` — is NOT ours).
+
+## Measured reference numbers
+
+| Thing | Number |
+|---|---|
+| LM 0.6B full E2E (17 songs, 16 epochs) | ~70 s wall |
+| LM 4B, checkpointed | ~12.4 GB trainer-owned, ~1.06–1.34 s/micro-step; a real run auto-stops ~epoch 29 at target 4.0 ≈ 12 min (bf16: ~9.5 min) |
+| DiT full-depth, LoRA r16 attn-only, crop 750 frames (**S=375**) | ~147 ms/step, ~23 GB; crop ceiling ~1000–1250 frames @24 GB. **Frames, not tokens: `S = frames / patch_size` and patch is 2, so "crop 750" is 750 frames = 30 s = S 375.** |
+| DiT full-depth **LoKR dim512 + MLP**, crop ~690 frames (S≈344), RTX 5090 | **~195 ms/step, ~2.7 s/epoch at 14 songs** (was 1902 ms/step before the 2026-07-30 `apply()` fix). A 14-song adapter reaches ma5 0.6 in **~9 min / ~196 epochs**, vs ~90 min before |
+| Preprocess | ~6 GB VRAM (vae-chunk 384; 1024 costs ~17 GB), ~3 s/song |
+| Codes decode (audition) | ~200 ms warm 30 s clip; ~3.1 s full song |
+| TF32 vs BF16 GEMM (5090) | ~95 vs ~165 TFLOP/s; end-to-end lever 1.256× at 4B |
+
+## Pending / open decisions (Rob's list)
+
+1. ~~One `dev-rebuild` owed~~ — **done**: the 2026-07-28 17:05 rebuild (during the SuperSep work) postdates every one of those source edits; `ace-train train-dit --help` shows `--no-target-mlp` and `ace-server.exe` carries the audition fixes.
+1b. **Trigger stamper awaits Rob's approval**: `node server/scripts/stamp-adapter-triggers.mjs --datasets D:\Ace-Step-Latest\Datasets-LoRA-LoKR` prints the proposed table for 179 adapters (28 from real dataset.json ground truth, 151 from the filename). Nothing is written without `--apply`. Also outstanding: the per-adapter **manual trigger override** UI — the server already accepts `triggerSpecs` entries with `source:'override'` and a `path`, but no control emits them yet.
+2. **bf16 listen test**: train twin LM adapters (f32-window vs `--weights bf16`), A/B via audition; ship-call by ear.
+3. **LoKR for DiT** (`dit-adapter.h` has the parameterization seam ready) — Rob's preferred adapter type.
+4. **ConvRot bases**: refused by the trainer pending a convrot spike (`…-convrot-*` GGUFs exist).
+5. **BF16 `out_prod` vendor patch**: would ~2.5× DiT crop @24 GB, unlock 12 GB cards, and remove the bf16-lever's dst-rounding error — requires forking the ggml submodule (CI re-inits from upstream; see the rejected-P1 analysis in the levers plan).
+6. **Top-K DiT adapter quality** on small cards: runs, but musical usefulness unmeasured.
+7. Micro-batching: closed with measurements; revisit only if the graph-build overhead picture changes.
+8. UI `minVram` hint string may be optimistic at r128+MLP defaults.
+
+## Where the deep documents live
+
+`docs/plans/` is **gitignored (local-only)** — on Rob's machine the full design/implementation plans exist: `2026-07-27-training-studio-design.md` (master design), `-dataset-studio-implementation.md`, `-preprocess-implementation.md`, `-lm-trainer-implementation.md`, `2026-07-28-lm-4b-training.md`, `-dit-trainer-implementation.md`, `-lm-speed-levers.md`, `-codes-preview.md`. Each contains the frozen contracts and verification ladders. The commit history (`git log --oneline --grep=training`) carries the measured numbers per milestone.

docs/release-notes-v1.2.0.md

@@ -0,0 +1,88 @@
+# v1.2.0
+
+The StableStep release. Two flagship features: **StableStep**, a Stable Audio 3-powered
+instrumental refiner that finally kills VAE fizz at the source, and **MIDI Studio**,
+audio-to-MIDI transcription running on our own native GGML port of MuScriptor. Plus
+timestep-gated adapters, parameter profiles, two new MDMAchine plugins, and a stack of fixes.
+
+## ✨ StableStep — Stable Audio 3 instrumental refining
+
+Generated tracks carry a characteristic high-frequency "fizz" from the ACE-Step
+autoencoder. StableStep re-renders the instrumental through **Stable Audio 3**
+(SDEdit-style partial re-noising, 8-step distilled rectified flow) so the fizz band is
+*regenerated* with real spectral detail instead of filtered. Vocals are never touched:
+BS-RoFormer splits them out (lead **and** backing), PP-VAE polishes them, and they're
+remixed over the refined instrumental — the exact complement split guarantees nothing
+is lost, and lyrics stay bit-identical.
+
+- **One toggle + a strength slider** (0.10–0.60) in Post-Processing. The refine prompt is
+  derived from each track's own caption automatically (vocal descriptors stripped).
+- **Runs natively in the C++ engine** — no Python. Two backends, selectable in-app:
+  - **GGML** (4 GGUF files, ~5.8 GB): CUDA, Vulkan, and CPU. The fastest option on
+    NVIDIA in our testing — ~2 s of compute for a 30 s clip on an RTX 5090.
+  - **ONNX Runtime / TensorRT** (~12 GB): NVIDIA alternative path.
+- **Numerically validated end to end**: every ported component matches the reference
+  implementation at cosine > 0.9998; the two backends agree with each other at 0.9999.
+- **Model Manager → StableStep tab** downloads either backend set from
+  [scragnog/HOT-Step-CPP-StableStep](https://huggingface.co/scragnog/HOT-Step-CPP-StableStep),
+  with license acceptance built in (Stability AI Community License — free for individuals
+  and orgs under $1M revenue). *Powered by Stability AI.*
+- New engine surface for tinkerers: `POST /sa3-refine` (strength/steps/sampler/backend)
+  and SuperSep `level=4` — a dedicated vocals+instrumental split.
+
+## 🎼 MIDI Studio — audio-to-MIDI on the native engine (#80)
+
+Transcribe any library track or uploaded WAV/MP3 into multi-track MIDI (34 instrument
+groups + drums) using **MuScriptor** (Kyutai & Mirelo) — ported phase by phase to our own
+C++/GGML `ace-midi` engine and validated **byte-exact** against the reference
+implementation. A 3.5-minute track transcribes in ~50 s on an RTX 5090; no Python anywhere.
+
+- **Live piano roll** fills in while transcription runs, with per-channel instrument
+  coloring; crossfade playback slider between the original audio and the MIDI rendition,
+  plus per-instrument mute/solo.
+- **Three model sizes** (small 103M / medium 307M / large 1.4B) with in-app gated-weight
+  download flow (weights are CC BY-NC 4.0 — non-commercial).
+- Engine work along the way: exact-parity mel frontend + KV-cache greedy decode, F16 KV
+  cache to fix CUDA decode corruption, byte-exact event decode + MIDI writer.
+- `ace-midi` ships in every platform bundle.
+
+## 🎛 Adapter system
+
+- **Timestep-dependent adapter gating** (interval experts / MoE) — restrict any stacked
+  adapter to a step window, e.g. one adapter shapes structure early, another handles
+  detail late. UI windows are evaluated **per step** (not raw t), and they now compose
+  correctly with Adapter VRAM quant (previously a silent 32 GB blowup).
+- **PEFT DoRA support** + per-module `alpha_pattern`.
+- **`runtime_lowrank` mode** — factor-apply without materialized deltas: the lowest-VRAM
+  way to run big adapter stacks.
+- **Merge (low VRAM)** — opt-in requant of merged weights back to the base's native
+  quant type (~¼ merged-DiT VRAM on a Q8 base).
+- LoKr Kronecker-apply self-test (`HOTSTEP_KRON_TEST`); `gain_domain` parse fix.
+
+## 📋 Parameter profiles
+
+Save, apply, and delete named full-state snapshots of your generation parameters —
+in-app, with JSON import/export, inline rename, and click-to-inspect.
+
+## 🔌 Plugins (MDMAchine)
+
+- **MD HT Scheduler V3** and **MD Trajectory Anchor V5**, with user manuals.
+- Max inference steps raised 200 → 300 (UI + engine clamp).
+
+## 🔧 Fixes & polish
+
+- **Storm streaming**: stream died instantly (premature `close` on body consumption);
+  "Keep DiT & VAE loaded" is now honored between stream slots.
+- **In-app restart** was dead on Windows (ping-as-sleep hang meant taskkill never fired).
+- **Queue**: deleted songs pruned from the persisted queue on load; Nuke All Generations
+  clears the queue store and recent-songs cache.
+- **Logs**: GGML debug spam dropped at source with digit-insensitive dedup; terminal
+  stick-to-bottom pin survives layout shifts.
+- MCP: generating model name appended to song titles.
+
+---
+
+**Models:** StableStep sets are on
+[Hugging Face](https://huggingface.co/scragnog/HOT-Step-CPP-StableStep) or one click away
+in the Model Manager. MuScriptor weights are gated (free) — request access via the links
+inside MIDI Studio.