civ
901 lines
36 KiB
Markdown
901 lines
36 KiB
Markdown
# Architecture
|
|
|
|
Technical reference for omnivoice.cpp, the GGML port of OmniVoice
|
|
(k2-fsa/OmniVoice). This document covers the model, the conversion to
|
|
GGUF, the inference pipeline, the GGML graph conventions, and the CLI
|
|
tools.
|
|
|
|
## Upstream model
|
|
|
|
OmniVoice (Xiaomi / k2-fsa, Apache 2.0) is a multilingual zero-shot
|
|
text-to-speech system covering 646 languages. It targets three modes :
|
|
|
|
voice cloning reference audio plus reference transcript drive the
|
|
target speaker identity
|
|
voice design six attribute categories (gender, age, pitch, style,
|
|
volume, emotion) drive a synthesised speaker
|
|
auto voice no reference, the model picks a coherent speaker per
|
|
utterance
|
|
|
|
The system is a non autoregressive, mask-predict (MaskGIT) generative
|
|
model running on top of a Qwen3 backbone with custom audio
|
|
input/output and a separate audio tokenizer. The audio tokenizer is
|
|
the Higgs Audio v2 codec (`bosonai/higgs-audio-v2-tokenizer`,
|
|
Apache 2.0), which combines a HuBERT semantic stream, a DAC acoustic
|
|
stream, and an 8-codebook residual vector quantiser at 25 frames per
|
|
second over 24 kHz mono audio.
|
|
|
|
Single public checkpoint : `k2-fsa/OmniVoice` (3.1 GB).
|
|
|
|
Backbone Qwen3 0.6B (28 layers, hidden 1024, GQA 16/8)
|
|
Audio codebooks 8 residual, 1024 entries each plus 1 mask token
|
|
Audio framerate 25 Hz
|
|
Hop length 960 samples
|
|
Sample rate 24 kHz mono
|
|
Semantic SR 16 kHz (HuBERT input)
|
|
MaskGIT steps 32 default, configurable
|
|
|
|
## Build
|
|
|
|
```
|
|
git clone --recurse-submodules https://github.com/ServeurpersoCom/omnivoice.cpp.git
|
|
cd omnivoice.cpp
|
|
./buildcuda.sh # NVIDIA GPU
|
|
./buildvulkan.sh # AMD/Intel GPU (Vulkan)
|
|
./buildcpu.sh # CPU only
|
|
./buildall.sh # all backends, runtime DL loading
|
|
```
|
|
|
|
The GGML submodule lives at `https://github.com/ServeurpersoCom/ggml.git`
|
|
and provides two custom ops required by the codec :
|
|
`GGML_OP_SNAKE` and `GGML_OP_COL2IM_1D`. Both have CPU, CUDA, Metal,
|
|
and Vulkan kernels.
|
|
|
|
## Model conversion
|
|
|
|
```
|
|
./checkpoints.sh # hf download k2-fsa/OmniVoice -> checkpoints/OmniVoice/
|
|
./convert.py # 2 GGUFs in BF16 -> models/
|
|
./quantize.sh # base LM Q8_0 (tokenizer stays at native dtype)
|
|
```
|
|
|
|
Outputs :
|
|
|
|
```
|
|
models/omnivoice-base-BF16.gguf 1.2 GB LLM + audio_emb + audio_heads + tokenizer
|
|
models/omnivoice-base-Q8_0.gguf 626 MB quantized base, 1.9x smaller
|
|
models/omnivoice-tokenizer-F32.gguf 702 MB HuBERT + DAC + RVQ + fc/fc2 (native F32)
|
|
```
|
|
|
|
The audio tokenizer GGUF preserves the source dtype 1:1. The reference
|
|
checkpoint stores the codec at F32, so the GGUF stays F32 to avoid
|
|
truncation noise across the 8-stage RVQ residual chain. Late codebooks
|
|
fall below 50 percent codebook match against the reference if any
|
|
intermediate weight is rounded to BF16.
|
|
|
|
Quantisation policy : Q8_0 only on the base LM. The 612 M parameter
|
|
backbone is small enough that lower quants degrade quality without
|
|
meaningful size gains.
|
|
|
|
## GGUF layout
|
|
|
|
`omnivoice-base-{quant}.gguf` (arch `omnivoice-lm`) :
|
|
|
|
```
|
|
metadata
|
|
general.architecture omnivoice-lm
|
|
block_count 28
|
|
embedding_length 1024
|
|
feed_forward_length 3072
|
|
head_count 16
|
|
head_count_kv 8 (GQA 2:1)
|
|
key_length 128
|
|
vocab_size 151676
|
|
context_length 40960
|
|
layer_norm_rms_eps 1e-6
|
|
rope_freq_base 1e6
|
|
omnivoice.tie_word_embeddings true
|
|
omnivoice.num_audio_codebook 8
|
|
omnivoice.audio_vocab_size 1025
|
|
omnivoice.audio_mask_id 1024
|
|
omnivoice.audio_codebook_weights [8, 8, 6, 6, 4, 4, 2, 2]
|
|
omnivoice.special.denoise 151669
|
|
omnivoice.special.lang_start 151670
|
|
omnivoice.special.lang_end 151671
|
|
omnivoice.special.instruct_start 151672
|
|
omnivoice.special.instruct_end 151673
|
|
omnivoice.special.text_start 151674
|
|
omnivoice.special.text_end 151675
|
|
tokenizer (Qwen2 BPE, 151676 vocab, 151387 merges, 33 added_tokens)
|
|
|
|
tensors (312)
|
|
llm.embed_tokens.weight (151676, 1024)
|
|
llm.norm.weight (1024,)
|
|
llm.layers.0..27.{q,k,v,o}_proj.weight GQA, no bias
|
|
llm.layers.0..27.self_attn.{q_norm, k_norm}.weight per-head RMSNorm (128,)
|
|
llm.layers.0..27.{input,post_attention}_layernorm.weight RMSNorm
|
|
llm.layers.0..27.mlp.{gate,up,down}_proj.weight SwiGLU, no bias
|
|
audio_embeddings.weight (8200, 1024) 8 codebooks * 1025 vocab
|
|
audio_heads.weight (8200, 1024) audio output, no bias
|
|
```
|
|
|
|
`omnivoice-tokenizer-{quant}.gguf` (arch `omnivoice-tokenizer`) :
|
|
|
|
```
|
|
metadata
|
|
omnivoice.sample_rate 24000
|
|
omnivoice.semantic_sample_rate 16000
|
|
omnivoice.downsample_factor 320
|
|
omnivoice.codebook_size 1024
|
|
omnivoice.codebook_dim 64
|
|
omnivoice.acoustic.encoder_hidden_size 64
|
|
omnivoice.acoustic.decoder_hidden_size 1024
|
|
omnivoice.acoustic.hidden_size 256
|
|
omnivoice.acoustic.n_codebooks 9 (only 8 used)
|
|
omnivoice.acoustic.hop_length 960
|
|
omnivoice.acoustic.upsampling_ratios [8, 5, 4, 2, 3]
|
|
omnivoice.acoustic.downsampling_ratios [8, 5, 4, 2, 3]
|
|
omnivoice.semantic.hidden_size 768 (HuBERT base)
|
|
omnivoice.semantic.intermediate_size 3072
|
|
omnivoice.semantic.num_attention_heads 12
|
|
omnivoice.semantic.num_hidden_layers 12
|
|
omnivoice.semantic.num_feat_extract_layers 7
|
|
omnivoice.semantic.conv_dim [512]*7
|
|
omnivoice.semantic.conv_kernel [10, 3, 3, 3, 3, 2, 2]
|
|
omnivoice.semantic.conv_stride [5, 2, 2, 2, 2, 2, 2]
|
|
omnivoice.semantic.num_conv_pos_embeddings 128
|
|
omnivoice.semantic.num_conv_pos_embedding_groups 16
|
|
omnivoice.semantic.layer_norm_eps 1e-5
|
|
|
|
tensors (486)
|
|
acoustic_encoder.* DAC encoder, 5 blocks, downsamples 8 5 4 2 3
|
|
acoustic_decoder.* DAC decoder, 5 blocks, upsamples 8 5 4 2 3
|
|
encoder_semantic.* semantic conv blocks
|
|
semantic_model.* HuBERT base, weight_norm folded
|
|
quantizer.quantizers.0..7.{codebook.embed, project_in.{w,b}, project_out.{w,b}}
|
|
fc.{weight, bias} 1024 -> 1024 (after concat acoustic + semantic)
|
|
fc2.{weight, bias} 1024 -> 256 (before DAC decoder)
|
|
```
|
|
|
|
Single weight_norm fold at convert time :
|
|
`semantic_model.encoder.pos_conv_embed.conv.weight`, formula
|
|
`weight = v * g / ||v||_{dim=(0,1)}` matching
|
|
`torch._weight_norm(v, g, dim=2)`. Validated bit-perfect, max abs diff
|
|
3.9e-7 against the PyTorch reference.
|
|
|
|
## Component architecture
|
|
|
|
### Qwen3 0.6B backbone with custom IO
|
|
|
|
Standard Qwen3 modulo two changes :
|
|
|
|
input embed hybrid text plus audio, weighted sum across 8
|
|
codebooks gated by `audio_mask`
|
|
output head custom `audio_heads` Linear (8200, 1024), no text
|
|
`lm_head`
|
|
|
|
```
|
|
input_ids [B, 8, S] int (text on row 0, audio codes on rows 1..7)
|
|
audio_mask [B, S] bool
|
|
|
|
text_emb = embed_tokens(input_ids[:, 0, :]) (B, S, 1024)
|
|
shifted = input_ids * audio_mask + offsets[None, :, None]
|
|
offsets = arange(8) * 1025
|
|
audio_emb = audio_embeddings(shifted).sum(dim=1) (B, S, 1024)
|
|
inputs = where(audio_mask, audio_emb, text_emb) (B, S, 1024)
|
|
|
|
x = qwen3_forward(inputs, attention_mask, position_ids) (B, S, 1024)
|
|
|
|
logits_flat = x @ audio_heads.weight.T (B, S, 8200)
|
|
logits = reshape (B, 8, S, 1025)
|
|
```
|
|
|
|
Qwen3 specifics already in llama.cpp :
|
|
|
|
28 layers, hidden 1024, intermediate 3072
|
|
16 query heads + 8 KV heads (GQA 2:1), head_dim 128
|
|
per-head RMSNorm on Q and K (q_norm, k_norm shape (128,)) before RoPE
|
|
no bias on Q/K/V/O/MLP
|
|
RoPE theta = 1e6
|
|
SwiGLU MLP
|
|
tie_word_embeddings = true (`lm_head` absent, output goes through audio_heads)
|
|
|
|
### MaskGIT decoder
|
|
|
|
Iterative non autoregressive decoder, no KV cache. Each step is a full
|
|
prefill of the LLM on the current input.
|
|
|
|
Prompt (per item, broadcast across 8 codebooks) :
|
|
|
|
```
|
|
[<|denoise|>]?
|
|
<|lang_start|> {iso_code or "None"} <|lang_end|>
|
|
<|instruct_start|> {style or "None"} <|instruct_end|>
|
|
<|text_start|> {ref_text + " " + text} <|text_end|>
|
|
{ref_audio_codes}?
|
|
{MASK x num_target_tokens}
|
|
```
|
|
|
|
Unconditional prompt for CFG = the trailing `num_target_tokens` mask
|
|
tokens only. Batched (cond + uncond) doubles the batch dim.
|
|
|
|
```
|
|
for step in 0..num_step-1 :
|
|
forward(input_ids, audio_mask, attention_mask) (2B, 8, S, 1025)
|
|
log_probs = log_softmax(c + cfg_scale * (c - u))
|
|
log_probs[..., MASK_ID] = -inf
|
|
if class_temp > 0 :
|
|
keep_top_k_ratio(log_probs, 0.1)
|
|
gumbel_sample(temp = class_temp)
|
|
pred = argmax(log_probs)
|
|
score = log_probs.max - layer_idx * layer_penalty (5.0)
|
|
if pos_temp > 0 :
|
|
score += gumbel * pos_temp
|
|
score[already_unmasked] = -inf
|
|
topk_idx = topk(score.flatten(), schedule[step])
|
|
tokens[topk_idx] = pred[topk_idx]
|
|
update batch_input_ids cond and uncond
|
|
```
|
|
|
|
Schedule of newly unmasked positions per step is computed from
|
|
`_get_time_steps(t_start=0, t_end=1, num_step, t_shift=0.1)` then
|
|
`ceil(N_total * (t[step+1] - t[step]))`. 32 steps default.
|
|
|
|
KV cache is not usable across MaskGIT steps. The attention is fully
|
|
bidirectional, so the prefix hidden states depend on the current
|
|
target state through every layer. As tokens get progressively
|
|
unmasked the K and V tensors of the prefix at every layer above the
|
|
embeddings drift, which forbids the standard prefix-cache trick that
|
|
works for causal LMs. Each step is therefore a full prefill of the
|
|
LLM at cost `2 * forward_full(B, S)` (the 2 accounts for the cond +
|
|
uncond CFG rows).
|
|
|
|
Determinism. With `class_temperature = 0` and `position_temperature = 0`
|
|
the decoder is bit deterministic. Higher temperatures rely on a
|
|
seedable Philox4x32-10 PRNG. The pipeline threads the Philox counter
|
|
across MaskGIT calls so that chunked inference matches the global RNG
|
|
drift of the PyTorch reference.
|
|
|
|
#### Inner-loop optimisations
|
|
|
|
The num_step iterations of one chunk run on a fixed shape (`S`, `K`, `B'`) so
|
|
the per-step overhead can be cut without touching the math.
|
|
|
|
`pipeline_tts_llm_forward_batched` accepts a `T_audio` parameter that
|
|
narrows the GPU output to the audio window only. The MaskGIT decoder
|
|
reads cond logits at `[S - T, S)` on row 0 and uncond logits at
|
|
`[0, T)` on row 1, so the full `[B', V, K, S]` tensor is wasteful.
|
|
With `T_audio > 0` the function builds two `ggml_view_4d` over those
|
|
ranges, makes them contiguous via `ggml_cont`, and only those two
|
|
sub-tensors are flagged as graph outputs. The GPU to CPU copy shrinks
|
|
from `B' * V * K * S` floats to `2 * V * K * T_audio` floats, around
|
|
5.6x less on the typical voice cloning shape (S ~ 1880, T ~ 350).
|
|
When `T_audio == 0` the function falls back to the full output, used
|
|
by the debug dump path that needs every position.
|
|
|
|
`MaskgitBatchedCtx` holds the inputs that stay constant across the
|
|
steps : the F32 audio mask and its complement, the RoPE position
|
|
vector, and the F16 attention bias. The bias is the heaviest piece,
|
|
`B' * S * S` F16 conversions per step (about 7 M ops on the typical
|
|
shape). `pipeline_tts_llm_batched_ctx_init` precomputes the bias once
|
|
per chunk, with a single `ggml_fp32_to_fp16` call for each of the two
|
|
distinct values (1.0 and 0.0) hoisted out of the conversion loop. The
|
|
context also keeps the original int32 pointers so the debug loop path
|
|
can hand them down to the single forward unchanged.
|
|
|
|
Both optimisations preserve the math exactly, the only side effect is
|
|
a slight reordering of the GPU FP32 reductions when the scheduler
|
|
fuses the new output nodes differently, which moves the audio cosine
|
|
similarity by a few times 1e-6. Token-level results stay 100 percent
|
|
exact against the PyTorch reference.
|
|
|
|
### Audio tokenizer pipeline
|
|
|
|
Encode (voice cloning reference path) :
|
|
|
|
```
|
|
ref_audio @ 24 kHz (1, 1, T_samples)
|
|
-> resample 16 kHz (kaiser polyphase)
|
|
-> pad 160 each side
|
|
-> HuBERT.feature_extractor (320x downsample)
|
|
-> HuBERT.feature_projection (LayerNorm + Linear)
|
|
-> + pos_conv_embed (folded)
|
|
-> 12 transformer layers
|
|
-> mean over 13 hidden states (1, 768, T_sem)
|
|
-> downsample by 2 (semantic_downsample_factor)
|
|
-> SemanticEncoder (conv blocks) (1, 768, T_frames)
|
|
-> e_acoustic (DAC encoder, 5 down-blocks) (1, 256, T_frames)
|
|
-> concat dim=1 (1, 1024, T_frames)
|
|
-> fc Linear (1024 -> 1024) (1, 1024, T_frames)
|
|
-> RVQ encode (8 codebooks residual) (1, 8, T_frames) int @ 25 fps
|
|
```
|
|
|
|
Decode (TTS path) :
|
|
|
|
```
|
|
codes [B, 8, T] int
|
|
-> RVQ decode :
|
|
for k in 0..7 :
|
|
e_k = codebook[k].embed[codes[k, :]] (B, T, 64)
|
|
p_k = e_k @ project_out[k].W.T + bias[k] (B, T, 1024)
|
|
out += p_k
|
|
-> transpose (B, 1024, T)
|
|
-> fc2 Linear (1024 -> 256) (B, 256, T)
|
|
-> acoustic_decoder DAC :
|
|
conv1 (256 -> 1024, k=7, pad=3)
|
|
for block in 0..4, ratios [8, 5, 4, 2, 3] :
|
|
snake1 (alpha)
|
|
conv_t1 (IC -> OC, k=2*r, stride=r,
|
|
padding=ceil(r/2), output_padding=r%2)
|
|
for res_unit in 0..2, dilations [1, 3, 9] :
|
|
snake1 (alpha)
|
|
conv1 (OC, OC, k=7, dil=d, pad=3*d)
|
|
snake2 (alpha)
|
|
conv2 (OC, OC, k=1)
|
|
residual add
|
|
snake1 (alpha) final
|
|
conv2 (32 -> 1, k=7, pad=3)
|
|
-> audio (B, 1, 960*T)
|
|
```
|
|
|
|
960x upsample = 8 * 5 * 4 * 2 * 3. T_in @ 25 fps -> T_out @ 24 kHz exact.
|
|
|
|
### DAC decoder block channels
|
|
|
|
```
|
|
block 0 : IC=1024 OC=512 stride=8 K=16 pad=4 output_pad=0
|
|
block 1 : IC=512 OC=256 stride=5 K=10 pad=3 output_pad=1
|
|
block 2 : IC=256 OC=128 stride=4 K=8 pad=2 output_pad=0
|
|
block 3 : IC=128 OC=64 stride=2 K=4 pad=1 output_pad=0
|
|
block 4 : IC=64 OC=32 stride=3 K=6 pad=2 output_pad=1
|
|
final : 32 -> 1
|
|
```
|
|
|
|
PyTorch ConvTranspose1d formula :
|
|
`T_out = (T_in - 1)*stride - 2*padding + dilation*(kernel - 1) + output_padding + 1`
|
|
|
|
With our parameters (d=1, k=2*s, p=ceil(s/2), op=s%2) the formula
|
|
collapses to `T_out = stride * T_in` exactly for all five blocks.
|
|
|
|
### Snake activation
|
|
|
|
DAC reference formula (Hugging Face `Snake1d.forward`) :
|
|
`y = x + (alpha + 1e-9).reciprocal() * sin(alpha * x)^2`
|
|
|
|
`ggml_snake(x, a, inv_b)` computes `y = x + sin^2(a * x) * inv_b`.
|
|
Mapping :
|
|
|
|
a = alpha (loaded direct, BF16 to F32)
|
|
inv_b = 1/(alpha + 1e-9) (precomputed CPU side at load, F32)
|
|
|
|
Both stored as F32 `[1, C]` tensors. `alpha` shape in checkpoint :
|
|
`(1, C, 1)`, ggml ne = (1, C, 1). C lives on ne[1]. Loader reads C from
|
|
`mt->ne[1]`.
|
|
|
|
### ConvTranspose1d via GEMM + col2im_1d
|
|
|
|
PyTorch `nn.ConvTranspose1d(IC, OC, kernel=K, stride=s, padding=p)` with
|
|
weight shape `(IC, OC, K)`. GGML decomposition :
|
|
|
|
```
|
|
1. Permute weight (IC, OC, K) PyTorch -> (IC, K*OC) ggml at load time.
|
|
Layout : dst[(oc*K + k) * IC + ic] = src[ic*OC*K + oc*K + k]
|
|
This makes k vary faster than oc inside the K*OC axis, matching
|
|
what ggml_compute_forward_col2im_1d_impl expects :
|
|
col_data[(oc * K + k) + t_in * K_OC]
|
|
|
|
2. Build runtime graph :
|
|
xt = ggml_cont(ctx, ggml_transpose(ctx, x)) # [IC, T_in]
|
|
col = ggml_mul_mat(ctx, w, xt) # [K*OC, T_in]
|
|
y = ggml_col2im_1d(ctx, col, stride, OC, padding) # [T_no_op, OC]
|
|
if (output_pad > 0)
|
|
y = ggml_pad(ctx, y, output_pad, 0, 0, 0) # right-pad zeros
|
|
if (bias)
|
|
y = ggml_add(ctx, y, bias_2d)
|
|
```
|
|
|
|
Validated math : `T_no_op = (T_in - 1)*stride + K - 2*pad`. Adding
|
|
`output_pad` right-pad gives the PyTorch output size exactly.
|
|
|
|
### RVQ codec
|
|
|
|
Per-codebook tensors (k = 0..7) :
|
|
|
|
```
|
|
codebook.embed (1024, 64) PyTorch -> ggml ne=(64, 1024)
|
|
project_in.weight (64, 1024) PyTorch -> ggml ne=(1024, 64) encode-only
|
|
project_in.bias (64,) encode-only
|
|
project_out.weight (1024, 64) PyTorch -> ggml ne=(64, 1024)
|
|
project_out.bias (1024,)
|
|
```
|
|
|
|
Decode graph (per codebook k, accumulated) :
|
|
|
|
```
|
|
codes_k = ggml_view_1d(codes, T, k * stride) # [T] i32
|
|
e_k = ggml_get_rows(embed[k], codes_k) # [64, T]
|
|
p_k = ggml_mul_mat(project_out_w[k], e_k) # [1024, T]
|
|
p_k = ggml_add(p_k, project_out_b[k])
|
|
acc += p_k
|
|
```
|
|
|
|
Encode (residual loop) :
|
|
|
|
```
|
|
residual = embeddings_in
|
|
for k in 0..7 :
|
|
e_k = project_in[k](residual)
|
|
codes_k = argmin_i ||e_k - codebook[k].embed[i]||^2
|
|
quantized = project_out[k](codebook[k].embed[codes_k])
|
|
residual -= quantized
|
|
```
|
|
|
|
### HuBERT semantic encoder
|
|
|
|
12 transformer layers Pre-LN, GELU FFN, MHA 12 heads * 64 dim, biases
|
|
on all QKVO. Pre-conv feature extractor : 7 Conv1D layers, kernels
|
|
`[10, 3, 3, 3, 3, 2, 2]`, strides `[5, 2, 2, 2, 2, 2, 2]`, GroupNorm on
|
|
the first only, GELU between. Feature projection LayerNorm + Linear
|
|
(512 -> 768). Positional embedding via grouped Conv1D (128 kernel,
|
|
16 groups), `weight_norm` folded at convert time. Final LayerNorm.
|
|
|
|
Output computation :
|
|
|
|
```
|
|
mean(stack(all_13_hidden_states, dim=1), dim=1) # (B, T_sem, 768)
|
|
```
|
|
|
|
This is unusual : the encoder averages across the initial input plus
|
|
the 12 transformer layer outputs, not just the last hidden state.
|
|
|
|
## Long-form TTS pipeline
|
|
|
|
`pipeline_tts_synthesize_long` orchestrates inputs longer than the
|
|
chunking threshold. It mirrors `_generate_chunked` in
|
|
`omnivoice/models/omnivoice.py`.
|
|
|
|
```
|
|
1. Estimate total target tokens via duration_estimate_tokens.
|
|
2. If T_total <= chunk_threshold_sec * frame_rate, run a single shot
|
|
pipeline_tts_synthesize and skip chunking.
|
|
3. Otherwise split text on punctuation with chunk_text_punctuation,
|
|
targeting chunk_duration_sec seconds per chunk.
|
|
4. Generate chunks sequentially :
|
|
- chunk 0 with no reference (auto voice / voice design path) or
|
|
with the external reference (cloning path)
|
|
- in the auto voice case, the audio tokens of chunk 0 become the
|
|
voice prompt for chunks 1..N, locking in the speaker identity
|
|
5. Cross-fade decoded chunks with cross_fade_chunks(rate, 0.3 s).
|
|
6. Apply post-processing on the merged waveform.
|
|
```
|
|
|
|
A shared Philox counter `ctr_lo` is threaded across MaskGIT calls so
|
|
PRNG state advances continuously between chunks, matching the global
|
|
`torch.cuda.manual_seed` behaviour on the reference side.
|
|
|
|
### Text chunking
|
|
|
|
`chunk_text_punctuation(text, chunk_len, min_chunk_len)` splits text on
|
|
sentence-ending punctuation (skipping abbreviation periods), then
|
|
merges sentences into chunks of at most `chunk_len` UTF-8 codepoints.
|
|
Undersized chunks (< `min_chunk_len`) are merged into a neighbour.
|
|
The function operates on UTF-8 strings and treats length as codepoints,
|
|
matching Python `len(str)` semantics. Per chunk character budget :
|
|
|
|
```
|
|
n_chars = utf8_codepoint_count(full_text)
|
|
avg_tokens_per_char = T_total / n_chars
|
|
chunk_len = (int)(chunk_duration_sec * frame_rate / avg_tokens_per_char)
|
|
```
|
|
|
|
`add_punctuation(text)` appends a terminal `.` (Latin) or its CJK
|
|
equivalent when missing. Used on the reference transcript when
|
|
`preprocess_prompt` is on.
|
|
|
|
### Audio post-processing
|
|
|
|
`audio-postproc.h` is a strict math port of `omnivoice/utils/audio.py`
|
|
plus the relevant `pydub.silence` routines. All public functions take
|
|
and return float32 mono PCM in [-1, 1] at the pipeline rate (24 kHz).
|
|
Silence detection runs on int16 samples to match pydub bit-for-bit.
|
|
|
|
```
|
|
remove_silence(buf, min_silence_ms, keep_silence_ms,
|
|
seek_step_ms, threshold_dbfs)
|
|
|
|
Splits buf on contiguous silent regions where every
|
|
seek_step_ms-long frame stays below threshold_dbfs (RMS, S16,
|
|
default -50 dBFS), keeps keep_silence_ms of leading and trailing
|
|
silence around each retained segment, and concatenates the result.
|
|
|
|
cross_fade_chunks(chunks, rate, fade_seconds)
|
|
|
|
Concatenates audio chunks with a linear cross-fade of fade_seconds
|
|
at each junction.
|
|
|
|
peak_normalize_half(buf)
|
|
|
|
Scales buf so peak |x| equals 0.5. Used in pure auto voice when no
|
|
reference RMS is available.
|
|
|
|
fade_and_pad(buf, rate, fade_seconds, pad_seconds)
|
|
|
|
Applies a linear fade in / fade out and pads silence at the start
|
|
and end. Default fade 0.05 s, pad 0.05 s. Mirrors final reference
|
|
post-step.
|
|
```
|
|
|
|
`ref_rms` is plumbed end to end and decides the volume branch :
|
|
|
|
ref_rms < 0 pure auto voice, peak_normalize_half on the
|
|
cross-faded waveform
|
|
ref_rms < 0.1 quiet reference, rescale by ref_rms / 0.1
|
|
otherwise no-op
|
|
|
|
When a reference WAV is provided, the CLI computes its RMS on the F32
|
|
samples after optional silence trimming and passes it down. The same
|
|
quantity is used on the PyTorch side.
|
|
|
|
### Voice modes
|
|
|
|
```
|
|
auto voice no ref-wav. Chunk 0 generates with no reference,
|
|
subsequent chunks reuse chunk 0 audio tokens as the
|
|
voice prompt. peak_normalize_half on output.
|
|
|
|
voice design no ref-wav, --instruct provides one or more attribute
|
|
markers (gender, age, pitch, style, volume, emotion)
|
|
resolved by voice_design.h to the EN/ZH instruct
|
|
string the reference uses. Chunking behaves like
|
|
auto voice.
|
|
|
|
voice cloning --ref-wav and --ref-text provided. The reference is
|
|
resampled to 16 kHz, run through the audio tokenizer
|
|
encoder, and the resulting RVQ codes are reused as
|
|
the voice prompt for every chunk. The reference RMS
|
|
sets the target loudness.
|
|
```
|
|
|
|
## Public API
|
|
|
|
Two layers, picked by use case.
|
|
|
|
### Top-level public ABI : src/omnivoice.h
|
|
|
|
Single-header, plain C99, linkage `extern "C"`. The opaque `ov_context`
|
|
handle aggregates the GGML backend pair, the LM pipeline, the audio
|
|
tokenizer codec, the BPE tokenizer and the voice-design vocabulary.
|
|
One init, one free, one synthesize call covers the full TTS path.
|
|
Same names, same struct layout, same calling convention from C, C++,
|
|
Python ctypes, Rust bindgen, Go cgo or any other binding generator.
|
|
|
|
```c
|
|
#include "omnivoice.h"
|
|
|
|
struct ov_init_params iparams;
|
|
ov_init_default_params(&iparams);
|
|
iparams.model_path = "models/omnivoice-base-Q8_0.gguf";
|
|
iparams.codec_path = "models/omnivoice-tokenizer-F32.gguf";
|
|
|
|
struct ov_context * ov = ov_init(&iparams);
|
|
|
|
struct ov_tts_params params;
|
|
ov_tts_default_params(¶ms);
|
|
params.text = "Hello world.";
|
|
params.lang = "English";
|
|
|
|
struct ov_audio audio = { 0 };
|
|
enum ov_status rc = ov_synthesize(ov, ¶ms, &audio);
|
|
if (rc == OV_STATUS_OK) {
|
|
/* audio.samples is a malloc'd buffer of audio.n_samples floats
|
|
at audio.sample_rate Hz, audio.channels = 1 (mono) */
|
|
}
|
|
ov_audio_free(&audio);
|
|
ov_free(ov);
|
|
```
|
|
|
|
Status codes :
|
|
|
|
```
|
|
OV_STATUS_OK 0
|
|
OV_STATUS_INVALID_PARAMS -1 (mutually exclusive ref inputs etc.)
|
|
OV_STATUS_INSTRUCT_INVALID -2 (instruct rejected by VoiceDesign)
|
|
OV_STATUS_GENERATE_FAILED -3 (any internal generate / decode fail)
|
|
OV_STATUS_OOM -4 (output samples allocation failed)
|
|
OV_STATUS_CANCELLED -5 (cancel callback returned true)
|
|
```
|
|
|
|
`ov_tts_params` exposes `cancel` and `cancel_user_data`. The pipeline
|
|
polls between chunks of long-form output, so cancel granularity is
|
|
roughly `chunk_duration_sec` (15 s by default).
|
|
|
|
The MaskGIT sampler config is flattened directly into `ov_tts_params`
|
|
as seven `mg_*` fields ; `ov_tts_default_params` initialises them to
|
|
the reference defaults (`num_step=32, guidance_scale=2.0, t_shift=0.1,
|
|
layer_penalty_factor=5.0, position_temperature=5.0,
|
|
class_temperature=0.0, seed=42`).
|
|
|
|
`ov_version()` returns a static string of the form
|
|
`"MAJOR.MINOR.PATCH (git-hash, date)"`. The macros `OV_VERSION_MAJOR`,
|
|
`OV_VERSION_MINOR`, `OV_VERSION_PATCH` are also available at
|
|
compile time for feature-detection.
|
|
|
|
### ABI guarantee
|
|
|
|
`tests/abi-c.c` is built on every build with
|
|
`-std=c99 -Wall -Werror -pedantic`. It includes the public header,
|
|
calls every entry through its early-return path, and is wired into
|
|
the default build target. Any regression that breaks plain C
|
|
consumability fails the main build, not an opt-in step.
|
|
|
|
The static library `libomnivoice-core.a` is the default build
|
|
artefact. For binding consumers, configure with
|
|
`-DOMNIVOICE_SHARED=ON` to add a `libomnivoice.so` (or `.dll` /
|
|
`.dylib`) shared target that exports only the `ov_*` symbols ;
|
|
every internal `pipeline_*` and `backend_*` stays hidden behind
|
|
`-fvisibility=hidden`. Install rules follow `GNUInstallDirs`.
|
|
|
|
### Low-level API : src/pipeline-tts.h, src/pipeline-codec.h
|
|
|
|
Direct access to the LM forward (`pipeline_tts_llm_forward`,
|
|
`pipeline_tts_llm_forward_batched`), the MaskGIT-only path
|
|
(`pipeline_tts_generate`), the codec encode / decode
|
|
(`pipeline_codec_encode`, `pipeline_codec_decode`), the instruct
|
|
resolver (`pipeline_tts_resolve_instruct`) and the manual init / free
|
|
(`pipeline_tts_load`, `pipeline_codec_load`, `backend_init`).
|
|
|
|
Used by `--llm-test` and `--maskgit-test` in `omnivoice-tts`, by
|
|
`omnivoice-codec` for the standalone codec roundtrip, and by the
|
|
Python cossim harness through dump files.
|
|
|
|
This layer is intentionally not part of the public ABI (C++ types in
|
|
the signatures, no visibility export). It exists for the in-tree
|
|
debug paths and stays available as long as the bundled CLI tools
|
|
need it. The handle layer above is the recommended entry for
|
|
everything else.
|
|
|
|
## CLI tools
|
|
|
|
### omnivoice-tts
|
|
|
|
End-to-end synthesis : text on stdin, WAV file on disk. Verbatim
|
|
`--help` (the binary also prints an `omnivoice.cpp <hash> (<date>)`
|
|
banner line first) :
|
|
|
|
```
|
|
Usage: omnivoice-tts --model <gguf> --codec <gguf> [options] -o <out.wav> < text.txt
|
|
|
|
Required:
|
|
--model <gguf> LLM GGUF (F32 / BF16 / Q8_0)
|
|
--codec <gguf> Codec GGUF (omnivoice-tokenizer-*.gguf)
|
|
-o <path> Output WAV (24 kHz mono). '-' streams to stdout (pipe friendly).
|
|
|
|
Input:
|
|
stdin Target text to synthesise. With -o '-', stdin is read
|
|
incrementally and synthesis starts as soon as the first
|
|
sentence boundary is reached. With -o file.wav, stdin is
|
|
read fully then synthesised in one shot.
|
|
--srt <path> Dub an SRT: synth each cue into its time slot, write one
|
|
timeline WAV ready to mux. Pairs with --ref-wav / --ref-rvq
|
|
for a cloned voice. Per cue duration comes from the SRT.
|
|
|
|
Optional:
|
|
--format <fmt> WAV output format: wav16, wav24, wav32 (default: wav16)
|
|
--lang <str> Language label (default 'None')
|
|
--instruct <str> Style instruction (default 'None')
|
|
--duration <sec> Output duration in seconds (default: estimate from text)
|
|
--no-denoise Omit the <|denoise|> prefix
|
|
--ref-wav <path> Reference WAV for voice cloning
|
|
--ref-text <path> Transcript file for the reference (required with --ref-wav / --ref-rvq)
|
|
--ref-rvq <path> Pre-encoded reference codes from omnivoice-codec (replaces --ref-wav)
|
|
--seed <int> Sampling seed (default: -1 for random)
|
|
--steps <int> MaskGIT decode steps (default: 32, fewer is faster)
|
|
--no-preprocess-prompt Skip ref-wav silence trim and ref-text terminal punctuation
|
|
--chunk-duration <sec> Long-form chunk duration (default: 15.0, <= 0 disables chunking)
|
|
--chunk-threshold <sec> Activate chunking above this estimated duration (default: 30.0)
|
|
--stream-by-line Flush synthesis at each newline, one WAV header per line (-o '-')
|
|
|
|
Debug:
|
|
--no-fa Disable flash attention
|
|
--clamp-fp16 Clamp hidden states to FP16 range
|
|
--dump <dir> Dump intermediate tensors (f32) to <dir>
|
|
--llm-test <input.bin> Full LLM forward, dump audio_logits
|
|
--maskgit-test Greedy MaskGIT decoder, dump audio_tokens [K, T]
|
|
(no codec decode, reads target text from stdin)
|
|
```
|
|
|
|
### omnivoice-codec
|
|
|
|
Audio tokenizer round-trip : WAV to RVQ codes, RVQ codes to WAV.
|
|
Verbatim `--help` :
|
|
|
|
```
|
|
Usage: omnivoice-codec --model <gguf> -i <input>
|
|
|
|
Required:
|
|
--model <gguf> Codec GGUF (omnivoice-tokenizer-*.gguf)
|
|
-i <path> Input. WAV -> encode, .rvq -> decode
|
|
|
|
Optional:
|
|
--format <fmt> WAV output format: wav16, wav24, wav32 (default: wav16)
|
|
|
|
Output is auto-named next to input : clip.wav -> clip.rvq, clip.rvq -> clip.wav.
|
|
Encode applies the TTS reference preprocessing (RMS auto-gain, silence trim,
|
|
hop truncation); the resulting .rvq feeds omnivoice-tts --ref-rvq directly.
|
|
```
|
|
|
|
The `.rvq` file is a small binary container with shape `[8, T]` int32
|
|
codes plus a header carrying the sample rate and frame rate.
|
|
|
|
## Module map
|
|
|
|
```
|
|
src/
|
|
backend.h GGML backend init, scheduler factory, env override
|
|
weight-ctx.h Generic weight context for GGUF loaders
|
|
gguf-weights.h mmap GGUF, gf_load_tensor, gf_get_*
|
|
audio-io.h WAV read, mono write (S16 / S24 / F32)
|
|
audio-resample.h Kaiser polyphase 24 kHz <-> 16 kHz
|
|
audio-postproc.h remove_silence, peak_normalize_half, fade_and_pad,
|
|
cross_fade_chunks. Strict pydub / utils.audio port.
|
|
wav.h WAV header reader (PCM16/24/F32, mono/stereo)
|
|
philox.h Philox4x32-10 counter-based PRNG, PyTorch CUDA aligned
|
|
debug.h Tensor dumper for cossim tests
|
|
|
|
bpe.h Qwen2 / GPT-2 byte-level BPE tokenizer, GGUF loader
|
|
lang-map.h Language name to ISO 639-3 ID resolution
|
|
(auto-generated from omnivoice/utils/lang_map.py)
|
|
voice-design.h Speaker attribute validation and EN / ZH instruct
|
|
resolution (mirrors voice_design.py)
|
|
text-chunker.h chunk_text_punctuation, add_punctuation, END_PUNCTUATION
|
|
duration-estimator.h RuleDurationEstimator port (per-script weights,
|
|
Unicode category fallback)
|
|
|
|
rvq-codec.h Residual VQ encode + decode (8 codebooks)
|
|
dac-decoder.h DAC acoustic decoder (5 blocks, ratios 8 5 4 2 3)
|
|
dac-encoder.h DAC acoustic encoder (mirror of decoder)
|
|
semantic-enc.h SemanticEncoder convs (768 -> 768)
|
|
hubert-enc.h HuBERT base (feature extractor + pos_conv +
|
|
12 transformer layers + final LN)
|
|
|
|
qwen3-enc.h Qwen3 transformer building blocks
|
|
omnivoice-llm.h OmniVoice TTS LLM weights and graph helpers
|
|
prompt-tts.h Prompt builder (denoise + lang + instruct + text +
|
|
ref + mask) and CFG batch stacking
|
|
maskgit-tts.h Iterative non autoregressive decoder, configurable
|
|
step count (32 default), CFG, layer penalty, gumbel
|
|
sampling, deterministic in greedy mode
|
|
|
|
pipeline-codec.{h,cpp} Audio tokenizer end-to-end (encode and decode)
|
|
pipeline-tts.{h,cpp} Full TTS orchestration, single shot and chunked,
|
|
plus low-level entries kept available for the
|
|
debug paths and the cossim test harness
|
|
omnivoice.{h,cpp} Public ABI : opaque ov_context handle, plain C99
|
|
header in extern "C", consumable from C, C++,
|
|
Python ctypes, Rust bindgen, Go cgo
|
|
|
|
tools/
|
|
omnivoice-tts.cpp CLI : text to WAV (auto / design / clone)
|
|
omnivoice-codec.cpp CLI : codes <-> WAV
|
|
quantize.cpp GGUF requantizer
|
|
version.cmake Embeds the git short hash into the binary
|
|
|
|
tests/
|
|
debug-tts-cossim.py Byte-level comparison of every pipeline stage
|
|
against the PyTorch reference, voice design path
|
|
debug-clone-cossim.py Same, voice cloning path
|
|
cross-decode.py Cross check : decode C++ tokens through PyTorch
|
|
codec and vice versa
|
|
prompt.txt Long-form English TTS sample
|
|
ref-audio.wav Voice cloning reference clip
|
|
ref-text.txt Transcript matching ref-audio.wav
|
|
abi-c.c Plain C99 smoke test for the public ABI ; built
|
|
with -Wall -Werror -pedantic on every build,
|
|
locks in C consumability and symbol linkage
|
|
```
|
|
|
|
## GGML conventions
|
|
|
|
### Tensor shape and layout
|
|
|
|
PyTorch shape `(out, in)` for a Linear weight stores as ggml
|
|
`ne[0]=in, ne[1]=out`. The GGUF tensor-shape array is reversed, so
|
|
reading `reversed(t.shape)` from gguf-py yields the PyTorch shape
|
|
directly.
|
|
|
|
For PyTorch `Conv1d` weight `(OC, IC, K)`, ggml ne is `(K, IC, OC)`.
|
|
The kernel axis is innermost (contiguous in memory).
|
|
|
|
For PyTorch `ConvTranspose1d` weight `(IC, OC, K)`, the convert-time
|
|
permutation to ggml `(IC, K*OC)` rearranges
|
|
`(oc*K + k) * IC + ic` so that `ggml_col2im_1d` receives the correct
|
|
column matrix.
|
|
|
|
`ggml_mul_mat(A, B)` : with A.ne[0] = K (must match B.ne[0]),
|
|
A.ne[1] = M, B.ne[1] = N, output has ne = (N, M). In PyTorch terms,
|
|
A is `(M, K)`, B is `(N, K)`, output is `(M, N)`, which equals
|
|
`A @ B^T`.
|
|
|
|
### Custom GGML ops
|
|
|
|
Provided by the `ServeurpersoCom/ggml` fork :
|
|
|
|
`ggml_snake(ctx, x, a, inv_b)` : `y = x + sin^2(a * x) * inv_b`.
|
|
F32 / F16 / BF16 input/output. CPU + CUDA + Metal + Vulkan.
|
|
|
|
`ggml_col2im_1d(ctx, a, s0, oc, p0)` : scatter-add `[K*OC, T_in]`
|
|
columns into `[T_out, OC]` signal where
|
|
`T_out = (T_in - 1)*s0 + K - 2*p0`. Layout requires k to vary faster
|
|
than oc inside the K*OC axis. F32 / F16 / BF16. All backends. The
|
|
fork also folds the padding crop into this op via the `p0` parameter,
|
|
removing a follow-up `ggml_view` for the typical ConvTranspose1d use
|
|
case.
|
|
|
|
### Backend lifecycle
|
|
|
|
`backend_init("MOD")` then `backend_sched_new(bp, max_nodes)`. Backend
|
|
handles are shared across modules in the same binary, refcounted. The
|
|
GPU backend is the default, the CPU backend is kept as a scheduler
|
|
fallback.
|
|
|
|
## Validation
|
|
|
|
The reference comparison harness is `tests/debug-tts-cossim.py` and
|
|
`tests/debug-clone-cossim.py`. They run the same input through the
|
|
PyTorch reference (with TF32 disabled, eager attention) and through the
|
|
C++ binary, dump each pipeline stage to disk, and report cosine
|
|
similarity per stage. Latest run, chunked path, English long-form
|
|
prompt :
|
|
|
|
```
|
|
TTS chunked: Logits cos=1.000000 max 3.5e-04
|
|
Step0 pred_tokens 99.93% (2 FP flips)
|
|
Tokens 1.000000 exact 100.00%
|
|
Audio 0.999991
|
|
|
|
Clone chunked: Lf hidden cos=1.000000 max 1.9e-03
|
|
Logits cos=1.000000
|
|
Step0 pred_tokens 100.00%
|
|
Tokens 1.000000 exact 100.00%
|
|
Audio 0.999989
|
|
```
|
|
|
|
The few Step0 token flips are argmax ties at the FP epsilon (~2e-5
|
|
between top1 and top2 at those positions), inherent to the mixed cuBLAS
|
|
vs GGML kernel arithmetic. They resorb over the 32 MaskGIT steps so
|
|
the final tokens match bit for bit and decoded audio cosine is
|
|
> 0.9999.
|
|
|
|
## Glossary
|
|
|
|
RVQ Residual Vector Quantisation. Stack of codebooks where
|
|
each one quantises the residual from the previous
|
|
codebook reconstruction.
|
|
|
|
DAC Descript Audio Codec. Convolutional encoder/decoder over
|
|
residual VQ codes.
|
|
|
|
HuBERT Hidden-Unit BERT. Transformer encoder pretrained with
|
|
masked acoustic unit prediction. Used here to extract
|
|
semantic embeddings from raw audio.
|
|
|
|
Snake Periodic activation introduced in BigVGAN,
|
|
`y = x + (1/alpha) * sin^2(alpha * x)`. Replaces
|
|
LeakyReLU in the DAC encoder/decoder.
|
|
|
|
CFG Classifier-Free Guidance. The model is run twice
|
|
(conditional and unconditional) and the outputs combined
|
|
as `c + scale * (c - u)` to amplify the conditional
|
|
signal.
|
|
|
|
MaskGIT Masked Generative Image Transformer (Chang et al.,
|
|
arXiv:2202.04200). Iterative non autoregressive decoder
|
|
where masked tokens are progressively unmasked over a
|
|
fixed number of steps, prioritising high-confidence
|
|
positions per step. Originally introduced for image
|
|
generation, adapted here to audio codes.
|
|
|
|
Philox Counter-based PRNG used by PyTorch CUDA. Thread safe and
|
|
skip-ahead friendly, well suited to deterministic
|
|
chunked inference.
|