Speech from scratch
A tiny text-to-speech model trained on one voice, first predicting raw audio directly, then spectrograms through a pretrained vocoder. Alongside it, tests of open voice models and a trick to make one of them generate faster.
Question
Why does nearly every speech model predict spectrograms instead of raw audio? The plan was to build the naive version first, hear exactly how it fails, then upgrade one step at a time.
Approach
Voice data. 157 recordings of one speaker, made on 2026-03-14 by reading from a list of 500 short prompts (greetings and everyday lines). Whisper (a speech-to-text model) transcribed each clip, and clips were matched back to their prompt text. Clips that matched poorly were dropped.
Own TTS models. Both are small causal transformers (4 layers, 256 wide) that read the text one character at a time and then generate audio step by step, with a stop signal deciding when to end.
| Version | Predicts per step | Audio out |
|---|---|---|
| Raw waveform | 240 audio samples (10 ms at 24 kHz). Loss = squared error plus a shape-match term. | Samples joined end to end |
| Mel | One mel frame, 100 numbers describing which pitches are loud (a spectrogram slice, ~10.7 ms) | Pretrained BigVGAN vocoder turns frames into sound |
The mel version had a second run with noise added to the training inputs. A model trained only on clean inputs can fall apart once it has to read its own imperfect outputs.
Open voice models.
| Model | What was done |
|---|---|
| PersonaPlex-7B | A full-duplex voice model (it listens and talks at once). Hosted at full precision on an A100 80GB with a turn-based test UI and a real-time WebSocket mode. |
| Sesame CSM-1B | Text in, audio out, conditioned on three of the voice recordings. Built into a streaming web app. Then used for the parallel-heads experiment below. |
| Qwen voice model | One zero-shot voice clone and one sample from a version fine-tuned on the voice. |
Parallel heads. CSM-1B makes each audio frame as 32 codes, and a small decoder fills in 31 of them one after another. New heads were trained on the voice recordings to predict 4 codes per step, giving 8 steps per frame instead of 31. A hybrid made the first 4 codes the normal way and the rest in parallel (11 steps).
Results
Lengths of the generated clips:
| Output | Clips | Length |
|---|---|---|
| Raw waveform, training sentences | 10 | 2.2–6.0 s. 4 of 10 stop at exactly 6.0 s, the generation cap, so the model never said “done” |
| Mel + vocoder, overfit on one clip | 1 | 2.17 s, next to the same clip passed straight through the vocoder (2.15 s) |
| Mel + vocoder, full run | 5 train, 5 held out | 3.1–16.0 s. 2 held-out clips stop at exactly 16.0 s, the cap |
| Mel + vocoder, noise run | 5 train, 5 held out | 2.0–10.2 s. None hit the cap |
| Qwen zero-shot clone | 1 | 3.8 s |
| Qwen fine-tuned | 1 | 2 min 44 s |
| CSM-1B, normal decoding (31 steps) | 3 | 3.2–3.8 s |
| CSM-1B, hybrid (11 steps) | 3 | 4.0–4.9 s |
| CSM-1B, parallel (8 steps) | 3 | All exactly 10.0 s |
Speed work on the CSM-1B app:
| Change | Outcome |
|---|---|
| bfloat16 weights | About 2× faster on A100. Kept |
| INT8 (bitsandbytes) | 3.5× slower, dequantizing costs more than it saves on a 1B model. Reverted |
| FP8 (torchao) | Would not build against PyTorch 2.8. Reverted |
| Fewer codes per frame (32 to 16, then 24) | Halves the decoder passes |
Findings
- The raw-waveform model often failed to end. 4 of its 10 training-sentence clips ran to the 6-second cap.
- A clip passed straight through the vocoder gives a reference next to the model’s output, so any gap in quality comes from the transformer, not the vocoder.
- Noisy training inputs lined up with fewer runaway clips. No clip in the noise run hit the cap, against 2 of 10 in the clean run.
- Fully parallel heads broke stopping in CSM-1B. All three parallel outputs ran to exactly 10.0 s, while the hybrid stayed close to normal length. Keeping the first few codes sequential looks necessary.
- On a 1B model, 8-bit quantization made generation slower, not faster.