A 336M language model from scratch
A 24-layer transformer trained from zero on 1.8B tokens of FineWeb-Edu on one rented GPU, then used to test whether new layers stacked on a frozen model can add anything.
Question
How good a base language model can one consumer GPU train in a night? And once it is trained, can more depth be added cheaply by freezing it and training only new layers on top?
Approach
A plain decoder-only transformer, trained once over fresh data with no repeats. The tokenizer is a 32,768-piece SentencePiece model trained on FineWeb-Edu (about 4.7 bytes of text per token).
| Layers x width x heads | 24 x 1024 x 16 |
| Parameters | 335.9M |
| Data | FineWeb-Edu sample-10BT, 1.8B tokens (28,000 steps x 64K tokens) |
| Optimizer | Muon on the block matrices with a “cautious” mask (updates that disagree with the gradient are dropped), AdamW on the rest |
| Hardware | One RunPod RTX 5090, about 9.5 hours, resumed once from a checkpoint |
| Metric | Bits per byte (bpb): how many bits the model needs per byte of unseen text; lower is better |
The stacking test then took the finished model, froze it, and added 12 or 24 new layers on top. The new layers started as a no-op (zeroed output projections), so step one matched the base exactly. Each test got 2,000 steps (130M tokens) of held-out shards. A control simply kept training the unfrozen base for the same 2,000 steps.
Results
Base run, validation bpb as training went on:
| step | tokens seen | val bpb | next-token accuracy |
|---|---|---|---|
| 1,000 | 65M | 1.400 | 26.5% |
| 4,000 | 262M | 1.232 | 31.1% |
| 10,000 | 655M | 1.108 | 35.2% |
| 20,000 | 1.31B | 1.026 | 37.9% |
| 28,000 | 1.84B | 0.989 | 39.6% |
Stacking, all scored on the same validation slice (20 batches):
| run | layers | trainable | val bpb after 2,000 steps | accuracy |
|---|---|---|---|---|
| base (no extra training) | 24 | 336M | 0.989 | 39.6% |
| control: base, keep training | 24 | 336M | 0.993 | 39.4% |
| 24 frozen + 12 new | 36 | 151M | 0.997 | 39.1% |
| 24 frozen + 24 new | 48 | 302M | 1.008 | 38.5% |
Findings
- A 336M model reached 0.989 bpb on FineWeb-Edu in one pass over 1.8B tokens, on a single 5090 in about 9.5 hours.
- Nothing beat the finished base at a 130M-token budget. Not more training of the base, and not frozen-plus-new layers.
- More new layers did worse (+12 layers: 0.997, +24 layers: 1.008). The same pattern showed up earlier in a tiny 64-wide toy.
- The new layers did learn (their output weights grew to about a quarter of the base layers’ size), so this is not a dead-layer bug. A stacking stage seems to need a token budget on the order of the original run, not 2,000 steps.
- The validation slice is small, so differences under about 0.01 bpb should be read as noise. The control used a lower learning rate than the stacking runs.
Next
- Distillation: the same 24 x 1024 shape trained against Llama 3.2 1B as a teacher (on Llama’s tokenizer, same 1.8B-token budget) so the two runs can be compared at equal bpb. The recipe and data pipeline are ready.
- A multi-token output head (one forward pass predicts several next bytes at once) was prepared for the same GPU. A 3M-parameter laptop version predicted the first byte right about 60% of the time and emitted about 1.6 bytes per step at a 0.8 confidence cutoff.