Research – Foundations

A 336M language model from scratch

A 24-layer transformer trained from zero on 1.8B tokens of FineWeb-Edu on one rented GPU, then used to test whether new layers stacked on a frozen model can add anything.

StartedOct 16, 2026
Last updateOct 18, 2026
bits per byte on held-out FineWeb-Edu after one pass over 1.8B tokens0.989

Question

How good a base language model can one consumer GPU train in a night? And once it is trained, can more depth be added cheaply by freezing it and training only new layers on top?

Approach

A plain decoder-only transformer, trained once over fresh data with no repeats. The tokenizer is a 32,768-piece SentencePiece model trained on FineWeb-Edu (about 4.7 bytes of text per token).

Layers x width x heads 24 x 1024 x 16
Parameters 335.9M
Data FineWeb-Edu sample-10BT, 1.8B tokens (28,000 steps x 64K tokens)
Optimizer Muon on the block matrices with a “cautious” mask (updates that disagree with the gradient are dropped), AdamW on the rest
Hardware One RunPod RTX 5090, about 9.5 hours, resumed once from a checkpoint
Metric Bits per byte (bpb): how many bits the model needs per byte of unseen text; lower is better

The stacking test then took the finished model, froze it, and added 12 or 24 new layers on top. The new layers started as a no-op (zeroed output projections), so step one matched the base exactly. Each test got 2,000 steps (130M tokens) of held-out shards. A control simply kept training the unfrozen base for the same 2,000 steps.

Results

Base run, validation bpb as training went on:

step tokens seen val bpb next-token accuracy
1,000 65M 1.400 26.5%
4,000 262M 1.232 31.1%
10,000 655M 1.108 35.2%
20,000 1.31B 1.026 37.9%
28,000 1.84B 0.989 39.6%

Stacking, all scored on the same validation slice (20 batches):

run layers trainable val bpb after 2,000 steps accuracy
base (no extra training) 24 336M 0.989 39.6%
control: base, keep training 24 336M 0.993 39.4%
24 frozen + 12 new 36 151M 0.997 39.1%
24 frozen + 24 new 48 302M 1.008 38.5%

Findings

  • A 336M model reached 0.989 bpb on FineWeb-Edu in one pass over 1.8B tokens, on a single 5090 in about 9.5 hours.
  • Nothing beat the finished base at a 130M-token budget. Not more training of the base, and not frozen-plus-new layers.
  • More new layers did worse (+12 layers: 0.997, +24 layers: 1.008). The same pattern showed up earlier in a tiny 64-wide toy.
  • The new layers did learn (their output weights grew to about a quarter of the base layers’ size), so this is not a dead-layer bug. A stacking stage seems to need a token budget on the order of the original run, not 2,000 steps.
  • The validation slice is small, so differences under about 0.01 bpb should be read as noise. The control used a lower learning rate than the stacking runs.

Next

  • Distillation: the same 24 x 1024 shape trained against Llama 3.2 1B as a teacher (on Llama’s tokenizer, same 1.8B-token budget) so the two runs can be compared at equal bpb. The recipe and data pipeline are ready.
  • A multi-token output head (one forward pass predicts several next bytes at once) was prepared for the same GPU. A 3M-parameter laptop version predicted the first byte right about 60% of the time and emitted about 1.6 bytes per step at a 0.8 confidence cutoff.