Parameter Golf, a 16 MB language model
Three days of experiments on OpenAI's Parameter Golf challenge, where the whole model must fit in 16 MB and train in 10 minutes. The best run scored well but came out too large, and nothing was submitted.
Question
OpenAI’s Parameter Golf asks for the best language model whose code plus compressed weights fit in 16 MB, trained in under 10 minutes on 8 GPUs. The score is bits per byte on the FineWeb validation set (how many bits the model needs per byte of unseen text; lower is better). The question here was which ideas buy the most quality per byte.
Approach
Each version started from the challenge’s starter script or from a published leaderboard entry and changed one or two things. Runs went through a Docker image on rented multi-GPU machines (4 or 8 A100s). Vocabulary was the challenge’s 1,024-token tokenizer.
| version | idea |
|---|---|
| v2 | Per-matrix codebook quantization: cluster each weight matrix’s values and store small indices instead of 8-bit numbers |
| v3, v4 | Hard-example replay: keep the batches with the highest loss and train on them again, later with a schedule that ramps replay up |
| v5 | Mixed sequence lengths during training |
| v6 | 12 layers with 4-bit codebook weights, so a deeper model fits the same budget |
| v7, v8 | Depth recurrence: 9 layers run twice for 18 effective layers, then a learned scale per pass |
| v9 – v11 | Optimizer and batch tuning, 14 layers with a 6-bit codebook |
| v12 – v14 | The same ideas grafted onto stronger leaderboard scripts (sliding-window eval, LoRA test-time training, longer context) |
| v15 – v25 | Learning rate, warmdown, weight averaging, batch size, 10 to 16 layers, per-layer bit widths |
| v26 | In-run distillation: train a 32-layer teacher for 5 minutes, then a 10-layer student against it |
| v27 | Train 20 layers for 5 minutes, keep every second layer, continue with 10 |
Side tools: a script to re-quantize a saved model at different bit widths without retraining, and a shard scorer that ranks training documents by repetition and vocabulary spread.
Results
The full 10-minute run below used a 14-layer, 26.2M-parameter model with a 1M-token batch, on top of a leaderboard script that already used sliding-window evaluation.
| run | val bpb | artifact size | valid entry |
|---|---|---|---|
| Challenge naive baseline (9 layers, published) | 1.2244 | under 16 MB | yes |
| Sliding-window eval entry this run built on (9 layers, published) | 1.1925 | under 16 MB | yes |
| This run, at the 10-minute stop, plain eval, before compression | 1.2000 | – | – |
| This run, compressed to int8 + zlib, sliding-window eval | 1.1696 | 24.3 MB | no |
The run took 4,589 steps in 600 seconds (about 131 ms per step on 8 GPUs). An early laptop smoke test of the Apple MLX version reached 2.41 bpb after 200 steps.
Findings
- More layers bought real quality: 14 layers beat the published 9-layer entry by about 0.02 bpb.
- That gain was paid for in bytes. At 8-bit precision the 14-layer model weighed 24.3 MB, about 50% over the cap. The codebook quantization was built to close that gap.
- Most of the leaderboard’s progress in those days came from many small stacked tweaks (sliding-window eval, fp16 embeddings, mixed bit widths, weight decay), not one big idea.
Next
Closed. No entry was submitted. The experiment shaped later work: the Muon optimizer, the bits-per-byte metric and the FineWeb data format carried straight into the 336M model.