Research – Foundations

Parameter Golf, a 16 MB language model

Three days of experiments on OpenAI's Parameter Golf challenge, where the whole model must fit in 16 MB and train in 10 minutes. The best run scored well but came out too large, and nothing was submitted.

StartedMar 19, 2026
Last updateMar 21, 2026
bits per byte on FineWeb val, but at 24.3 MB, over the 16 MB cap1.170

Question

OpenAI’s Parameter Golf asks for the best language model whose code plus compressed weights fit in 16 MB, trained in under 10 minutes on 8 GPUs. The score is bits per byte on the FineWeb validation set (how many bits the model needs per byte of unseen text; lower is better). The question here was which ideas buy the most quality per byte.

Approach

Each version started from the challenge’s starter script or from a published leaderboard entry and changed one or two things. Runs went through a Docker image on rented multi-GPU machines (4 or 8 A100s). Vocabulary was the challenge’s 1,024-token tokenizer.

version idea
v2 Per-matrix codebook quantization: cluster each weight matrix’s values and store small indices instead of 8-bit numbers
v3, v4 Hard-example replay: keep the batches with the highest loss and train on them again, later with a schedule that ramps replay up
v5 Mixed sequence lengths during training
v6 12 layers with 4-bit codebook weights, so a deeper model fits the same budget
v7, v8 Depth recurrence: 9 layers run twice for 18 effective layers, then a learned scale per pass
v9 – v11 Optimizer and batch tuning, 14 layers with a 6-bit codebook
v12 – v14 The same ideas grafted onto stronger leaderboard scripts (sliding-window eval, LoRA test-time training, longer context)
v15 – v25 Learning rate, warmdown, weight averaging, batch size, 10 to 16 layers, per-layer bit widths
v26 In-run distillation: train a 32-layer teacher for 5 minutes, then a 10-layer student against it
v27 Train 20 layers for 5 minutes, keep every second layer, continue with 10

Side tools: a script to re-quantize a saved model at different bit widths without retraining, and a shard scorer that ranks training documents by repetition and vocabulary spread.

Results

The full 10-minute run below used a 14-layer, 26.2M-parameter model with a 1M-token batch, on top of a leaderboard script that already used sliding-window evaluation.

run val bpb artifact size valid entry
Challenge naive baseline (9 layers, published) 1.2244 under 16 MB yes
Sliding-window eval entry this run built on (9 layers, published) 1.1925 under 16 MB yes
This run, at the 10-minute stop, plain eval, before compression 1.2000 – –
This run, compressed to int8 + zlib, sliding-window eval 1.1696 24.3 MB no

The run took 4,589 steps in 600 seconds (about 131 ms per step on 8 GPUs). An early laptop smoke test of the Apple MLX version reached 2.41 bpb after 200 steps.

Findings

  • More layers bought real quality: 14 layers beat the published 9-layer entry by about 0.02 bpb.
  • That gain was paid for in bytes. At 8-bit precision the 14-layer model weighed 24.3 MB, about 50% over the cap. The codebook quantization was built to close that gap.
  • Most of the leaderboard’s progress in those days came from many small stacked tweaks (sliding-window eval, fp16 embeddings, mixed bit widths, weight decay), not one big idea.

Next

Closed. No entry was submitted. The experiment shaped later work: the Muon optimizer, the bits-per-byte metric and the FineWeb data format carried straight into the 336M model.