Research – Foundations

Learning the update rule

Attempts to replace the optimizer with something learned, or to build a better update from a weight's own history. Small networks that propose updates, learned corrections to the gradient, and history-based fixes on a toy. The learned optimizers finished well behind Muon.

StartedMay 24, 2026
Last updateNov 2, 2026
bits per byte for the best learned optimizer, vs 2.04 for Muon on the same model2.89

Question

An optimizer is a fixed rule that turns a gradient into a weight change. Muon, for example, orthogonalizes it (flattens its strongest directions so no single one dominates). Can a small network learn a better rule from experience? And can a weight’s past movement say where it should go next?

Approach

Most learned rules were trained by lookahead: apply the proposed update to a copy of the model, score the copy on a batch, and push that score back into the rule.

Rule Date Scripts What is learned
“Buddy” optimizer May 24 lopt, train_lopt One tiny MLP shared by every weight. It reads the gradient and momentum of each number and outputs its update, replacing Muon on the weight matrices. Set up to imitate Muon first, then switch to lookahead; the runs below used lookahead only.
Attention optimizer May 24 attn_opt, train_attn Rows of each weight’s momentum attend to each other, so the update mixes across weights the way Muon’s orthogonalization does.
Update gate May 24 gate, train_gate A learned 0 to 1 scale per tensor on top of Muon. Starts fully open.
Learned Muon coefficients May 25 train_muoncoef The polynomial inside Muon’s orthogonalization, tuned by lookahead instead of fixed.
Gradient correction side-net May 25 train_lookahead_attn Proposes a correction to the raw gradient, judged on a separate batch. The real model keeps training on the raw gradient.
Predicting weight paths May 25 trend_predict, predict_tok, predict_joint, predict_bucket, predict_attn, predict_wattn, predict_synthetic Predict each weight’s next change from its recent values, singly, jointly, as up/flat/down, or with attention over time or over weights.
AdaMuon Jun 2 optim.py A hand-written rule, not learned: each tensor’s momentum is set by how well its new gradient lines up with the old momentum.
Learned error signal Jun 9 metaopt, metalo, metalo_tf A small network adds a correction to the error at the output (the “predicted minus true” signal backprop starts from), trained by lookahead through the update. First on an MLP language model, then on a one-block transformer.
Oracle distillation Jun 9 oracdist Snapshots of one run each get some tokens right. Train the live model toward whichever snapshot was least wrong on each token.
Weight-history fixes Nov 2 histopt, histopt_lm Every 50 steps, look at how each neuron’s weights moved. Extrapolate neurons still heading the same way, roll back ones that got less useful, or push apart pairs that are converging. Each fix is kept only if it lowers loss on a held-out probe set. First on a parity toy over 20 seeds, then on a small MLP language model, plus a variant that scales each row’s Adam step by how straight its path has been.

The May runs used a 1-layer, 1.18M-parameter transformer on FineWeb (1,024-piece vocabulary), 3,000 steps and 49M tokens each, on an RTX 5090 or similar. Scores are bits per byte: how surprised the model is by each byte of text, lower is better.

Results

Learned optimizers vs hand-written ones (same model, data and budget, May 24–25)

Optimizer Bits/byte vs Muon
Muon, the control 2.037 –
AdamW 2.296 +0.26
Plain SGD 2.649 +0.61
Buddy, hidden width 64 2.891 +0.85
Buddy, hidden width 256 3.321 +1.28
Attention optimizer, residual, meta lr 0.03 3.213 +1.18
Attention optimizer, first version, meta lr 0.1 3.475 +1.44

Findings

  • Every learned optimizer lost, and by a lot. The best buddy was 0.85 bits per byte behind Muon and also behind plain SGD. A bigger buddy did worse.
  • Mixing across weights, which is what makes Muon work, did not rescue the learned version. The attention optimizer was worse than the per-weight buddy, though a residual form with a lower meta learning rate recovered 0.26.
  • Lookahead must be scored on a different batch from the one the gradient came from. Scored on the same batch, the side-net learned only to scale every update up, which just overfits that batch.
  • Predicting a weight’s next change from its own history did not work on real training. A synthetic control (targets that are a known function of the same inputs) was set up to check whether the predictor or the data was at fault.
  • Averaging the weights of snapshots from one run fails because they sit in different basins. That is why oracle distillation pushes outputs, not weights.
  • On the parity toy, extrapolating weight trends was a mixed bag, and rolling neurons back needs a forward pass per neuron, too costly for a real model. Only “push converging rows apart” was carried to the language model. A raw cosine there says every output row has collapsed, since they all share one common direction, so similarity has to be measured after removing the mean row.

Next

  • Run histopt_lm: push-apart and per-row step sizes vs plain Adam, 5 seeds each.
  • Score the 20-seed parity toy comparison and the June 9 learned error signal.