Research – Foundations

What one training step does

Probes of gradients and single updates in tiny FineWeb transformers. How steady a gradient's direction is, what happens when one step is repeated or reweighted, and where training forgets.

StartedMay 24, 2026
Last updateJun 9, 2026
average cosine between one gradient and the next on weight matrices (about 0.7 on the final norm)0.04

Question

A training step takes the gradient of one batch (the direction that lowers loss on it) and moves every weight a little that way. What does that step actually do to the model? Do batches agree on where to go, does pushing harder help, and is the forgetting seen during training real or just noise?

Approach

Small transformers on FineWeb text (the 10B-token sample, cut into a 1,024-piece vocabulary), trained on an RTX 5090 or similar. Most probes load a trained model, take gradients without applying them, and compare.

Group Dates Scripts What was looked at
Drift and snapshots May 24–25 snapshot_compare, weight_divergence, same_init_diff_data, single_tensor_sweep, greedy_focus, alternate_train, warmup_freeze, perturb_search Which weights keep moving and which settle. How far models drift apart from the same start on different data. Which single tensor, trained alone, helps most. How much the output moves when random weights are nudged.
Gradient history May 25 train.py (gradient history), grad_inspect, analyze_snapshots, batch_grad_corr, gather_grads, inspect_grad, top_updates, sum_deltas, trend_predict Per-tensor gradient history over a run. Cosine between consecutive gradients, net motion vs back-and-forth, whether similar batches give similar gradients, whether weight paths can be extrapolated.
Loss weighting Jun 8 bands, layerbands, tokw, temps, disw, diverse, twolm, deeplm Splitting or reweighting tokens by difficulty: copies (or layers) that each learn only easy, medium or hard tokens; hard-token weighting (the Rho-1 idea); temperature-controlled focus; weighting by disagreement between differently seeded copies. Two side models test how far a hidden state can “coast” to predict several tokens ahead.
Forgetting Jun 9, early tokgrad, snapens, trajclass, overfitdir, gradcomp Per-token gradients that cancel inside a batch. Tokens an older snapshot gets right and the latest gets wrong. Each token’s right/wrong history across snapshots (stable, learned, lost, or flickering). Whether a fully fit direction agrees across batches more than a single gradient does.
One step, up close Jun 9, late morning trainsave, gradlook, grad5, stepimprove, applytwice, greedyfit, gradpath, gradpath2, gradpath3, gradvalue, quarters A small one-layer transformer (64 wide, 30,000 steps by default) is trained first. Then: the spread of gradient values, five gradients at the same point compared, one step’s gain on its own batch, the same gradient applied twice, reapplying one gradient until it stops helping, how hard fitting one batch changes what the next batch wants, weight value vs gradient size, and four models each trained on a random quarter of the weights.

Scores are in bits per byte: how surprised the model is by each byte of text, lower is better.

Results

Optimizer and step settings (1-layer transformer, 1.18M parameters, 3,000 steps, 49M tokens unless noted, May 24–25)

Setup Bits/byte vs Muon
Muon (orthogonalized steps), the control 2.037 –
AdamW, same budget 2.296 +0.26
Plain SGD, lr 0.1 2.649 +0.61
Muon, optimizer state reset every 976 steps 2.042 +0.005
Muon, batch 1 with 64-step accumulation (98M tokens) 1.928 −0.11
Muon, batch 64, Muon lr 0.04 (98M tokens) 1.990 −0.05
Plain SGD on a 3.7M model, the run whose gradient history is used below 2.609 –

The two runs at 98M tokens saw twice the data, so they are not a fair line-up with the control.

Gradient direction over time

Tensor Mean cosine between consecutive gradients
Final layer norm about 0.7
Weight matrices about 0.04

Loss weighting

Probe Result
bands (copies trained on one difficulty band each) Segregating tokens by difficulty hurt. The model wants the whole sequence.
temps (soft focus on high-loss tokens) Weighting by raw loss lost, and lost more the sharper the focus.

Findings

  • On weight matrices, a gradient’s direction barely carries over to the next step (cosine about 0.04), while the final norm layer is steady (about 0.7). This led to AdaMuon in optim.py, a Muon variant that sets each tensor’s momentum from how well its new gradient lines up with the old one.
  • The update rule matters more than most tweaks. Muon beat AdamW by 0.26 bits per byte at the same budget, and plain SGD was 0.6 behind.
  • Pointing the gradient at hard tokens did not help. High loss mixes two things: tokens the model could still learn and tokens that are just unpredictable. Weighting by loss spends effort on the second kind. Disagreement between differently seeded models was proposed as the better signal (disw).
  • Wiping the optimizer’s memory every 976 steps cost almost nothing (2.042 vs 2.037).

Next

On the afternoon of June 9 the work moved on to learned update rules (see Learning the update rule).