New architectures, small and fast
About forty small language-model designs tried against a plain transformer between May and November. Routing tokens through shared units, learned tokenizers, vocabularies placed on a circle, memory written into a matrix while reading, and attention swapped for cheaper parts.
Question
The transformer is one design among many possible ones. Can a different layout of the same small parts (routing, memory, positions, the output head) predict text better at the same size, or do the standard pieces each earn their place?
Approach
Each idea was built as a small standalone script and trained on FineWeb text, as characters, raw bytes, or a 1,024-piece token vocabulary. They ran on an RTX 5090 or similar. The controls were a plain transformer (tinylm, nano, and from October baseline.py, “the number every routing experiment has to beat”) and a small MLP language model (mlplm, which also compared backprop against predictive coding).
| Theme | Dates | Scripts | Idea |
|---|---|---|---|
| Ablating the plain transformer | May 24–25 | train.py variants | Remove or reorder parts of a 1-layer transformer: no MLP, attention only, MLP before attention, multiscale position codes, very narrow widths. |
| Routing through shared units | Jun 17, Aug 25 | pyramidlm, diamondlm, routepool, inspect_routes, skipgate | A tree-shaped output head (predict a coarse group, then the token). A diamond of blocks with a gate at each rank. A pool of 8 blocks where each token picks its own path of 3 (336 possible paths). Blocks that each token may skip. |
| Tokens, positions, vocabulary | Jun 12–15, Aug 20–24 | purechar, boundary, tokboundary, multipos, twostream, angle, angle_nd, slots, slots_seq | A learned boundary loop that chops characters into chunks, replacing the tokenizer. Several positions or several embeddings per token. Every token as a single angle on a circle, or a point on a torus, read through noise. A learned split of the vocabulary into groups, predicting group then token. |
| Memory written while reading | Jun 10–11, Aug 3–16, Aug 28–Sep 9, Oct 18–23 | fastweight_toy, fastweight_lm, bmat, bmat2, hyperchain, growmat, state2d, wrap, cube, medium, stuff, bit_ref, livetree | The context lives in a matrix (or cube, or table of cells) that each token writes into and reads from, instead of a growing cache. livetree has no gradients at all: the model is a character tree that deepens where paths repeat. stuff measures how many random bytes fit in one matrix. |
| Cheaper stand-ins for attention | Jul 9, Aug 26–28, Sep 12, Oct 15–18 | scan0, scan1, bare, rot, split, correct, plan, speed, reads, auxhead, deep, freerun | A running angle pushed by each byte. Concatenate ten tokens and use an MLP. One 8×8 matrix reused through its four rotations. A head where each hidden number owns 32 output tokens. Attention replaced by a per-head correction matrix that mixes each new token into the previous output (one small matmul per token instead of a pass over the cache). Per-layer losses, an auxiliary head that shapes the target, free-running training on the model’s own output. |
| Attention-style switches in the dial model | Nov 14–15 | attnline, runs/attn_sweep* | Selective reads, selective writes, slots and “entangled” lines inside a non-transformer model of spinning dials. Covered in full under Neurons as spinning dials. |
Scores are in bits per byte: how surprised the model is by each byte of text, lower is better.
Results
Plain transformer ablations (1 layer, 256 wide, 3,000 steps, 49M tokens, Muon, May 24–25)
| Variant | Params | Bits/byte | vs control |
|---|---|---|---|
| Plain 1-layer transformer, the control | 1.18M | 2.037 | – |
| Multiscale positions, 8 scales | 1.08M | 2.023 | −0.014 |
| Multiscale positions at 512, 64, 16, 4 | 1.09M | 2.028 | −0.009 |
| MLP before attention | 1.18M | 2.173 | +0.14 |
| No MLP | 0.66M | 2.219 | +0.18 |
| Minimal attention-only | 0.52M | 2.606 | +0.57 |
| 3 layers, attention only, no layer norm | 1.05M | 2.688 | +0.65 |
| Width 16 | 0.03M | 3.063 | +1.03 |
| Width 4 | 0.006M | 3.577 | +1.54 |
The ablation runs used gradient clipping at 1.0.
Other results
| Experiment | Result | Compared against |
|---|---|---|
| stuff: 128 random bytes (1,024 bits) folded into one 64×64 matrix, then read back | 66.6% of bytes recovered at step 100,000 | Chance is 0.4% (1 in 256). |
| Dial model with entangled lines (attn_sweep3) | 2.441 bits/byte | A param-matched wider dial line at 2.459. The other attention-style switches moved the score by 0.01 or less. |
Findings
- At one layer every standard part earns its place. Dropping the MLP cost 0.18 bits per byte, moving it before attention cost 0.14, and stripping down to attention alone cost 0.57 or more.
- The one matched win was positional. Encoding position at several scales beat a learned position table by about 0.01 bits per byte with 9% fewer parameters. It is a single run each, so the gap is small evidence.
- Several new designs turned into known ones once made to work. The fast-weight chain becomes linear attention when its writes carry token-dependent values. The 8×8 rotation block does nothing without a nonlinearity around it, since a linear chain absorbs it. With a fixed query, the “wrap” matrix’s outer part only adds a constant.
- A single small matrix holds a lot. 4,096 numbers stored 128 random bytes well enough to read back two thirds of them, with only the packing to learn.
Next
- Run baseline.py and the routing models (routepool, diamondlm, skipgate) at the same size and data, and rank them against the control.
- Measure how many random bytes one 64×64 matrix can hold as the byte count grows.
- Time the correction-matrix attention against standard attention at long context.