Research – Foundations

New architectures, small and fast

About forty small language-model designs tried against a plain transformer between May and November. Routing tokens through shared units, learned tokenizers, vocabularies placed on a circle, memory written into a matrix while reading, and attention swapped for cheaper parts.

StartedMay 24, 2026
Last updateNov 15, 2026
bits per byte with multiscale positions, vs 2.037 for the plain 1-layer control, with 9% fewer parameters2.023

Question

The transformer is one design among many possible ones. Can a different layout of the same small parts (routing, memory, positions, the output head) predict text better at the same size, or do the standard pieces each earn their place?

Approach

Each idea was built as a small standalone script and trained on FineWeb text, as characters, raw bytes, or a 1,024-piece token vocabulary. They ran on an RTX 5090 or similar. The controls were a plain transformer (tinylm, nano, and from October baseline.py, “the number every routing experiment has to beat”) and a small MLP language model (mlplm, which also compared backprop against predictive coding).

Theme Dates Scripts Idea
Ablating the plain transformer May 24–25 train.py variants Remove or reorder parts of a 1-layer transformer: no MLP, attention only, MLP before attention, multiscale position codes, very narrow widths.
Routing through shared units Jun 17, Aug 25 pyramidlm, diamondlm, routepool, inspect_routes, skipgate A tree-shaped output head (predict a coarse group, then the token). A diamond of blocks with a gate at each rank. A pool of 8 blocks where each token picks its own path of 3 (336 possible paths). Blocks that each token may skip.
Tokens, positions, vocabulary Jun 12–15, Aug 20–24 purechar, boundary, tokboundary, multipos, twostream, angle, angle_nd, slots, slots_seq A learned boundary loop that chops characters into chunks, replacing the tokenizer. Several positions or several embeddings per token. Every token as a single angle on a circle, or a point on a torus, read through noise. A learned split of the vocabulary into groups, predicting group then token.
Memory written while reading Jun 10–11, Aug 3–16, Aug 28–Sep 9, Oct 18–23 fastweight_toy, fastweight_lm, bmat, bmat2, hyperchain, growmat, state2d, wrap, cube, medium, stuff, bit_ref, livetree The context lives in a matrix (or cube, or table of cells) that each token writes into and reads from, instead of a growing cache. livetree has no gradients at all: the model is a character tree that deepens where paths repeat. stuff measures how many random bytes fit in one matrix.
Cheaper stand-ins for attention Jul 9, Aug 26–28, Sep 12, Oct 15–18 scan0, scan1, bare, rot, split, correct, plan, speed, reads, auxhead, deep, freerun A running angle pushed by each byte. Concatenate ten tokens and use an MLP. One 8×8 matrix reused through its four rotations. A head where each hidden number owns 32 output tokens. Attention replaced by a per-head correction matrix that mixes each new token into the previous output (one small matmul per token instead of a pass over the cache). Per-layer losses, an auxiliary head that shapes the target, free-running training on the model’s own output.
Attention-style switches in the dial model Nov 14–15 attnline, runs/attn_sweep* Selective reads, selective writes, slots and “entangled” lines inside a non-transformer model of spinning dials. Covered in full under Neurons as spinning dials.

Scores are in bits per byte: how surprised the model is by each byte of text, lower is better.

Results

Plain transformer ablations (1 layer, 256 wide, 3,000 steps, 49M tokens, Muon, May 24–25)

Variant Params Bits/byte vs control
Plain 1-layer transformer, the control 1.18M 2.037 –
Multiscale positions, 8 scales 1.08M 2.023 −0.014
Multiscale positions at 512, 64, 16, 4 1.09M 2.028 −0.009
MLP before attention 1.18M 2.173 +0.14
No MLP 0.66M 2.219 +0.18
Minimal attention-only 0.52M 2.606 +0.57
3 layers, attention only, no layer norm 1.05M 2.688 +0.65
Width 16 0.03M 3.063 +1.03
Width 4 0.006M 3.577 +1.54

The ablation runs used gradient clipping at 1.0.

Other results

Experiment Result Compared against
stuff: 128 random bytes (1,024 bits) folded into one 64×64 matrix, then read back 66.6% of bytes recovered at step 100,000 Chance is 0.4% (1 in 256).
Dial model with entangled lines (attn_sweep3) 2.441 bits/byte A param-matched wider dial line at 2.459. The other attention-style switches moved the score by 0.01 or less.

Findings

  • At one layer every standard part earns its place. Dropping the MLP cost 0.18 bits per byte, moving it before attention cost 0.14, and stripping down to attention alone cost 0.57 or more.
  • The one matched win was positional. Encoding position at several scales beat a learned position table by about 0.01 bits per byte with 9% fewer parameters. It is a single run each, so the gap is small evidence.
  • Several new designs turned into known ones once made to work. The fast-weight chain becomes linear attention when its writes carry token-dependent values. The 8×8 rotation block does nothing without a nonlinearity around it, since a linear chain absorbs it. With a fixed query, the “wrap” matrix’s outer part only adds a constant.
  • A single small matrix holds a lot. 4,096 numbers stored 128 random bytes well enough to read back two thirds of them, with only the packing to learn.

Next

  • Run baseline.py and the routing models (routepool, diamondlm, skipgate) at the same size and data, and rank them against the control.
  • Measure how many random bytes one 64×64 matrix can hold as the byte count grows.
  • Time the correction-matrix attention against standard attention at long context.