Neurons as spinning dials
Language models where each neuron is a dial that turns, and neurons pull on each other like coupled clocks instead of passing numbers through weight matrices. Trained on raw FineWeb bytes, at 15k to 62k parameters.
Question
Can memory live in angles instead of numbers? A dial that only rotates never fades, so a line of dials could hold context without decaying. The question is whether neurons built this way can learn to predict text at all, and how close they get to ordinary networks of the same size.
Approach
A family of small models, each named “something-line”, all trained to predict the next character or byte of FineWeb text.
| Model | Dates | Idea |
|---|---|---|
| spinline | Jun 6 | A line of dials, each with its own clock speed. Every character turns every dial by a learned amount. Memory is the accumulated angle. |
| coupline | Jun 7 – Nov 1 | A memory line plus a “char neuron”: a fresh set of dials born from the current character. The two pull on each other’s angles for a few rounds (Kuramoto coupling, where each dial drifts toward or away from the others), then the char neuron’s pattern is matched to the nearest prototype to pick the next character. |
| scanline, coupgrid, coupens | Jun 7 | Variants: a read head that walks the line dial by dial; several lines side by side with neighbour-only links; an ensemble where each line makes its own prediction. |
| circline, fastline, vecline | Jun 10 | A detour to rings of ordinary neurons with a “notepad” memory. In fastline and vecline the notepad is a matrix written during the run, so cross-dimension wiring comes from the data, not from training. |
| keepline | Oct 24 | coupline, but the char neuron is carried from one character to the next instead of born fresh. |
| gructrl | Oct 24 | A plain GRU (a standard recurrent network) sized to match any coupline config by parameter count. The control. |
| sphereline | Oct 25 | Dials become needles on a sphere (3D unit vectors) instead of on a circle, so a pull has a direction, not just an amount. Adds raw-byte input, an 8-bit readout, and stacked layers. |
| attnline | Nov 14–15 | sphereline plus switches borrowed from attention and from quantum computing: selective reads, selective writes, a key–value store, slow layers, sparse writes, rigidly carried “twin” dials, and “entangled” lines. With every switch off it is sphereline exactly. |
The sphere runs use 40.7M bytes of FineWeb, predicted byte by byte (256 possible values). Windows are 1,024 bytes. Early sphere runs were on an RTX 5090 or similar. The attnline sweeps ran on a 48 GB A6000, 3,000 steps each at batch 64, about 1.5 to 2.2 hours per run.
Bits per byte is how surprised the model is by each next byte; lower is better. Uniform guessing over 256 values is 8.
Results
Sphere dials on raw bytes (validation, 20-batch final eval)
| Run | Params | Setup | Steps | Bits/byte |
|---|---|---|---|---|
| Byte table input | 55,520 | 16 line dials, 64 char dials, 3 rounds | 50,000 | 2.592 |
| Bit readout | 15,584 | Same sizes, byte read in and out as 8 bits, ~16x smaller tables | 30,000 (stopped) | 2.754 (last val) |
| Batch ramp 64 to 256 | 52,224 | 48 line dials, 64 char dials, 5 rounds, readout neurons | 10,000 | 2.411 |
attnline switches (same 52k base, same batches, 3,000 steps)
| Variant | Params | Bits/byte | vs base |
|---|---|---|---|
| Base (all switches off) | 52,224 | 2.473 | – |
| Wider line, 52 dials (param control) | 55,824 | 2.459 | −0.014 |
| Selective read | 52,228 | 2.463 | −0.010 |
| Selective write | 56,448 | 2.462 | −0.011 |
| Key–value slots | 56,321 | 2.470 | −0.003 |
| Twin dials (carried, never pulled) | 52,804 | 2.473 | 0.000 |
| Entangled lines (no new params) | 52,224 | 2.441 | −0.032 |
| 4D dials | 54,208 | 2.482 | +0.009 |
| 6D dials | 58,176 | 2.529 | +0.056 |
| 8D dials | 62,144 | 2.560 | +0.087 |
| Slow layers (layer l ticks every K^l bytes), K = 2 / 4 / 8 | 52,224 | 2.730 / 2.774 / 2.783 | +0.26 to +0.31 |
| Sparse writes, 1 tick in 4 / 8 | 52,224 | 3.044 / 3.215 | +0.57 / +0.74 |
| Slow 4 + sparse 8 | 52,224 | 3.282 | +0.81 |
Checks on trained models (load weights, evaluate only)
| Check | Base | Entangled |
|---|---|---|
| No change | 2.505 | 2.477 |
| Memory wiped every 256 bytes | 2.521 | 2.494 |
| Memory wiped every 64 bytes | 2.586 | 2.563 |
| Entanglement switched off | – | 4.256 |
| Entangled dial pairs shuffled | – | 4.875 |
These checks use a different eval draw than the training-end numbers above, so compare within the table only.
Where in the window the bits go (base model): 2.545 for bytes 0–128, 2.491 for 128–512, 2.481 for 512–1,024.
Findings
- Dials can learn text. A 52k-parameter sphere model reaches 2.41 bits per byte on raw bytes, and it does use context: wiping its memory every 64 bytes costs about 0.08 bits per byte, and later bytes in a window are predicted better than early ones.
- Entangled lines were the only switch to clearly beat the param-matched control (2.441 vs 2.459 for a wider line), with no extra parameters. The trained model leans on the links hard: cutting them or shuffling the pairing sends it to 4.3 and 4.9.
- The attention-style switches (selective read, selective write, slots) each moved the score by about 0.01 at most, no better than just adding dials. Twin dials changed nothing.
- More dimensions per dial made things worse: 3D beat 4D, 6D and 8D. A 2D (circle) run in the same sweep went to NaN.
- Forcing structure in time hurt badly. Slow upper layers cost about 0.3 bits per byte and sparse writes cost 0.6 to 0.8. Every dial needs to be writable on every byte at this size.
- In quantum-computing terms, a coupline dial is a qubit stuck on the equator: phase only, with the readout acting like an overlap between two states. The pulling that settles the dials has no quantum counterpart, since quantum gates only rotate and never settle.
Next
- A precession switch (a turn around the pull direction, the way two real spins act on each other) ran out of GPU memory and needs a smaller run.
- Run to the end: a protected 16-dial register, three-twin frames, entanglement passing downward, and a learned entanglement strength.
- Run the GRU control and a small transformer on the same bytes, at the same parameter count, so the dial models have a fair baseline.