Research
Every project, with its results and what it taught. Negative results included.
Biology and chemistry
Models that read DNA, a cell simulated from physics and chemistry, and later, many cells together.
- Learning to run a water simulationA transformer that learns to advance a molecular simulation of 512 water molecules, one step at a time, and is checked against the real physics engine it learned from.0.6 pson the true path before the model drifts 0.1 nm away
- Reading whole SARS-CoV-2 genomesA 2M-parameter transformer that reads entire coronavirus genomes one DNA letter at a time, trained on older samples and tested on variants collected later.99.8%of letters predicted correctly in genomes collected after the training cutoff
- Filling in gaps in bacterial genomesSmall transformers trained on 20 complete E. coli genomes, learning to fill in DNA letters that have been hidden.
- A map of everything inside a human cellA catalog of every kind of thing in a human cell, each one written as a JSON body of physics and chemistry. It is the data a cell simulator will be built from.195kinds of thing catalogued, from water to viruses
- A 3D simulator of living cellsA browser simulator where cells take in glucose, turn it into ATP and protein inside their own volume, grow, divide and starve. Version 0, built to grow.
Foundations
The machine learning work that came first, including what didn’t work.
- Giving a frozen LLM a memoryThe injection half of Project Salience, a large project that also covered prompt compression. Memory vectors added straight into the hidden states of a frozen Llama, with only small projector networks trained, scaled from 3B to 8B and tested on whether the model answers from memory it was never shown as text.+0.355F1 lift from injection on an 8B model, 147 held-out questions (0.43 with memory, 0.08 without)
- Squeezing a prompt into a few tokensThe compression half of Project Salience. Learned "soft tokens" (vectors fed to a frozen Llama in place of real words) were fitted to stand in for a whole prompt or conversation, down to 2 tokens, and then masked so only a handful of numbers per token could change.16 of 2,048numbers free per soft token, and the model still stays within KL 0.22 of reading the full prompt
- Neurons as spinning dialsLanguage models where each neuron is a dial that turns, and neurons pull on each other like coupled clocks instead of passing numbers through weight matrices. Trained on raw FineWeb bytes, at 15k to 62k parameters.2.41bits per byte on raw web text from a 52k-parameter dial model
- A 336M language model from scratchA 24-layer transformer trained from zero on 1.8B tokens of FineWeb-Edu on one rented GPU, then used to test whether new layers stacked on a frozen model can add anything.0.989bits per byte on held-out FineWeb-Edu after one pass over 1.8B tokens
- Learning without backpropA week of spiking networks trained to predict FineWeb text with local, brain-style learning rules, compared against the same networks trained with ordinary gradients.2.8bits per character, the spiking baseline the later variants were measured against
- New architectures, small and fastAbout forty small language-model designs tried against a plain transformer between May and November. Routing tokens through shared units, learned tokenizers, vocabularies placed on a circle, memory written into a matrix while reading, and attention swapped for cheaper parts.2.023bits per byte with multiscale positions, vs 2.037 for the plain 1-layer control, with 9% fewer parameters
- What one training step doesProbes of gradients and single updates in tiny FineWeb transformers. How steady a gradient's direction is, what happens when one step is repeated or reweighted, and where training forgets.0.04average cosine between one gradient and the next on weight matrices (about 0.7 on the final norm)
- Learning the update ruleAttempts to replace the optimizer with something learned, or to build a better update from a weight's own history. Small networks that propose updates, learned corrections to the gradient, and history-based fixes on a toy. The learned optimizers finished well behind Muon.2.89bits per byte for the best learned optimizer, vs 2.04 for Muon on the same model
- Parameter Golf, a 16 MB language modelThree days of experiments on OpenAI's Parameter Golf challenge, where the whole model must fit in 16 MB and train in 10 minutes. The best run scored well but came out too large, and nothing was submitted.1.170bits per byte on FineWeb val, but at 24.3 MB, over the 16 MB cap
- When a small model stops memorizingA sweep on 3-digit addition to see how the amount of training data, and whether each example shows its worked steps, moves the point where a small model starts getting unseen problems right. From 10k worked examples, a 4-layer model gets 99.5% of them.99.5%exact answers on 5,000 never-seen addition problems, from 10k worked examples
- Speech from scratchA tiny text-to-speech model trained on one voice, first predicting raw audio directly, then spectrograms through a pretrained vocoder. Alongside it, tests of open voice models and a trick to make one of them generate faster.
- Quantum circuits, from a Bell pair to next-byte predictionLearning how quantum computers work by building small circuits, a noisy-chip decay sweep, a plain-Python simulator, and an 8-qubit circuit trained to predict the next byte of text.5.38 bitsper byte from an 8-qubit circuit, against 3.82 for a plain count table