Research – Foundations

Learning without backprop

A week of spiking networks trained to predict FineWeb text with local, brain-style learning rules, compared against the same networks trained with ordinary gradients.

StartedJun 2, 2026
Last updateJun 6, 2026
bits per character, the spiking baseline the later variants were measured against2.8

Question

Backpropagation needs every weight to know about an error far downstream. Brains don’t seem to work that way. Can a network learn to predict text when each connection only sees its own two ends, or only a reward signal, and how far behind gradient training does that leave it?

Approach

Most models here are spiking networks: neurons that build up charge and fire a 0/1 spike when it crosses a threshold (leaky integrate-and-fire). The contrastive and predictive-coding learners use simpler smooth units. Characters of FineWeb text go in, and output “motor” neurons that collect energy give the next-character guess. Around 18 scripts, written June 2 to 6, 2026.

Thread Scripts What it tries
Spiking reservoir spike, trace, reservoir, fineweb A fixed, randomly wired recurrent pool holds a fading trace of recent text. Only the readout learns, by a reward signal plus “eligibility traces” (a short memory of which connections were just active). spike and trace are visual tools for watching one character’s energy ripple through the network.
Retrieval bank retrieval The reservoir’s state probes a second bank of neurons. The best match fires and returns its stored next-character counts. Keys and counts are learned online and locally.
Training the recurrence train_rec Three ways to train the reservoir’s own wiring: readout only, a local rule driven by the readout error, and SPSA (nudge every weight at random, replay, keep what helped).
Gradient reference surrogate, unified The same spiking nets trained end to end by backprop through time, with a smooth stand-in for the spike on the backward pass. The comparison point for everything else.
Contrastive / equilibrium learning chl Run each character twice: once free, once with the output nudged toward the right answer. Each connection changes by the difference in its two ends’ joint activity. Fully local.
Predictive coding pc The network predicts the next character, measures its own surprise, and feeds that error back into the hidden state. Each weight learns from error times activity at its own location.
Evolution strategies es No gradients at all. Spray 40 random perturbations around the current weights, score each on a fresh slice of text, move toward the ones that did best.
Input encoding ear, bitear, ear2 Characters enter as the bits of their Unicode code point instead of one neuron per character, so “a” and “A” already share structure. ear2 adds a fading delay line holding the last 12 characters in order.
Wiring as geometry unigrid, branch Neurons on a sheet, connected by distance instead of a weight table. In branch, each neuron has a position and 8 learnable “tendrils”; a connection exists wherever a tendril tip lands near another neuron.
Probes onenode, trace_act Train a single node out of 100 and see how far it can move predictions. Record spike rasters on text the network learned versus text it never saw.

Results

Model Result
ear.py (bit-ear input, 1,024 spiking neurons, surrogate gradients, Adam) About 2.8 bits per character, the baseline ear2 was built to be compared against
Fixed input projection Training the input wiring was unnecessary; the unified fabric keeps it frozen

Bits per character is how surprised the model is by each next character; lower is better.

Findings

  • Gradient training through the spikes (with a smooth stand-in for the spike on the way back) became the reference every local rule was measured against.
  • A frozen random input layer was enough. Later models kept it fixed and write-protected and only trained the rest.
  • Feeding characters as code-point bits gives related characters shared structure for free, and ear.py, ear2.py, es.py and branch.py all adopted it after bitear.
  • Evolution strategies, contrastive learning and predictive coding all avoid backprop, but each pays for it: ES needs dozens of forward passes per step, and the two-phase contrastive rule doubles the work per character.
  • By June 6 the work moved from learning rules to wiring and geometry (unigrid, branch), and then to the dial and oscillator models in “Neurons as spinning dials”.

Next

The question of what a single neuron should be carried forward into the oscillator models.