Reading whole SARS-CoV-2 genomes
A 2M-parameter transformer that reads entire coronavirus genomes one DNA letter at a time, trained on older samples and tested on variants collected later.
Question
Can a very small model learn a whole virus genome, all 29,903 letters at once, well enough to predict variants it has never seen? The test is causal: train only on genomes collected up to a date, then score genomes collected after it.
Approach
About 9.4 million aligned genomes were streamed from Nextstrain’s public SARS-CoV-2 set. One in 450 was kept (chosen by a hash of its name), genomes with more than 1% unknown letters were dropped, and only those with a full collection date stayed. Each genome is one training example, read left to right, predicting the next letter.
| Data | 11,937 aligned genomes, 29,903 bases each, collected 2020-02-19 to 2026-10-07 |
| Unknown letters | 0.52% of all bases, never scored |
| Split | oldest 10,744 for training (up to 2022-11-17), newest 1,193 held out |
| Model | 4 attention layers, width 208, 4 heads, 2.09M parameters |
| Context | the whole genome in one pass (29,902 positions) |
| Optimizer | Muon on the block matrices, AdamW on the rest |
| Training | 1,500 steps, 24 minutes on one RTX 4090 |
Results
Bits per base measures how surprised the model is by each letter. 2.0 is a blind guess between four letters; lower is better.
| Train (up to Nov 2022) | Held out (after Nov 2022) | |
|---|---|---|
| Accuracy | 99.9% | 99.8% |
| Bits per base | 0.004 | 0.020 |
| Bits per whole genome | about 120 | about 600 |
The learning curve has a sharp shape:
| Step | Held-out accuracy | Held-out bits per base |
|---|---|---|
| 25 | 34.5% | 1.952 |
| 50 | 34.8% | 1.933 |
| 75 | 38.9% | 1.868 |
| 100 | 52.1% | 1.563 |
| 125 | 75.9% | 0.835 |
| 150 | 96.2% | 0.171 |
| 175 | 98.9% | 0.071 |
| 500 | 99.6% | 0.044 |
| 1,000 | 99.8% | 0.024 |
| 1,500 | 99.8% | 0.020 |
Findings
- The model sat near 34% accuracy for the first 50 or so steps, then shot up to 98.8% by step 175. That drop is the model memorising the shared genome. Everything after it is slow work on the mutations.
- Most of the genome is the same in every sample, so high per-letter accuracy alone says little. The useful number is the gap: about 600 bits of surprise per held-out genome against about 120 for training genomes.
- Held-out error kept falling for the full run (0.044 at step 500, 0.020 at step 1,500), so the later variants were not simply out of reach.
- The date split is weaker than it looks. Variants that dominated in 2023 were already circulating before the November 2022 cutoff, so some “future” genomes resemble training ones.
Next
- Train the masked version: the model sees the whole genome in both directions, some stretches are hidden, and it fills them back in. It will warm-start from this checkpoint.
- Fill in a gene: hide a whole region such as the spike gene (bases 21,563–25,384), let the model fill it back, and see which letters it is sure about and where several letters fit.
- Swap experiment: paste a region from one genome into another, re-fill a different region, and see which positions elsewhere in the genome change. It works on synthetic data but has not been run on the real model.
- Compare against a simple baseline: always predict the most common letter at each position.
- Score the held-out genomes by collection year, and map error along the genome. The expectation is a pile-up in the spike gene.