Research – Biology and chemistry

Reading whole SARS-CoV-2 genomes

A 2M-parameter transformer that reads entire coronavirus genomes one DNA letter at a time, trained on older samples and tested on variants collected later.

StartedNov 10, 2026
Last updateNov 10, 2026
of letters predicted correctly in genomes collected after the training cutoff99.8%

Question

Can a very small model learn a whole virus genome, all 29,903 letters at once, well enough to predict variants it has never seen? The test is causal: train only on genomes collected up to a date, then score genomes collected after it.

Approach

About 9.4 million aligned genomes were streamed from Nextstrain’s public SARS-CoV-2 set. One in 450 was kept (chosen by a hash of its name), genomes with more than 1% unknown letters were dropped, and only those with a full collection date stayed. Each genome is one training example, read left to right, predicting the next letter.

Data 11,937 aligned genomes, 29,903 bases each, collected 2020-02-19 to 2026-10-07
Unknown letters 0.52% of all bases, never scored
Split oldest 10,744 for training (up to 2022-11-17), newest 1,193 held out
Model 4 attention layers, width 208, 4 heads, 2.09M parameters
Context the whole genome in one pass (29,902 positions)
Optimizer Muon on the block matrices, AdamW on the rest
Training 1,500 steps, 24 minutes on one RTX 4090

Results

Bits per base measures how surprised the model is by each letter. 2.0 is a blind guess between four letters; lower is better.

Train (up to Nov 2022) Held out (after Nov 2022)
Accuracy 99.9% 99.8%
Bits per base 0.004 0.020
Bits per whole genome about 120 about 600

The learning curve has a sharp shape:

Step Held-out accuracy Held-out bits per base
25 34.5% 1.952
50 34.8% 1.933
75 38.9% 1.868
100 52.1% 1.563
125 75.9% 0.835
150 96.2% 0.171
175 98.9% 0.071
500 99.6% 0.044
1,000 99.8% 0.024
1,500 99.8% 0.020

Findings

  • The model sat near 34% accuracy for the first 50 or so steps, then shot up to 98.8% by step 175. That drop is the model memorising the shared genome. Everything after it is slow work on the mutations.
  • Most of the genome is the same in every sample, so high per-letter accuracy alone says little. The useful number is the gap: about 600 bits of surprise per held-out genome against about 120 for training genomes.
  • Held-out error kept falling for the full run (0.044 at step 500, 0.020 at step 1,500), so the later variants were not simply out of reach.
  • The date split is weaker than it looks. Variants that dominated in 2023 were already circulating before the November 2022 cutoff, so some “future” genomes resemble training ones.

Next

  • Train the masked version: the model sees the whole genome in both directions, some stretches are hidden, and it fills them back in. It will warm-start from this checkpoint.
  • Fill in a gene: hide a whole region such as the spike gene (bases 21,563–25,384), let the model fill it back, and see which letters it is sure about and where several letters fit.
  • Swap experiment: paste a region from one genome into another, re-fill a different region, and see which positions elsewhere in the genome change. It works on synthetic data but has not been run on the real model.
  • Compare against a simple baseline: always predict the most common letter at each position.
  • Score the held-out genomes by collection year, and map error along the genome. The expectation is a pile-up in the spike gene.