Research – Biology and chemistry

Filling in gaps in bacterial genomes

Small transformers trained on 20 complete E. coli genomes, learning to fill in DNA letters that have been hidden.

StartedNov 16, 2026
Last updateNov 16, 2026

Question

A bacterial genome is about 5 million letters, far too long to read in one pass the way a virus genome can be. Can a small model that sees only a short window still learn enough about E. coli to fill in hidden stretches of its DNA? Training on many strains of one species is meant to show which parts are shared and which vary.

Approach

Complete genomes were pulled from NCBI RefSeq, the public reference collection. The RefSeq list holds 5,593 complete E. coli genomes; the first 20 by accession number were taken, starting with the lab strain K-12 MG1655.

The model is masked: it sees the window in both directions, some letters are replaced by a “hidden” token, and only those are scored. It is the same code used for the SARS-CoV-2 work, pointed at bacteria. The three models compare depth (4 against 8 layers) and window length.

Data 20 complete E. coli genomes, about 102 million bases in all (4.6 to 5.8 million each, including plasmids)
Vocabulary A, C, G, T, unknown, a separator between genomes, and the hidden token
Model 1 4 attention layers, width 128, about 0.79M parameters
Model 2 8 attention layers, width 128, about 1.59M parameters
Model 3 same 8-layer model, a longer window
Score bits per hidden base (how surprised the model is by each hidden letter; 2.0 is a blind guess, lower is better)

Next

  • Score each model on held-out DNA and compare the 4-layer and 8-layer versions, and the short and long windows.
  • Hold out whole strains, so the test is a genome the model has never seen any part of.
  • Use the fill tool from the SARS-CoV-2 work: hide a gene, fill it back, and see where the model is sure and where several letters fit.