Filling in gaps in bacterial genomes
Small transformers trained on 20 complete E. coli genomes, learning to fill in DNA letters that have been hidden.
Question
A bacterial genome is about 5 million letters, far too long to read in one pass the way a virus genome can be. Can a small model that sees only a short window still learn enough about E. coli to fill in hidden stretches of its DNA? Training on many strains of one species is meant to show which parts are shared and which vary.
Approach
Complete genomes were pulled from NCBI RefSeq, the public reference collection. The RefSeq list holds 5,593 complete E. coli genomes; the first 20 by accession number were taken, starting with the lab strain K-12 MG1655.
The model is masked: it sees the window in both directions, some letters are replaced by a “hidden” token, and only those are scored. It is the same code used for the SARS-CoV-2 work, pointed at bacteria. The three models compare depth (4 against 8 layers) and window length.
| Data | 20 complete E. coli genomes, about 102 million bases in all (4.6 to 5.8 million each, including plasmids) |
| Vocabulary | A, C, G, T, unknown, a separator between genomes, and the hidden token |
| Model 1 | 4 attention layers, width 128, about 0.79M parameters |
| Model 2 | 8 attention layers, width 128, about 1.59M parameters |
| Model 3 | same 8-layer model, a longer window |
| Score | bits per hidden base (how surprised the model is by each hidden letter; 2.0 is a blind guess, lower is better) |
Next
- Score each model on held-out DNA and compare the 4-layer and 8-layer versions, and the short and long windows.
- Hold out whole strains, so the test is a genome the model has never seen any part of.
- Use the fill tool from the SARS-CoV-2 work: hide a gene, fill it back, and see where the model is sure and where several letters fit.