M5: A Whole Genome Bacterial Encoder at Single Nucleotide Resolution
A linear attention mechanism is described to extend the context length of an encoder only transformer, called M5 in this report, to a multi-million single nucleotide resolution foundation model pretrained on bacterial whole genomes. The linear attention mechanism used approximates a full quadratic attention mechanism tightly and has a simple and lightweight implementation for the use case when the key-query embedding dimensionality is low. The M5-small model is entirely trained and tested on one A100 GPU with 40gb of memory up to 196K nucleotides during training and 2M nucleotides during testing. We test the performance of the M5-small model and record notable improvements in performance as whole genome bacterial sequence lengths are increased as well as demonstrating the stability of the full multi-head attention approximation used as sequence length is increased.
Code (0)
등록된 구현이 없습니다.
Tasks
GPUMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Whole genome single nucleotide polymorphism genotyping of Staphylococcus aureus
Next-generation sequencing technology enables routine detection of bacterial pathogens for clinical diagnostics and genetic research. Whole genome sequencing has been of importance in the epidemiologic analysis of bacter…
Revolutionising Bacterial Genomics: Graph-Based Strategies for Improved Variant Identification
A significant advancement in bioinformatics is using genome graph techniques to improve variation discovery across organisms. Traditional approaches, such as bwa mem, rely on linear reference genomes for genomic analyses…
DiversityRevealing the Shape of Genome Space via K-mer Topology
Despite decades of effort, understanding the shape of genome space in biology remains a challenge due to the similarity, variability, diversity, and plasticity of evolutionary relationships among species, genes, or other…
DiversityWhole-Genome Phenotype Prediction with Machine Learning: Open Problems in Bacterial Genomics
How can we identify causal genetic mechanisms that govern bacterial traits? Initial efforts entrusting machine learning models to handle the task of predicting phenotype from genotype return high accuracy scores. However…
Decision MakingOrthologs from maxmer sequence context
Context-dependent identification of orthologs customarily relies on conserved gene order or whole-genome sequence alignment. It is shown here that short-range context--as short as single maximal matches--also provides an…