TLDR: NABench is a large-scale, systematic benchmark introduced to standardize the evaluation of Nucleotide Foundation Models (NFMs) for fitness prediction. It aggregates 2.6 million mutated sequences from 162 high-throughput assays across diverse DNA and RNA families, including DMS and SELEX experiments. The benchmark rigorously assesses 29 NFMs across zero-shot, few-shot, transfer learning, and supervised settings, revealing performance heterogeneity across tasks and nucleic-acid types. NABench provides crucial insights into model strengths and weaknesses, establishing reproducible baselines to advance nucleic acid modeling for applications in RNA/DNA design and synthetic biology.
Understanding how changes in DNA and RNA sequences affect their function, or ‘fitness,’ is a fundamental challenge in biology. This knowledge is crucial for everything from identifying disease-causing genetic variations to designing new biological tools and therapies. Recently, advanced computer models, known as Nucleotide Foundation Models (NFMs), have emerged with the promise of directly predicting these fitness effects from sequence data alone.
However, comparing and evaluating these powerful new models has been a significant hurdle. Researchers often use different datasets and processing methods, making it difficult to truly understand which models perform best and why. This inconsistency has slowed down progress in developing and applying NFMs.
To address this critical need, a new research initiative introduces NABench, a comprehensive and large-scale benchmark specifically designed for nucleic acid fitness prediction. NABench aims to provide a standardized platform for evaluating NFMs, ensuring fair and robust comparisons across various DNA and RNA families.
NABench is impressive in its scale and diversity. It brings together data from 162 high-throughput experiments, compiling a massive collection of 2.6 million mutated sequences. These sequences span a wide array of nucleic acid types, including messenger RNA (mRNA), transfer RNA (tRNA), ribozymes, aptamers, and DNA elements like enhancers, promoters, and exons. The benchmark includes data from two primary experimental techniques: Deep Mutational Scanning (DMS), which studies the effects of small mutations on known sequences, and Systematic Evolution of Ligands by Exponential Enrichment (SELEX), which explores the functionality of randomly synthesized sequences.
Beyond just collecting data, NABench standardizes the way this data is split for training and testing, and provides rich metadata to ensure high quality. It also offers a unified evaluation suite to rigorously test 29 different foundation models. These models represent diverse computational architectures, such as BERT, GPT, and Hyena, and are assessed across four key evaluation settings: zero-shot prediction (predicting without any prior task-specific training), few-shot prediction (training with very limited labeled data), transfer learning (applying knowledge from one task to another), and supervised learning (training with ample labeled data).
The initial findings from NABench reveal that no single model or architectural style consistently outperforms all others across every scenario. For instance, state-space models, particularly those from the Evo family, showed a clear advantage in zero-shot prediction, demonstrating strong intrinsic knowledge without specific training. However, when labeled data was introduced, BERT-like models often showed remarkable adaptability and improved significantly in supervised and few-shot settings.
The research also highlighted important trade-offs between model performance and computational efficiency. While some state-of-the-art models achieved slightly better results, they sometimes required a substantially larger number of parameters, making them computationally intensive. Another key insight was the challenge models faced in generalizing to synthetic SELEX sequences, suggesting that current genomic foundation models are not yet fully equipped to handle completely novel, randomly generated sequences without specific prior information.
Also Read:
- Generative AI’s Expanding Influence in Biological Research
- STAR-VAE: Advancing Molecular Design with Latent Variable Transformers
In conclusion, NABench provides an invaluable resource for the scientific community. By offering a standardized and extensive benchmark, it enables researchers to accurately assess the capabilities and limitations of different nucleotide foundation models. This work is expected to accelerate advancements in rational DNA/RNA design, property prediction, and engineering optimization, ultimately supporting critical applications in synthetic biology and biochemistry. The code for NABench is openly available for researchers to use and contribute to, fostering collaborative progress in the field. You can find more details about this research paper here.
