TLDR: seqme is a new open-source Python library that provides a comprehensive and modular framework for evaluating computational methods in biological sequence design. It offers sequence-based, embedding-based, and property-based metrics, along with various embedding and property models, to assess the fidelity, diversity, and desired properties of designed biological sequences (small molecules, DNA, RNA, peptides, proteins). The library also supports iterative design evaluation, visualization, and is highly extensible, aiming to standardize benchmarking and accelerate drug discovery.
Recent advancements in computational methods for designing biological sequences have led to a critical need for robust evaluation metrics. Until now, a unified software library to implement these metrics was missing. Addressing this gap, researchers have introduced seqme, an open-source Python library designed for evaluating computational methods in biological sequence design.
What is seqme?
seqme is a modular and highly extensible Python library that provides a comprehensive suite of model-agnostic metrics. It’s built to assess the performance of methods that design biological sequences, focusing on how well the designed sequences match a target distribution and achieve desired properties. The library is versatile, supporting a wide array of biological sequences including small molecules, DNA, ncRNA, mRNA, peptides, and proteins.
Key Features and Metrics
The library categorizes its metrics into three main groups:
- Sequence-based metrics: These operate directly on the raw sequences. They include measures like Novelty (fraction of sequences not in the reference set), Uniqueness (fraction of unique sequences), and Diversity (average pairwise dissimilarity). These are crucial for identifying common failure modes such as overfitting.
- Embedding-based metrics: These metrics utilize learned vector representations (embeddings) of sequences. Examples include Fréchet Biological Distance (FBD) and Maximum Mean Discrepancy (MMD), which evaluate distributional similarity between designed and reference sequences. It also offers reference-free diversity metrics like Vendi score and methods to detect memorization, such as Authenticity.
- Property-based metrics: These assess sequences based on their derived properties. Metrics like Identity (mean and standard deviation of a property), Threshold (fraction of sequences exceeding a property threshold), Hit-rate (fraction satisfying conditions), and Hypervolume (for multi-objective optimization) are included.
seqme also provides a variety of embedding and property models. For instance, it includes ESM-2 for protein embeddings, RNA-FM for RNA, GENA-LM for DNA, and Hyformer for peptides and small molecules. For property prediction, it can compute physico-chemical properties, predict antimicrobial activity, and even integrate models like ESMFold for 3D protein structure prediction.
Beyond Basic Evaluation
The library offers additional functionalities to enhance evaluation:
- Fold functionality: This addresses biases that can arise from discrepancies in the number of designed and reference sequences. It splits sequences into groups and computes metrics multiple times, providing more stable estimates and standard deviations.
- Support for iterative designs: Beyond one-shot evaluations, seqme can assess design quality across training epochs or iterations of generative models, making it suitable for iterative sequence discovery pipelines.
- Visualizations: To make performance interpretation easier, seqme includes functions for visualizing metric results through customizable tables, parallel-coordinates charts, barplots, and trajectory plots for iterative designs.
Implementation and Availability
Implemented in Python (v3.10 or greater), seqme is designed for extensibility and ease of use. It features performance optimizations like caching sequence representations, which significantly speeds up computations, especially with large language models. The library is open-source and available on GitHub, with extensive documentation and tutorials to guide users. It can be installed via pip. More details can be found in the research paper.
Also Read:
- NABench: A New Standard for Evaluating Nucleotide Foundation Models
- Generative AI’s Expanding Influence in Biological Research
Impact and Future Vision
The creators of seqme envision it accelerating de novo drug discovery by providing a robust foundation for model benchmarking and offering comprehensive tools for defining stopping criteria in machine-learning training loops. By unifying evaluation efforts, seqme aims to facilitate the translation of computational advances into practical applications in biology and medicine.
