TLDR: PMKLC (Parallel Multi-Knowledge Learning-based Compressor) is a novel system designed for lossless compression of large-scale genomic databases. It addresses common challenges like inadequate compression ratios, low throughput, and poor robustness by introducing an automated multi-knowledge learning framework, a GPU-accelerated (s,k)-mer encoder, and parallel acceleration techniques like data block partitioning and Step-wise Model Passing. Benchmarking shows PMKLC-S (single GPU) and PMKLC-M (multi-GPU) achieve significant improvements in compression ratio (up to 73.6%), throughput (up to 1071%), robustness (up to 94.5%), and memory efficiency, offering a balanced and highly effective solution for genomic data management.
Managing the ever-growing volume of genomic data presents significant challenges for storage, sharing, and overall management. Traditional and existing learning-based compression methods often fall short, struggling with inadequate compression ratios, slow processing speeds (throughput), and a lack of robustness when dealing with diverse datasets. These limitations hinder their widespread adoption in both research and industry.
To address these critical issues, a new approach called the Parallel Multi-Knowledge Learning-based Compressor, or PMKLC, has been developed. This innovative system introduces several key designs aimed at achieving a balanced solution for genomic data compression, optimizing for compression ratio, throughput, robustness, and efficient resource consumption.
How PMKLC Works: A Closer Look at its Core Innovations
PMKLC is built upon four crucial designs that work in concert to deliver its superior performance:
- Automated Multi-Knowledge Learning-based Compression Framework (AMKLCF): This framework acts as the backbone of PMKLC, significantly enhancing both compression ratio and robustness. It integrates three types of neural network models: a Static Public Model (SPuM) that learns from diverse genomic data, a Static Private Model (SPrM) tailored to the specific dataset being compressed, and a Dynamic Model (DM) that self-learns and combines knowledge from all sources. An automated model selector intelligently chooses which models to use based on the dataset size, optimizing resource use.
- GPU-Accelerated (s,k)-mer Encoder (GskE): Genomic data often contains a high degree of redundancy. GskE is designed to efficiently extract this redundant information and reduce the data size. By leveraging GPU acceleration, it dramatically improves compression throughput and overall efficiency, while also reducing memory usage.
- Data Block Partitioning and Step-wise Model Passing (SMP): To achieve parallel acceleration, PMKLC divides large datasets into smaller blocks. The SMP mechanism is particularly clever for multi-GPU setups. It addresses the ‘cold-start’ problem, where new GPUs might start with untrained models, by passing a partially trained model from one GPU to the next. This ensures that all GPUs begin with a more informed model, further boosting efficiency and compression quality.
- Flexible Compression Modes: Recognizing varied application scenarios, PMKLC offers two modes: PMKLC-S, optimized for single-GPU environments, and PMKLC-M, designed for multi-GPU acceleration to achieve even higher throughput.
Also Read:
- Efficient Polar Decoding for Deletion Channels Using Neural Networks
- SToFM: A New Model for Understanding Spatial Transcriptomics Data Across Multiple Biological Scales
Benchmarking PMKLC: Impressive Results Across the Board
The effectiveness of PMKLC was rigorously tested against 14 other compression methods, including both traditional and learning-based approaches, using 15 real-world genomic datasets of varying species and sizes. The results highlight PMKLC’s significant advantages:
- Compression Ratio: PMKLC-S and PMKLC-M achieved average compression ratio improvements of up to 73.609% and 73.480% respectively, compared to baselines. This means it can significantly reduce the size of genomic data, making storage and transmission more efficient.
- Throughput: The system demonstrated remarkable speed. PMKLC-S showed average throughput improvements of up to 303.578%, while PMKLC-M, leveraging multiple GPUs, achieved an astounding average throughput improvement of up to 1071.043%. This makes PMKLC highly suitable for scenarios requiring fast data processing.
- Robustness: PMKLC-S and PMKLC-M also exhibited superior compression robustness, with improvements of 94.528% and 94.416% respectively, compared to some baselines. This indicates its stability and consistent performance across datasets with different characteristics and probability distributions.
- Memory Efficiency: The system proved to be highly memory-efficient, with average CPU memory savings of up to 81.277% for PMKLC-S and 83.967% for PMKLC-M. GPU memory savings were also substantial, reaching up to 58.763% and 57.486% respectively. This makes PMKLC viable for deployment on devices with limited resources.
In conclusion, PMKLC represents a significant advancement in lossless compression for large-scale genomics databases. By intelligently combining multi-knowledge learning, GPU acceleration, and parallel processing strategies, it offers a balanced and highly effective solution for the ongoing challenges of genomic data management. For more technical details, you can refer to the full research paper here.
