TLDR: A new method called phantom parallelism significantly reduces the energy consumption and training time of large neural networks. By compressing data into “phantom layers” before communication, it minimizes the most energy-intensive part of traditional model parallelism, leading to up to 50% energy savings for feed-forward networks and enabling training of smaller models on fewer GPUs with substantial overall energy reductions.
Training large neural network models, such as the powerful Large Language Models (LLMs) that drive many modern AI applications, is an incredibly energy-intensive and costly endeavor. These models often require weeks or even months of training on high-performance supercomputers equipped with specialized accelerators like GPUs. For instance, training GPT-3 reportedly consumed electricity equivalent to the annual consumption of 120 US households, leading to significant carbon emissions. While training is a one-time cost, the continuous inference (using the trained model) can incur even greater energy costs over a model’s lifetime, making the energy and carbon footprint of AI a formidable and potentially unsustainable challenge.
To address the sheer size and computational demands of these models, various parallel training methods have been developed. These include data parallelism, where different parts of the training data are processed simultaneously; pipeline parallelism, which partitions entire layers of a model across multiple GPUs; and tensor parallelism, which partitions individual layers across GPUs. While data parallelism is generally energy-efficient due to minimal communication, model parallelism (including tensor parallelism) incurs substantial energy costs because of extensive communication and synchronization between devices. This communication overhead is a major contributor to the overall energy consumption in large-model training.
A new strategy, called phantom parallelism, has been introduced as an alternative to traditional tensor parallelism, specifically designed to minimize the net energy consumption. This approach focuses on reducing the most energy-inefficient component of large neural network training: the communication between different parts of the model.
The core idea behind phantom parallelism is to introduce additional, smaller layers, referred to as “phantom layers,” with “ghost neurons” between the input and output layers within each processing unit. When information needs to be communicated between different parts of the model, it is first compressed into these smaller phantom layers. This compression significantly reduces the amount of data that needs to be transmitted, thereby lowering both computation and communication overheads. Upon receiving the compressed information, the receiving unit decompresses it locally before using it for further calculations.
The researchers derived new mathematical operations for both the forward and backward passes of the training process in phantom parallelism and implemented them as custom operations within an end-to-end training pipeline. They then compared its performance and energy efficiency against conventional tensor parallel training pipelines.
Experiments conducted on up to 256 GPUs on the FRONTIER supercomputer demonstrated significant gains. Phantom parallelism showed a notable reduction in communication overhead compared to tensor parallelism. For large model sizes, phantom parallelism consistently outperformed tensor parallelism in terms of execution time per training cycle. In some cases, tensor parallelism couldn’t even be executed due to memory limitations, while phantom parallelism, with its reduced memory footprint, could successfully train the models.
Crucially, the study found that phantom parallelism can deliver approximately a 50% reduction in the energy consumed to train Feed-Forward Networks (FFNs) when compared with conventional tensor parallel methods. Beyond this, the proposed approach also showed that it could train smaller “phantom models” to the same level of accuracy (model loss) using fewer GPUs than what was required for larger tensor parallel models on more GPUs. This opens up the possibility for even greater energy savings; for example, training a phantom parallel model on 8 GPUs consumed over two orders of magnitude less energy and an order of magnitude less training time than training a comparable tensor parallel model on 256 GPUs to the same target loss. For more technical details, you can refer to the original research paper.
Also Read:
- Streamlining LLM Inference: FlashSVD’s Approach to Memory Efficiency
- Making MoE LLMs Leaner and Faster with Expert-Aware Compression
While the initial study was limited to simpler FFN architectures, the principles are applicable to FFN components found within more complex neural networks, such as transformer models. This work represents a significant step towards developing more energy-conscious and sustainable AI/ML training and inferencing at scale. Future research will focus on generalizing phantom parallelism to full transformer architectures, extending its application to inference workloads, and integrating it with other parallel training methods like pipeline and data parallelism for broader deployment in next-generation AI systems.
