TLDR: This research paper investigates the energy consumption of parallel neural network training, focusing on ResNet50 and FourCastNet models. It analyzes how parallelization parameters like GPU count, global batch size, and local batch size impact training time, predictive performance, and energy use. Key findings include that energy consumption scales linearly with GPU hours, but the scaling factor varies by model and hardware. The study also highlights the trade-offs between speedup, accuracy, and energy efficiency, and compares the power profiles of A100 and H100 GPUs, emphasizing the need to balance these factors for sustainable AI development.
The rapid growth of artificial intelligence, particularly in training complex neural networks, has led to a significant increase in energy consumption. While parallel computing methods have been crucial for handling larger models and datasets and speeding up training, their environmental impact is often overlooked. A recent study by researchers at the Karlsruhe Institute of Technology, including Philipp Huber, David Li, Juan Pedro Gutiérrez Hermosillo Muriedas, Deifilia Kieckhefen, Markus Götz, Achim Streit, and Charlotte Debus, delves into this critical area. Their paper, titled “Energy Consumption in Parallel Neural Network Training,” explores the intricate relationship between parallelization parameters and the energy footprint of AI training.
The research addresses a significant gap in understanding how parallelization affects the energy consumption of neural network training. While parallelism is essential for managing large datasets and keeping training times reasonable, scaling up can sometimes lead to disproportionate energy use and even reduced model performance. The study aims to provide insights into this delicate balance between accuracy, training time, and energy efficiency.
Investigating Energy and Performance
To shed light on these dynamics, the researchers conducted extensive experiments using two different neural network models: ResNet50, a widely used image classification model, and FourCastNet, a more computationally intensive weather forecasting model. They varied key parallelization parameters, including the number of GPUs used, the global batch size (total samples processed in one update), and the local batch size (samples processed per GPU). The impact on predictive performance, overall training time, and energy consumption was meticulously evaluated.
A core finding was that energy consumption generally scales almost linearly with the amount of computational resources consumed, specifically GPU hours. However, the exact scaling factor varied considerably depending on the specific model being trained and the hardware used. This factor was systematically influenced by how many data samples and gradient updates were processed per GPU hour.
ResNet50: Insights into Image Classification Training
For ResNet50, experiments were conducted with both a fixed dataset size and a dataset size that increased proportionally with the number of GPUs. When keeping the dataset constant and increasing GPUs with a fixed local batch size, training time decreased, but a phenomenon known as “large batch effects” emerged, where prediction accuracy started to decline beyond a certain point. Energy consumption, in this scenario, continuously increased with more GPUs due to less-than-ideal speedup. Interestingly, CPU energy initially decreased, but GPU energy increased due to inefficiencies introduced by parallel processing.
Conversely, when the global batch size was kept constant, the large batch effects were avoided, and accuracy remained stable. Training time scaling was nearly optimal, though energy consumption still generally increased with more GPUs. For experiments where the dataset size scaled with the number of GPUs, keeping the local batch size constant resulted in constant training time but increasing GPU hours and energy consumption. If the global batch size was kept constant, accuracy continually improved as more data was added, and energy consumption scaled approximately linearly with the number of samples.
FourCastNet: Demanding Workloads and Hardware Differences
FourCastNet, being a more demanding model, showed different scaling behaviors. Its speedup diverged from ideal scaling much earlier, and large batch effects appeared at much lower global batch sizes compared to ResNet50. Increasing the number of GPUs for FourCastNet led to a substantial increase in energy consumption, especially during the fine-tuning phase of training.
The study also compared the energy efficiency of NVIDIA A100 and H100 GPUs. While H100 GPUs reduced training times and GPU hours by 25% to 30% compared to A100s, the total energy consumed differed by less than 5%. This was attributed to the higher power consumption of the H100 nodes. Furthermore, the power profiles of the GPUs revealed that A100s exhibited broad fluctuations, suggesting inefficient utilization and potential for hardware degradation. H100s, on the other hand, showed narrower fluctuations, indicating more efficient operation.
Also Read:
- Enhanced LLM Performance on AI PCs with New Inference Runtime
- New Framework Converts AI Models to Energy-Efficient Spiking Networks Without Retraining
Balancing Act for Sustainable AI
The research highlights that the energy consumed by GPUs is closely tied to the number of samples and gradient updates processed per GPU hour. A higher throughput of samples per GPU hour generally leads to higher GPU power consumption. The study concludes that understanding these relationships is vital for developing resource-efficient deep learning practices. Future AI research and development must carefully balance training time, predictive performance, and energy consumption to achieve more sustainable outcomes. For more details, you can refer to the full research paper here.
