TLDR: This research paper introduces a scalable High-Performance Computing (HPC) architecture based on SLURM for deploying various Large Language Models (LLMs). It leverages dynamic resource scheduling, containerized microservices, and REST APIs to efficiently manage CPU, GPU, and memory allocations in multi-node clusters. Experiments with Llama models show that smaller models can handle high concurrency with low latency, while larger models quickly saturate. The architecture supports advanced workflows like a ‘tribunal’ system for quality control and real-world applications such as RAG-based chatbots, demonstrating its flexibility and robustness for efficient LLM inference on large-scale HPC infrastructures.
Large Language Models (LLMs) have revolutionized how we interact with text, enabling advanced understanding and generation of human-like language. However, deploying these powerful models to serve a large number of users simultaneously presents significant computational hurdles. This research paper introduces a novel High-Performance Computing (HPC) architecture designed to overcome these challenges, making LLM inference both scalable and efficient.
The core of this new architecture is built upon the Simple Linux Utility for Resource Management (SLURM), a widely used system for managing computing clusters. By leveraging SLURM, the system can dynamically allocate resources like CPUs, GPUs, and memory across multiple nodes, ensuring optimal utilization for various LLM inference engines. This setup allows for the seamless integration of containerized microservices, which are essential for managing resources efficiently and providing flexible client interactions.
The architecture includes Representational State Transfer Application Programming Interfaces (REST APIs) for both single and bulk inference requests, making it easy for applications to interact with the LLMs. It also supports advanced workflows, such as a multi-step “tribunal” refinement process for improving chatbot response quality. This system works by orchestrating inference job execution across diverse HPC clusters, dynamically generating SLURM batch scripts to allocate the necessary hardware resources.
A key feature is its ability to integrate various inference engines like Text Generation Inferences (TGIs) and vLLMs, ensuring they run efficiently in parallel. The system also incorporates NGINX for load balancing, which unifies multiple inference endpoints into a single, manageable address, distributing requests effectively across the HPC cluster.
Extensive experiments were conducted using different Llama models, including Llama 3.2 (1B and 3B parameters) and Llama 3.1 (8B and 70B parameters). The results provide crucial insights into throughput, latency, and concurrency. Smaller models, such as the 1B parameter Llama 3.2, demonstrated impressive performance, handling up to 128 concurrent requests with very low latency, typically under 50 milliseconds. In contrast, larger models, like the 70B parameter Llama 3.1, reached saturation with as few as two concurrent users, experiencing latencies exceeding 2 seconds. This highlights a critical trade-off between model size and the number of simultaneous users an LLM can efficiently serve.
The research also illustrates real-world applications, such as the deployment of chatbots with retrieval-augmented generation (RAG). This demonstrates the architecture’s flexibility and robustness in handling complex tasks, allowing client-side applications to build custom logic while leveraging the powerful HPC backend. The “tribunal” system, for instance, ensures high-quality chatbot responses by running a three-step workflow (generate, critique, revise), which can adapt to system capacity, bypassing advanced steps during peak usage to maintain low latency.
Also Read:
- HyperFlexis: Optimizing LLM Serving for Diverse Performance Needs
- AdLoCo: Enhancing LLM Training Through Adaptive Batching and Multi-Instance Strategies
In conclusion, this SLURM-based HPC architecture offers a highly efficient, responsive, and fault-tolerant solution for deploying LLM inference on a large scale. While there are clear performance differences based on model size, the system effectively scales LLM inference, paving the way for more advanced and reliable AI applications in high-performance computing environments. You can read the full research paper for more details here.
