Bridging Worlds: Deploying GenAI Services in High-Performance Computing Centers

TLDR: This research paper details the experiences of Sandia National Laboratories in deploying containerized Generative AI (GenAI) services within an HPC center. It describes a ‘converged computing architecture’ that integrates traditional HPC platforms with Kubernetes, container registries, and object storage to support GenAI workloads. Through a case study of deploying the vLLM inference server for Large Language Models, the paper highlights practical considerations, deployment complexities across different container runtimes and platforms, and performance benchmarking results. The authors conclude by discussing challenges related to diverse user interfaces and configurations, proposing the need for new tools to simplify containerized GenAI application deployment in HPC environments.

Generative Artificial Intelligence (GenAI) is rapidly changing how we approach complex problems, and its applications are increasingly sought after in High-Performance Computing (HPC) centers. However, deploying these sophisticated GenAI services, which often involve inference servers, databases, and user interfaces, presents unique challenges in traditional HPC environments. A recent paper titled “Experience Deploying Containerized GenAI Services at an HPC Center” by Angel M. Beltre, Jeff Ogden, and Kevin Pedretti from Sandia National Laboratories, sheds light on their practical experiences and solutions for integrating GenAI workloads into established HPC infrastructures.

The core of their work revolves around a “converged computing architecture” that blends the strengths of traditional HPC systems with cloud-like capabilities. This architecture is designed to support the deployment of containerized GenAI services alongside existing HPC modeling and simulation tasks, ensuring reproducibility and efficient resource utilization.

A Unified Computing Vision

The converged architecture brings together several key components:

• HPC Platforms: These are the workhorses for large-scale scientific computing, equipped with high-performance networking and parallel filesystems. Systems like Sandia’s “Hops” and “El Dorado” with significant GPU resources are being adapted to run GenAI workloads, even though they weren’t originally designed for persistent services.
• Kubernetes Platforms: Offering a more cloud-native approach, Kubernetes excels at deploying and managing long-lived, containerized services with high uptime. Sandia uses Red Hat’s OpenShift platform, including clusters like “Goodall” and “CEE-OpenShift,” which are equipped with powerful NVIDIA GPUs for advanced GenAI use cases.
• Container Registries: These act as central repositories for storing and sharing container images. Sandia leverages local GitLab servers and Red Hat Quay for managing container images, facilitating their transfer between different computing platforms and ensuring security scanning.
• Object Storage: A site-wide object storage service provides a globally accessible mechanism for storing large data objects, such as GenAI models. This is crucial for sharing data across HPC and Kubernetes platforms, especially when local filesystems aren’t externally mounted due to security concerns. Sandia’s S3 object storage, with its high bandwidth connections, supports this critical function.

Real-World Deployment: The vLLM Case Study

To illustrate their approach, the researchers present a detailed case study on deploying the vLLM inference server, which serves Large Language Models (LLMs) like Llama 4 Scout and Llama 3.1 405B. The workflow covers everything from downloading models to benchmarking their performance:

• Model Acquisition: Models are typically downloaded from platforms like Hugging Face and then stored in local object storage to ensure availability across all platforms, even when disconnected from the internet.
• Deployment Challenges: Deploying vLLM across different environments (HPC with Podman or Apptainer, and Kubernetes with Helm charts) highlights significant differences in user interfaces and configuration. For instance, Apptainer’s default behavior of running containers as the calling user, rather than root, required specific adjustments to prevent crashes.
• Accessing Services: Once deployed, accessing the GenAI inference server requires network routing. On HPC platforms, this can be done via SSH tunnels for individual users or a “Compute-as-Login” mode for broader, persistent access. Kubernetes platforms, in contrast, offer secure ingress routing as a built-in feature.
• Performance Benchmarking: The paper includes benchmarking results comparing the performance of vLLM on different HPC and Kubernetes platforms with various GPUs (NVIDIA H100, AMD MI300a). While specific performance numbers vary, the study provides a reproducible methodology for evaluating GenAI inference services. For example, the Hops platform (NVIDIA H100) showed higher throughput than El Dorado (AMD MI300a) for the Llama 4 Scout model, though these were unoptimized runs.
• Multi-node Inference: For extremely large models that don’t fit on a single node, multi-node inference using frameworks like Ray is necessary. The paper notes that while essential for memory, multi-node inference can be less reliable and doesn’t always improve performance.

Also Read:

• Streamlining FFRDC Operations with On-Premises AI
• AI-Enhanced Fuzzing: Large Language Models Bring Semantic Understanding to Vulnerability Discovery

Lessons Learned and Future Opportunities

The experience shared in this paper underscores the benefits of using containers for achieving reproducible software environments in GenAI deployments. However, it also highlights significant complexities in the user experience due to:

• Differences in container runtime user interfaces (e.g., Podman vs. Apptainer vs. Kubernetes).
• Variations in computing platforms requiring different container versions or configurations (e.g., CUDA vs. ROCm).
• The multitude of configuration options for GenAI applications and services (e.g., online vs. offline mode, single-node vs. multi-node).
• Site-specific configurations at different computing centers, which can hinder portability.

The authors see a clear opportunity for the HPC container community to develop tools that simplify these deployment challenges. They envision a “package manager” for containerized applications and services that could abstract away these complexities, making it easier for users to deploy GenAI workloads across diverse environments. This would significantly reduce the burden on HPC center users and unlock the full potential of GenAI in scientific research.

This research provides valuable insights for anyone looking to deploy GenAI services in complex HPC environments, offering a roadmap for integrating these powerful new tools. You can read the full paper for more technical details at Experience Deploying Containerized GenAI Services at an HPC Center.