TLDR: This research paper, “AI Factories: It’s time to rethink the Cloud-HPC divide,” proposes an integrated dual-stack architecture for AI Factories. It aims to combine the raw performance of High-Performance Computing (HPC) supercomputers with the usability and elasticity of cloud-native technologies. The paper identifies key challenges in data management, compute orchestration (for both AI training and inference), and user experience, advocating for solutions like unified data infrastructures, high-performance Kubernetes, serverless Functions-as-a-Service (FaaS), and user-friendly Software-as-a-Service (SaaS) front-ends. This approach is crucial for optimizing resource utilization, making AI infrastructure accessible to a wider range of practitioners, and integrating AI Factories into a broader Cloud-Edge Compute Continuum to support sovereign AI initiatives.
Artificial Intelligence (AI) is rapidly becoming a cornerstone of national strategy, leading many countries, particularly in Europe, to invest heavily in what are known as AI Factories (AIFs). These facilities are designed to ensure technological independence and provide the necessary resources for local digital ecosystems. However, a significant challenge arises because many of these AIFs are built upon existing High-Performance Computing (HPC) supercomputers.
HPC systems are renowned for their raw processing power, making them ideal for demanding tasks like training large language models. Yet, they are not inherently designed for ease of use, accessibility, or serving as public-facing platforms for AI services such as inference or agentic applications. AI practitioners, on the other hand, are accustomed to the flexibility and user-friendliness of cloud-native technologies like Kubernetes and object storage, which are often difficult to integrate into traditional HPC environments.
This research paper, titled AI Factories: It’s time to rethink the Cloud-HPC divide, advocates for a novel approach: an integrated dual-stack architecture within supercomputers. The goal is to bridge the gap between HPC and cloud computing, combining high performance and hardware acceleration with the ease of use and service-oriented front-ends that AI developers expect. This convergence allows each paradigm to enhance the other, creating a more versatile and efficient infrastructure for AI.
The Need for Integration
The paper highlights that relying solely on HPC technologies cannot sustain the diverse demands of modern AI workloads. AI Factories require two distinct but interconnected stacks: one for dedicated hardware and traditional HPC technologies (like SLURM and GPFS), and another for virtualized resources and elastic cloud technologies (such as Object Storage and Kubernetes). Crucially, these stacks cannot operate in isolation.
Key areas for integration include:
- Resource Pools: Expensive resources, especially GPUs, must be shared efficiently between HPC and cloud schedulers to prevent wastage.
- Data Infrastructure: A unified data infrastructure is essential, allowing data catalogs, datasets, and pipelines to operate across both stacks. This requires ensuring high-performance parallel data access for both HPC systems (via Parallel File Systems) and cloud services (via Object Storage).
- Compute Infrastructure: High-performance containers with optimized dependencies and libraries should be available for both stacks, enabling ‘Acceleration-as-a-Service’ across environments.
- AI Infrastructure: A unified model catalog, user-friendly SaaS interfaces with automated provisioning, and seamless AI pipelines are needed across both ecosystems.
- External Integration: User-facing AI pipelines, like inference and agent-based workloads, require integration with external cloud resources through cloud federation endpoints.
Addressing Specific Challenges
The paper delves into specific challenges and proposed solutions for this dual-stack integration:
Data Management: Cloud providers excel with Object Storage for massive data lakes and catalogs. HPC often uses Parallel File Systems (PFS). The challenge is to build serverless data infrastructures on Object Storage with open catalogs and elastic query services, while also evolving cloud tools to deliver high-bandwidth Object Storage tightly integrated with PFS for unified, multi-tenant storage.
AI Training and Kubernetes: Kubernetes has become the standard for deploying AI workloads in the cloud. Supercomputing centers must support Kubernetes natively to attract AI startups. The paper argues against mere wrapper tools, emphasizing the need for large, multi-tenant Kubernetes clusters that can leverage expensive GPU resources from an integrated pool of both Slurm-managed HPC nodes and cloud-native nodes. For digital sovereignty, open-source solutions for High-Performance Kubernetes are crucial.
AI Inference and Burstability: Serverless computing (Functions-as-a-Service or FaaS) is ideal for dynamic, short-lived AI inference tasks and data preprocessing due to its rapid scaling and cost efficiency. HPC environments, traditionally designed for long-running jobs, need to support rapid burst allocation of compute resources to emulate cloud FaaS platforms. This involves integrating serverless technologies into both stacks and developing high-performance FaaS deployments with super-fast communication and access to accelerators.
Empowering AI Practitioners: Current supercomputers often require deep technical expertise. To open them to a broader user base, AIFs must offer higher-level PaaS and SaaS services, such as managed Jupyter Notebooks, Spark, Dask, and PyTorch environments. This includes automated resource provisioning and high-level visual interfaces to simplify the user experience and prevent resource waste.
Also Read:
- Benchmarking AI Systems as a Learning Problem with FlexBench
- Paving the Way for Sustainable 6G: A Deep Dive into Energy-Aware Network Design
Beyond AI Factories: The Compute Continuum
Looking ahead, AI Factories are envisioned to integrate within a broader ‘Cloud-Edge Continuum’. This means extending their reach to distributed AI pipelines across multiple data centers and edge locations. This federation is crucial as individual data centers face limits in power and cooling, and AI workloads continue to grow in demand. Such integration will require new resource allocation strategies and distributed AI software stacks that can operate across disparate sites.
In conclusion, the paper emphasizes that AI Factories represent a significant investment in Europe’s pursuit of AI innovation. Adopting an integrated dual-stack architecture, combining the strengths of HPC and cloud technologies, is vital to unlock new opportunities, ensure efficient resource utilization, and foster a thriving, accessible AI ecosystem for the future.
