TLDR: The Federated Inference Resource Scheduling Toolkit (FIRST) is a new framework that enables Inference-as-a-Service on distributed High-Performance Computing (HPC) clusters. It provides secure, cloud-like access to diverse AI models, including Large Language Models (LLMs), for scientific researchers without relying on commercial cloud infrastructure. Leveraging Globus Auth and Globus Compute, FIRST offers an OpenAI-compliant API, supports various inference backends, auto-scales resources, and provides both high-throughput batch and low-latency interactive modes. This system addresses the challenges of deploying scalable AI on HPC, ensuring data security and cost-effectiveness for scientific workflows, and has already processed billions of tokens in a proof-of-concept deployment.
Scientific research is increasingly relying on Artificial Intelligence (AI) models, from analyzing vast datasets to optimizing complex simulations. However, deploying these powerful AI tools on traditional High-Performance Computing (HPC) clusters has been a significant hurdle. These clusters, while powerful, are not designed for the dynamic, interactive nature of AI inference, often leading to complex authentication, scheduling, and resource management challenges. This is where the Federated Inference Resource Scheduling Toolkit (FIRST) steps in, offering a groundbreaking solution to bring cloud-like AI inference capabilities directly to scientific HPC environments.
Developed by researchers at Argonne National Laboratory and the University of Chicago, FIRST is a framework that enables “Inference-as-a-Service” across distributed HPC clusters. It provides scientists with secure, scalable, and on-premises access to a diverse range of AI models, including Large Language Models (LLMs), without needing to rely on commercial cloud infrastructure. This is particularly crucial for sensitive scientific data that cannot be shared externally.
Bridging HPC and AI with a Smart Architecture
The core of FIRST’s innovation lies in its architecture, which seamlessly integrates existing HPC infrastructure with modern AI serving needs. It comprises three main components:
- Inference Gateway API: This acts as the user’s entry point, offering an OpenAI-compliant API. This means researchers can use familiar tools and codebases to interact with AI models. It handles user authentication through Globus Auth, a robust identity and access management service, ensuring secure access across different institutions.
- Globus Compute: This is the communication backbone, allowing the Gateway API to remotely trigger AI inference tasks on any HPC system where a Globus Compute endpoint is deployed. It makes FIRST cluster-agnostic, meaning it can work with various underlying schedulers like Kubernetes, PBS, or Slurm. This component is also responsible for dynamically acquiring and managing compute resources.
- Model Serving Tools: These are the actual AI model serving frameworks, such as vLLM and Infinity, optimized for different GPUs (NVIDIA A100, H100, AMD MI250). FIRST is designed to be flexible, supporting a wide array of state-of-the-art inference frameworks and models.
One of FIRST’s clever features is its “hot node management.” Once compute nodes are acquired and a model is loaded, they remain active for a period, significantly reducing latency for subsequent requests. This is vital for interactive scientific workflows where immediate feedback is essential. The system also supports auto-scaling, dynamically adjusting resources based on demand, and offers both high-throughput batch processing for large-scale tasks and interactive modes for real-time engagement.
Real-World Impact and Performance
A proof-of-concept deployment on the Sophia HPC cluster at the Argonne Leadership Computing Facility (ALCF) has already demonstrated FIRST’s immense potential. In just ten months, the service processed 8.7 million inference requests, served 76 users, and generated over 10 billion tokens. This highlights its capability to handle substantial scientific workloads.
Performance evaluations show that while FIRST introduces a minimal overhead at very low request rates, its asynchronous gateway architecture provides superior scalability under high load conditions, outperforming direct access to inference engines like vLLM. When compared to commercial cloud offerings like OpenAI’s GPT-4o-mini, FIRST, running on dedicated HPC resources, demonstrated significantly higher throughput for concurrent requests, making it ideal for scientific tasks requiring massive data processing. The system’s auto-scaling capabilities further enhance its ability to adapt to varying demands, improving both throughput and responsiveness.
Also Read:
- RobotFleet: Centralized Planning for Heterogeneous Robot Fleets
- ROLL Flash: Boosting LLM Reinforcement Learning with Asynchronous Processing
Diverse Applications in Science
The research paper showcases several compelling case studies:
- Model Evaluation and Comparison: Researchers used FIRST to benchmark various GPT-style models against custom scientific models, reducing evaluation time by 40% due to the ability to instantly swap models.
- HPC Assistant with RAG: A chatbot was developed using FIRST’s embedding and inference services to provide on-demand support for HPC environments, leveraging Retrieval-Augmented Generation (RAG) to offer highly relevant guidance from manuals and documents.
- Synthetic Data Generation: The batch processing mode proved invaluable for generating billions of tokens of synthetic data for training machine learning models, completing tasks in hours that would otherwise take days.
FIRST represents a significant leap forward in democratizing AI access for scientific research. By securely hosting and scaling AI inference workloads on existing HPC systems, it empowers researchers without deep HPC expertise to leverage advanced AI capabilities while maintaining institutional security and resource policies. This framework is not just an alternative to commercial clouds; it’s a specialized solution designed to meet the unique demands of scientific computing. For more detailed information, you can read the full research paper here.
