TLDR: Murakkab is a new system designed to make agentic workflows in cloud platforms more resource-efficient. It introduces a declarative way to define workflows, separating logic from execution details, and uses a profile-guided optimizer and adaptive runtime to dynamically manage resources and configurations. This approach allows Murakkab to significantly reduce GPU usage, energy consumption, and cost while meeting performance and quality goals, addressing the inefficiencies of current fragmented AI application serving frameworks.
Agentic workflows, which coordinate multiple AI models and tools to complete complex tasks, are rapidly becoming the standard for AI applications. However, current frameworks struggle to serve these workflows efficiently. The core issue is that these systems treat workflows as opaque sequences of model and tool calls, tightly linking the agent’s logic with specific model and hardware choices. This fragmentation prevents systems from making smart trade-offs across factors like accuracy, speed, energy use, and cost, leading to wasted resources and missed service goals.
A new system called Murakkab aims to solve these inefficiencies. Developed by researchers from MIT CSAIL, Microsoft Azure Research – Systems, and Microsoft Azure, Murakkab introduces a fresh approach to managing agentic workflows. It uses a declarative abstraction that separates how a workflow is defined from how it’s actually run. This allows a profile-guided optimizer and an adaptive runtime to manage the entire process, from orchestrating workflow components to mapping them to the right models and hardware, and even dynamically adjusting execution to meet user-defined service-level objectives (SLOs).
Understanding the Problem
Today’s agentic workflows are often fragmented across different tools and platforms. Developers use frameworks like LangChain or LlamaIndex to combine models and tools, then call APIs from providers like OpenAI or Databricks, and finally rely on cloud providers for the computing infrastructure. Each layer has its own goals (e.g., latency, quality, cost), but there’s minimal coordination. This leads to several problems:
- Tight Coupling: Hard-coded parameters and hardware choices make it difficult for automated systems to optimize.
- Disjoint Orchestration: Workflow creation (frameworks) and resource management (cloud schedulers) operate in silos, leading to inefficient scheduling.
- Difficult Trade-offs: Balancing accuracy, latency, energy, and cost involves navigating a huge number of configurations, which is nearly impossible manually.
These limitations increase costs, degrade service quality, and waste resources. Furthermore, static definitions make workflows rigid; any update to models or hardware requires manual changes and redeployment.
Murakkab’s Solution: A Declarative and Adaptive Approach
Murakkab is built on two main principles: a declarative way to specify workflows and an adaptive, SLO-aware runtime system. This means developers can describe workflows as logical tasks and dependencies without worrying about specific models, tools, or hardware. Murakkab then takes over, integrating workflow orchestration with resource management. It can dynamically reconfigure workflow parameters and hardware settings based on offline profiles and real-time monitoring to optimize for quality, latency, energy, and cost.
How Murakkab Works in Practice
Murakkab manages the entire workflow lifecycle in three phases:
1. Workflow Development: Developers use a declarative approach, specifying high-level tasks and their data flow. They don’t need to define resource allocation or model selection. Murakkab uses an “Executor Library” that supports various LLMs, traditional machine learning models, and tools. An orchestrator, powered by an LLM with tool-calling capabilities, interprets these specifications and maps tasks to appropriate executors, creating a “Logical Workflow” – an abstract execution plan.
2. Optimization and Deployment: Murakkab builds detailed “Profiles” for both workflows and individual models. Workflow profiles capture accuracy, latency, and resource usage for different configurations. Model profiles detail performance across various hardware setups (e.g., GPU types, parallelism), including latency, energy consumption, and cost. A “Workflow Optimizer” then uses these profiles to transform the logical workflow into an “Executable Workflow.” It selects workflow parameters, models/tools for each executor, and hardware/parallelism strategies to meet SLOs while minimizing energy or cost. This optimization is formulated as a Mixed Integer Linear Program (MILP).
3. Execution: When a user request comes in, Murakkab retrieves the corresponding executable workflow from its registry. Users can specify SLOs (e.g., best, good, fair, basic for quality and latency). Murakkab continuously adapts to dynamic conditions. The optimizer runs in the background, typically every 60 minutes, to adjust to changing load and resource availability. An “Auto-Scaler” monitors per-model instance load and rapidly scales resources up or down to prevent SLO violations, especially given the high variability in agentic workflow demands.
Impressive Results
Evaluations on diverse workflows, including video question answering and code generation, show significant efficiency gains. Murakkab reduced GPU usage by up to 2.8 times, energy consumption by 3.7 times, and cost by 4.3 times compared to state-of-the-art baselines like LangGraph, all while maintaining user-defined service-level objectives. It also demonstrated the ability to adapt to dynamic resource availability, intelligently trading off cost for energy savings by leveraging different GPU types.
For example, in a scenario involving parallel video Q/A and code generation, Murakkab could intelligently decide to run object detection on a GPU and speech-to-text on a CPU. This optimized configuration met a 30-second latency goal while using fewer GPUs overall, showcasing its ability to make workflow-aware scheduling decisions that traditional systems cannot.
Also Read:
- HyperFlexis: Optimizing LLM Serving for Diverse Performance Needs
- Scaling Large Language Model Inference on HPC Clusters with SLURM
Looking Ahead
Murakkab represents a significant step forward in making agentic workflows more efficient and manageable in cloud environments. By providing a declarative programming model and an adaptive runtime, it decouples workflow logic from execution specifics, allowing for dynamic reconfiguration and resource-aware scheduling. The researchers see future opportunities in extending this adaptive co-optimization to larger clusters, supporting more diverse hardware, and specializing for new types of agentic applications. You can read the full research paper here.
