TLDR: This research paper provides a comprehensive comparative study of five local LLM runtimes on Apple Silicon: MLX, MLC-LLM, Ollama, llama.cpp, and PyTorch MPS. Evaluated on a Mac Studio M2 Ultra, the study measures performance across metrics like throughput, time-to-first-token (TTFT), long-context handling, quantization, streaming, batching, API compatibility, and deployment complexity. The findings indicate that MLX offers the highest sustained throughput and efficiency, making it ideal for throughput-critical production. MLC-LLM excels with lower TTFT for interactive workloads and robust long-context handling via paged KV caching. Ollama prioritizes developer ergonomics but lags in performance, while llama.cpp is efficient for single-stream use but lacks scalability. PyTorch MPS is deemed unsuitable for production-grade large model inference. The paper concludes that MLX and MLC-LLM are the most production-ready options, offering a practical hybrid path for on-device LLM serving.
Local inference of large language models (LLMs) on Apple Silicon devices is gaining significant traction due to its inherent advantages in privacy, cost control, and delivering low-latency responses directly on user devices. Unlike traditional cloud-based LLM services, running models locally means data never leaves the device, offering robust privacy guarantees. This approach is particularly appealing for applications like chat-based code generation, where context accumulates over time, making factors like the time it takes to get the first token (TTFT) and the smoothness of streaming responses crucial for a good user experience.
A recent research paper, Production-Grade Local LLM Inference on Apple Silicon: A Comparative Study of MLX, MLC-LLM, Ollama, llama.cpp, and PyTorch MPS, conducted a systematic study of five prominent local LLM runtimes on Apple Silicon: MLX, MLC-LLM, llama.cpp, Ollama, and PyTorch MPS. The study, performed on a Mac Studio with an M2 Ultra chip and 192 GB of unified memory, evaluated these frameworks using Qwen-2.5 family models and prompts ranging from a few hundred to 100,000 tokens. The goal was to understand their performance across various metrics relevant to production-grade deployment.
Understanding the Frameworks
Each framework brings a different set of design choices and trade-offs:
- PyTorch MPS: This is the baseline GPU backend for PyTorch on macOS. While easy to install, it’s often limited by memory constraints and performance gaps compared to other solutions, especially for larger models.
- llama.cpp: A lightweight C/C++ runtime known for its efficiency with quantized GGUF models. It offers strong single-stream performance but has limitations in batching and scalability for multi-user scenarios.
- Ollama: Built around llama.cpp, Ollama focuses on developer ergonomics, offering an OpenAI-compatible API and simple one-command model deployment. It prioritizes ease of use over peak performance.
- MLC-LLM: A TVM-based compiler and runtime that includes advanced features like paged KV caching, various quantization methods (AWQ/GPTQ), and built-in REST/SSE servers. It stands out for its out-of-the-box support for batching and API compatibility.
- MLX: An Apple-optimized engine tightly integrated with Metal and Core ML. It’s designed for raw throughput and efficiency on Apple GPUs and Neural Engine, featuring rotating KV caches and prompt cache support.
Key Findings and Performance Insights
The study evaluated these frameworks across several critical dimensions:
Throughput and Latency: MLX emerged as the leader in sustained generation throughput, achieving around 230 tokens/second. It also demonstrated very stable inter-token latency once decoding began. MLC-LLM was close behind at about 190 tokens/second but often delivered a faster Time-to-First-Token (TTFT), making it feel more responsive for interactive tasks. llama.cpp was efficient for short contexts but saw significant performance drops with longer inputs, while Ollama and PyTorch MPS lagged considerably in throughput.
Long-Context Handling: Efficiently managing long contexts (tens to hundreds of thousands of tokens) is vital for real-world applications. MLC-LLM excelled here with its vLLM-style paged KV caching, allowing it to sustain performance on contexts up to 128,000 tokens. MLX uses a configurable rotating KV cache and supports prompt cache files for shared prefixes, offering a good balance for typical chat and code generation contexts (4,000-32,000 tokens). Ollama provided simple prefix reuse but struggled with very long contexts, and llama.cpp and PyTorch MPS were not suitable for production-scale long-context inference.
Quantization Support: Quantization is crucial for running larger models efficiently on Apple GPUs. MLC-LLM and MLX offered the most flexible and production-viable quantization pipelines. MLC-LLM supported community-standard methods like AWQ and GPTQ, while MLX focused on Apple-first optimizations with mixed-bit formats (3/4/6/8-bit) and seamless integration with Metal and Core ML.
Streaming and Token Delivery: For interactive applications, smooth streaming and fast TTFT are paramount. Ollama provided the most turnkey streaming experience with a built-in SSE server. MLC-LLM offered lower TTFT for moderate prompts, making it responsive for chat. MLX, while requiring full prefill before streaming, delivered the most consistent inter-token latency once generation started, which is valuable for throughput-critical scenarios.
Batching and Concurrency: Handling multiple concurrent requests efficiently is a challenge for local LLM runtimes. MLC-LLM was the strongest in this area, providing a stable multi-worker HTTP/SSE server and kernels optimized for small micro-batches. MLX prioritized single-stream performance, requiring external orchestration for concurrency. None of the Apple-native runtimes matched the continuous batching capabilities of server-class solutions like vLLM on NVIDIA GPUs.
API Compatibility and Deployment: Developers often prefer OpenAI-compatible APIs. MLC-LLM offered the most complete out-of-the-box API story with a built-in REST server and cross-platform SDKs. Ollama provided the smoothest migration path for OpenAI API users. MLX relied on community wrappers for OpenAI compatibility but offered a leaner, more extensible stack. Ollama was the easiest to deploy for single-node setups, while MLX provided simple local installation but needed external wrappers for serving. MLC-LLM involved heavier DevOps due to TVM compilation.
Privacy and Security
All five frameworks operate entirely on local Apple Silicon hardware, ensuring strong privacy by default with no background telemetry. For enterprise compliance, additional measures like TLS termination, authentication, and encrypted volumes would need to be layered externally, typically via reverse proxies.
Also Read:
- LLMServingSim2.0: A Unified Platform for Simulating LLM Infrastructure with Diverse Hardware and Serving Strategies
- Boosting Mixture of Experts Performance with Enhanced Caching and Predictive Loading
Conclusion and Recommendations
The study concluded that MLX and MLC-LLM are currently the only production-ready runtimes for LLM inference on Apple Silicon. For Apple-first production deployments that prioritize raw efficiency and throughput, MLX is the recommended choice due to its optimization for Apple hardware. For latency-sensitive interactive chat and code generation workloads, especially those with growing contexts, MLC-LLM is preferred for its faster TTFT and smoother streaming at moderate prompt sizes, as well as its robust long-context handling.
Ollama remains ideal for prototyping and simple local APIs, while llama.cpp is efficient for single-user or embedded use but lacks scalability. PyTorch MPS is generally not viable for large-model inference on Apple Silicon due to memory and performance limitations. While Apple-native solutions are still catching up to the absolute performance of NVIDIA GPU inference solutions like vLLM, MLX and MLC-LLM are rapidly maturing into practical options for on-device, production-grade LLM inference.
