TLDR: Cold-RL is a new learned cache eviction algorithm for NGINX that uses offline reinforcement learning (a Deep Q-Network) to make smarter decisions about which data to remove from the cache. Unlike traditional methods like LRU, Cold-RL considers multiple factors like age, size, and hit count, leading to significantly higher cache hit ratios (up to 146% improvement under pressure) and substantial infrastructure cost savings (23% origin traffic reduction) with minimal performance overhead. It’s designed for production stability with strict latency budgets and immediate fallbacks to LRU if needed.
Web services today face immense pressure to deliver content quickly and efficiently, with NGINX acting as a crucial first line of defense by caching frequently accessed data. However, a fundamental challenge remains: when the cache is full, which data should be removed to make space for new content? Traditionally, NGINX and many other systems rely on a simple rule called Least-Recently-Used (LRU).
While LRU is easy to implement and understand, it has significant drawbacks. It’s ‘size-blind,’ meaning a large, less important file can evict many smaller, critical assets. It’s also ‘periodicity-blind,’ failing to adapt to predictable traffic bursts, and it assumes that recently used items will be needed again soon, which isn’t always true in real-world scenarios. This ‘good enough’ approach often leads to inefficiencies, especially under heavy traffic, causing the cache to ‘thrash’ and forcing critical assets to be re-fetched from the origin server, increasing costs and latency.
Introducing Cold-RL: A Smarter Way to Evict
A new research paper, Cold-RL: Learning Cache Eviction with Offline Reinforcement Learning for NGINX, introduces an innovative solution that replaces LRU’s simplistic approach with an intelligent, learned algorithm. Cold-RL views cache eviction not as a simple bookkeeping task, but as a prediction problem: which object is most likely to be reused before it expires?
Cold-RL leverages offline reinforcement learning, specifically a Deep Q-Network (DQN) policy, to make these critical eviction decisions. Instead of relying on fixed rules, Cold-RL learns from the long-term consequences of its eviction choices by replaying NGINX access logs through a high-fidelity cache simulator. This offline training process is crucial because it allows the system to learn without risking production stability.
How Cold-RL Works
When NGINX needs space in its cache, the Cold-RL module springs into action. It doesn’t evaluate the entire cache; instead, it intelligently samples a small number of the ‘coldest’ (least recently used) objects – typically between 8 and 32. For each of these candidate objects, it extracts six lightweight but powerful features: age, size, hit count, inter-arrival time, time-to-live (TTL) remaining, and last origin Round-Trip Time (RTT).
These features are then fed to a compact ONNX inference sidecar, which houses the trained DQN policy. The policy quickly determines which objects should be evicted. A strict timeout of 500 microseconds (half a millisecond) is enforced for this entire process. If the decision isn’t made within this tiny window, or if any failure occurs, the system immediately and gracefully falls back to the native LRU mechanism, ensuring uninterrupted service and production stability.
Impressive Performance and Real-World Impact
The evaluation of Cold-RL against traditional policies like LRU, LFU, Size-Based, ARC, and Hybrid baselines shows remarkable improvements, especially under high cache pressure. In one scenario with a 25 MB cache, Cold-RL boosted the hit ratio from 0.1436 to 0.3538 – a staggering 146% improvement over the best classical policy. Even at medium pressure (100 MB cache), it improved the hit ratio from 0.7530 to 0.8675, a 15% gain. Under low pressure, it gracefully matched classical methods.
Crucially, this enhanced intelligence comes with minimal overhead. The inference process adds less than 2% CPU overhead and maintains eviction latency within the microsecond budget. In a real-world production deployment, Cold-RL has been serving over 100 million daily requests for three months with zero crashes and has achieved a substantial 23% reduction in origin traffic. This translates directly into significant infrastructure cost savings, potentially millions of dollars annually for large deployments, by reducing egress bandwidth and origin server load.
Also Read:
- OS-R1: Reinforcement Learning Powers Intelligent Linux Kernel Optimization
- Optimizing LLM Performance on Mobile Devices with Dynamic Edge Routing
A Blueprint for Future Systems
Cold-RL demonstrates that machine learning can be safely and effectively deployed in performance-critical infrastructure like web proxies. By carefully constraining the learning problem, separating training from serving, and designing for failure, this work provides a blueprint for bringing learned intelligence to systems software. It shows that cache eviction decisions no longer need to be bound by predetermined rules; they can learn, adapt, and improve, all while operating at microsecond speeds and remembering patterns over days.
