TLDR: The Two Level Perceptron (TLP) predictor is a new hardware-based neural system designed to improve computer performance by efficiently managing memory access. It combines off-chip prediction, which anticipates whether data will be found in main memory, with adaptive prefetch filtering at the first-level cache. TLP uses two interconnected perceptron predictors (FLP and SLP) to selectively delay speculative memory requests and discard useless prefetches, significantly reducing DRAM transactions and achieving notable performance speedups for memory-intensive applications with a minimal hardware footprint of 7KB.
Modern computer applications often deal with vast amounts of data, far exceeding the capacity of a computer’s fast internal memory, known as the cache hierarchy. This leads to frequent accesses to the slower main memory (DRAM), causing significant delays and consuming more energy. Researchers are constantly looking for ways to make these memory accesses more efficient.
A new research paper introduces a novel solution called the Two Level Perceptron (TLP) predictor. This innovative system aims to tackle the challenges of memory-intensive applications by combining two crucial techniques: predicting whether data will be found in the main memory (off-chip prediction) and intelligently filtering unnecessary data requests (prefetch filtering) at the first-level data cache (L1D).
The TLP predictor is a hardware-based approach that uses a multi-level neural network, specifically perceptrons, to make smart decisions about memory access. It’s composed of two interconnected parts: the First Level Predictor (FLP) and the Second Level Predictor (SLP).
How TLP Works: Two Levels of Intelligence
The FLP is designed to accurately predict if a memory access will need to go all the way to the main memory (DRAM) or if it will be found in one of the faster caches. What makes FLP unique is its ‘selective delay’ mechanism. Unlike previous systems that might immediately send a request to DRAM if they predict an off-chip access, FLP uses confidence thresholds. If it’s highly confident, it sends a speculative request to DRAM in parallel with checking the L1D cache. If its confidence is moderate, it waits until the L1D cache is checked. This prevents unnecessary DRAM accesses for data that might actually be in the L1D, significantly reducing wasted bandwidth.
The SLP works in conjunction with the L1D cache to filter out ‘prefetches’. Prefetching is a technique where the system tries to guess what data the processor will need next and fetches it into the cache proactively. While useful, inaccurate prefetches can flood the cache with useless data, slowing things down. SLP leverages the off-chip predictions made by FLP to decide whether a prefetch request is likely to be useful or not. If SLP predicts a prefetch will likely end up in DRAM (meaning it’s probably useless), it discards the request, preventing cache pollution and saving memory bandwidth.
Also Read:
- Boosting LLM Performance: How Processing-Near-Memory Redefines KV-Cache Management
- FeNN-DMA: A Programmable RISC-V System for Advanced Spiking Neural Networks
Key Advantages and Performance
The TLP predictor is the first hardware proposal to combine both off-chip prediction and prefetch filtering using a multi-level perceptron approach. It’s also remarkably efficient, requiring only about 7KB of storage, making it practical for real-world designs.
Extensive evaluations comparing TLP with state-of-the-art methods like Hermes (for off-chip prediction) and PPF (for prefetch filtering) showed significant improvements. Across a wide range of single-core and multi-core workloads, TLP reduced the average number of DRAM transactions by 30.7% and 17.7% respectively, compared to a baseline system with advanced cache prefetchers but no off-chip prediction. This reduction in DRAM traffic translates directly into performance gains, with TLP achieving geometric mean performance speedups of 6.2% for single-core and 11.8% for multi-core workloads.
The research highlights that TLP is effective regardless of the specific L1D prefetching logic used, demonstrating its versatility. It particularly shines in memory-intensive applications, such as graph-processing workloads, where it significantly reduces the pressure on the memory subsystem.
This work represents a significant step forward in optimizing memory hierarchy performance, offering a cost-effective and efficient solution to a long-standing challenge in computer architecture. For more detailed information, you can read the full research paper here.
