TLDR: A new method called “Gradual Forgetting” uses logarithmic compression on input tokens to extend the context window of standard transformers without altering their architecture. Inspired by human memory, it creates a compressed representation of past information, allowing transformers to efficiently process much longer sequences and achieve better performance on language modeling tasks like WikiText-103 and PG-19.
In the rapidly evolving world of artificial intelligence, transformers have become the go-to architecture for processing sequences, from understanding language to forecasting time series. However, these powerful models face a significant hurdle: their ability to handle very long sequences is limited by the quadratic complexity of their self-attention mechanism. This means that as the input sequence gets longer, the computational cost grows exponentially, making it impractical for extremely long contexts.
Traditionally, researchers have tackled this problem by modifying the transformer’s internal architecture, introducing complex mechanisms like recurrence or auxiliary memory modules. While effective, these changes often add complexity and introduce new dependencies within the model.
A New Approach: Modifying the Input, Not the Architecture
A recent research paper titled “Gradual Forgetting: Logarithmic Compression for Extending Transformer Context Windows” by Billy Dickson and Zoran Tiganj from Indiana University Bloomington introduces an innovative alternative. Instead of altering the transformer’s core design, their method focuses on modifying the input representation itself. This approach is inspired by cognitive models of human memory, which suggest that our brains don’t store a perfect, verbatim record of every past event. Instead, memory forms a “mental timeline” where recent events are remembered with high resolution, while more distant events are recalled with gradually decreasing detail – a process known as logarithmic compression.
The core idea is to apply a scale-invariant logarithmic compression to the input tokens before they even reach a standard transformer. This results in a compressed representation that captures the essence of the distant past without needing to store every single token. Crucially, this compressed information is then processed by an *unmodified* transformer, preserving the architectural simplicity that makes transformers so appealing.
How Logarithmic Compression Works
The method uses a bank of unimodal temporal filters to create a log-compressed memory of the distant past. Imagine these filters as a series of “memory slots” that become progressively wider as they look further back in time. Recent events are captured by narrow, closely spaced filters, providing fine-grained detail. As you go further back, the filters become wider and more spread out, capturing the general gist of events but losing specific details. This is analogous to how our own memories work: we remember yesterday’s breakfast clearly, but last year’s breakfast is a blur.
This compressed representation, termed Scale-Invariant Temporal History (SITH), is then concatenated with a small number of the most recent, uncompressed tokens. This combined input – a fixed-size sequence of recent tokens plus the log-compressed memory of the distant past – is what the standard transformer processes. This clever preprocessing step allows the transformer to efficiently leverage information from a much longer context without incurring the full quadratic cost of attending to the entire sequence directly.
Impressive Results on Language Modeling
The researchers evaluated their approach on two major language modeling benchmarks: WikiText-103 and PG-19. Their findings were compelling: the input-level logarithmic compression consistently reduced perplexity (a measure of how well a probability model predicts a sample) compared to uncompressed baselines. Furthermore, performance improved consistently with longer compressed temporal contexts, demonstrating the effectiveness of this method in extending a transformer’s long-range memory.
On WikiText-103, their best model achieved a per-word perplexity of 23.56, which is competitive with, and in some cases outperforms, other transformer architectures of similar size that use more complex long-context memory approaches like Transformer-XL and Recurrent Memory Transformer (RMT). The key difference is that this new method achieves these gains through input preprocessing, rather than by carrying state across segments or modifying the attention mechanism itself.
Also Read:
- Boosting LLM Efficiency: How Token Permutation Makes Attention Sparser
- Understanding How Language Models Remember: The Influence of Time and Position
The Human Memory Connection
The paper emphasizes the profound inspiration drawn from cognitive science. The idea that memory resolution decreases logarithmically with time aligns with observed phenomena in human memory, such as the power-law decay of forgetting and the Weber–Fechner law. Neuroscience studies have even reported neurons that activate sequentially with logarithmically compressed temporal receptive fields. By integrating such cognitive principles, this research provides a principled framework for modeling domains where power-law decaying correlations naturally arise, from DNA sequences to natural language.
This work highlights that augmenting transformers with a scale-invariant memory, inspired by how humans remember, offers a simple and effective way to extend their context windows. It allows standard transformer architectures to capture long-range dependencies efficiently while maintaining computational tractability. For more details, you can read the full research paper here.
