TLDR: A new research study, JustDense, challenges the necessity of complex sequence mixers in Time Series Analysis (TSA). By systematically replacing these mixers with simple dense layers in various state-of-the-art TSA models, the study found that dense layers achieve comparable or even superior performance across 29 benchmarks and five different tasks. The findings suggest that simpler architectures can be highly effective for time series data, often leading to better performance and sometimes improved efficiency, thereby questioning the prevailing belief that more complex models are always better.
Time Series Analysis (TSA), which involves examining data that changes over time, is crucial across many fields, from finance and logistics to weather forecasting. Traditionally, statistical models were used, but more recently, complex sequence models with intricate mechanisms like attention have become the norm.
These modern TSA architectures typically follow a two-stage process: a sequence mixer gathers information across the temporal axis, and then a channel mixer refines representations within each time step. The MatrixMixer framework provides a unified way to understand these sequence mixers, interpreting them as L × L mixing matrices.
However, recent studies have started to question whether these complex sequence mixers are truly necessary. Some simpler models, like DLinear and TSMixer, have shown that basic linear layers or lightweight MLP blocks can perform as well as, or even better than, attention-based models for certain tasks. This suggests that the perceived benefits of complex mixers might actually come from other parts of the model’s design or optimization.
Introducing JustDense
To explore this question systematically, researchers proposed JustDense, an empirical study that replaces common sequence mixers in various well-established TSA models with simple dense layers. Grounded in the MatrixMixer framework, JustDense treats any sequence mixer as a mixing matrix and substitutes it with a dense layer. This approach helps isolate the mixing operation, providing a clearer understanding of its role.
The study conducted extensive experiments across 29 benchmarks, covering five key TSA tasks: classification, anomaly detection, imputation, long-term forecasting, and short-term forecasting. They tested this approach on seven state-of-the-art TSA models, including Transformer, Autoformer, PatchTST, Mamba, iTransformer, S-Mamba, and ModernTCN.
Surprising Results and Efficiency Insights
The findings were compelling: replacing complex sequence mixers with dense layers yielded comparable or even superior performance across all five tasks. This indicates that simple dense matrices can effectively enhance various TSA models, regardless of their original underlying mechanisms.
One reason for this unexpected performance is that dense matrices operate in a full L × L matrix space, allowing them to capture richer, high-rank interactions compared to the inherently low-rank nature of attention, Toeplitz, or semiseparable mixers. Furthermore, the optimization process itself, particularly gradient descent, tends to favor solutions with minimal nuclear norm, meaning dense matrices can naturally recover structured, low-rank behavior when appropriate, while also modeling additional dependencies.
The study also looked at efficiency. For models with an Encoder-Decoder structure (like Transformer and Autoformer), replacing attention with a dense layer increased the number of parameters but decreased training time. This is because the computational overhead of the attention mechanism is eliminated. For Encoder-only models (like iTransformer and PatchTST), the conversion to dense layers resulted in a reduction in both parameter count and training time, as dense layers can require fewer parameters than the complex computations of self-attention.
For SSM-based models (like Mamba and S-Mamba), dense layers led to more parameters but faster training. This is likely due to the highly optimized libraries for dense operations, such as cuBLAS, which perform parallel computations more efficiently in practice. Only for the convolution-based ModernTCN did training time increase while parameters decreased, aligning with the theoretical understanding that convolution operations are generally faster than dense operations.
Also Read:
- New Wavelet-Based Models Enhance Time Series Forecasting
- Two-Layer Transformers Prove Capable of Learning Any Complex Sequence Pattern
Conclusion: Simplicity Prevails
The research provides strong empirical evidence that the complexity often found in modern sequence mixers might be unnecessary for time series data. JustDense demonstrates that simple dense matrices can match or even surpass the performance of more specialized mixers. This challenges the common assumption that “deeper and more complex architectures are inherently better” in Time Series Analysis, suggesting that the unique characteristics of time series data may require different design principles than those borrowed from fields like natural language processing.
