TLDR: A new benchmark, SymbolBench, evaluates Large Language Models (LLMs) on their ability to uncover hidden scientific laws from real-world time series data. It tests LLMs across multivariate symbolic regression, Boolean network inference, and causal discovery. The study introduces a unified framework where LLMs act as both predictors and evaluators, sometimes combined with genetic programming. Findings indicate LLMs excel in continuous data tasks but struggle with discrete logical systems, and that providing contextual information significantly improves their performance, challenging the notion that simpler solutions are always superior.
For centuries, scientists have strived to uncover the hidden laws that govern our world, often by meticulously analyzing data collected over time. Johannes Kepler, for instance, revolutionized astronomy by deriving the laws of planetary motion from astronomical observations, which are essentially time series data. Today, in the age of artificial intelligence, the aspiration to automate this challenging process – extracting underlying symbolic laws from dynamic time series data – remains a core pursuit.
Large Language Models (LLMs) have demonstrated remarkable capabilities in various complex reasoning tasks. However, their potential to infer interpretable, context-aligned symbolic structures from time series data has largely remained unexplored. Traditional methods, such as genetic programming used in symbolic regression, often prioritize fitting data accurately but sometimes at the expense of creating easily understandable equations. Furthermore, previous attempts to use LLMs in this domain have been limited, often treating them merely as generic function generators without leveraging their deeper reasoning potential, leading to results that might lack real-world relevance or focus only on simple algebraic expressions.
To address these gaps and systematically evaluate the symbolic reasoning abilities of LLMs for time series, researchers have introduced a comprehensive benchmark called SymbolBench. This new benchmark is designed with four key objectives: to use real-world time series data with known underlying symbolic structures, to include a diverse range of symbolic forms with varying complexity, to ensure a large and balanced sample distribution across tasks, and to provide a unified framework for evaluation.
SymbolBench uniquely spans three core tasks, covering major types of time series data. The first is Multivariate Symbolic Regression, which deals with continuous data to recover complex equations, such as coupled Ordinary Differential Equations (ODEs) that describe dynamic interactions among variables. The second task is Boolean Network Inference, focusing on discrete systems to identify logical rules, often seen in biological networks. Lastly, Causal Discovery aims to uncover structured causal graphs from multivariate data, identifying how variables influence each other over time. Each task includes challenging, real-world examples from fields like biology, physics, and healthcare.
The research also proposes a unified framework that integrates LLMs with genetic programming to form a closed-loop symbolic reasoning system. In this innovative setup, LLMs play a dual role: they act as both predictors, generating candidate expressions, and evaluators, judging the quality and plausibility of these candidates. This iterative refinement process allows for continuous improvement of the generated symbolic structures, optionally incorporating efficient genetic programming tools in a hybrid approach.
Empirical results from SymbolBench reveal critical insights into the current strengths and limitations of LLMs in temporal symbolic reasoning. LLMs generally outperform traditional baselines in multivariate symbolic regression and causal discovery tasks. However, they fall short when it comes to Boolean network inference. This difference can be attributed to the nature of the tasks: CDEs and SCMs involve continuous data where coefficient optimization can compensate for minor structural inaccuracies, while Boolean networks require precise logical rule recovery without such flexibility. The study also found that LLM performance tends to degrade as problem difficulty increases, particularly with higher system dimensionality or when evaluated on out-of-distribution data.
Interestingly, while increasing computational effort (like through Chain-of-Thought prompting) does not consistently improve performance across all tasks, providing problem-specific context significantly boosts LLM performance. This suggests that context helps constrain the solution space and guides the models toward more accurate and relevant inferences. Furthermore, the research challenges the conventional wisdom that simpler symbolic structures are always more generalizable, showing that in complex scientific domains, more intricate expressions can sometimes lead to better generalization, especially when context is considered. The study also highlights the benefits of combining LLMs with genetic programming, where LLMs can guide the initialization or evaluation processes of genetic programming, leading to enhanced performance.
Also Read:
- Enhancing Mathematical Problem-Solving with Knowledge Graphs and Executable Code
- Evaluating Trust in AI: A New Benchmark for Multimodal Model Confidence
In conclusion, this research introduces SymbolBench as a vital tool for evaluating LLMs in scientific discovery from time series data. It demonstrates that while LLMs show great promise, particularly when provided with rich context and integrated into iterative refinement frameworks, there are still areas for improvement, especially in handling discrete logical systems. The findings pave the way for future advancements in automated scientific discovery, emphasizing the importance of domain knowledge, context alignment, and structured reasoning. You can read the full paper here.
