TLDR: A new machine learning framework called “Information Mechanics” is introduced, which uses information-theoretic uncertainty (surprisal) to directly analyze raw data. This model-free approach eliminates the need for explicit data distribution models, reduces bias, and offers enhanced interpretability and traceability. It achieves competitive performance across various tasks like supervised learning, anomaly detection, causal discovery, and reinforcement learning, presenting a human-understandable alternative to traditional neural networks.
A new research paper titled “A Theory of the Mechanics of Information: Generalization Through Measurement of Uncertainty (Learning is Measuring)” introduces a groundbreaking approach to machine learning. Authored by Christopher J. Hazard, Michael Resnick, Jacob Beel, Jack Xia, Cade Mack, Dominic Glennie, Matthew Fulp, David Maze, Andrew Bassett, and Martin Koistinen from Howso Incorporated, this work proposes a model-free framework that could reshape how we understand and implement artificial intelligence.
Traditional machine learning often relies on creating explicit models and making assumptions about how data is distributed. This can limit how flexible and understandable these systems are. The new framework, however, bypasses these limitations by directly analyzing raw data using a concept called ‘surprisal,’ which is a measure of information-theoretic uncertainty. Imagine learning not by building a complex mental model, but by constantly measuring how surprising new information is compared to what you already know.
This innovative method offers several key advantages. It eliminates the need to model data distributions, which in turn reduces inherent biases and allows for more efficient updates, including direct editing or deletion of training data. By quantifying relevance through uncertainty, the approach enables a wide range of tasks, from generating new data and discovering causal relationships to detecting anomalies and forecasting time series. A core philosophy behind this work is to ensure traceability, interpretability, and data-driven decision-making, offering a unified and human-understandable framework for machine learning.
The mathematical foundations of this theory create what the authors call a “physics” of information. This allows the techniques to be applied effectively to diverse and complex data types, even when data is missing. Empirical results suggest that this could be a viable alternative to neural networks for scalable machine learning and artificial intelligence, crucially maintaining human understandability of the underlying mechanics.
The Role of Surprisal
At the heart of this framework is ‘surprisal,’ or self-information. It quantifies how much information is gained or lost when observing a particular data element to predict another. Mathematically, it’s the negative logarithm of a probability, making it a natural way to measure uncertainty. In essence, if two data points are very similar, it would be unsurprising to use one in place of the other for a prediction. Surprisal provides a flexible tool to easily convert between distance and uncertainty, carrying uncertainty characteristics throughout the inference process.
The paper acknowledges that performing inference directly from data isn’t new, citing k-Nearest Neighbors (kNN) as an example. While kNN was largely set aside due to computational challenges with large datasets and the “curse of dimensionality,” this new approach improves upon it by using surprisal to measure uncertainty among data points, making it more robust and scalable.
How it Works: Learning as Measuring
Instead of optimizing a specific objective function, this system views learning as a process of measuring and characterizing uncertainty in the relationships among data. It estimates all these uncertainties using robust techniques focused on mean absolute differences. For any given inquiry, the system determines the probability of each piece of data being informative. It even uses a feature to predict itself, which, while seemingly overfitted, acts as a proxy for potentially relevant features not explicitly included in the dataset. This allows the system to compute the overall probability that one record is informative for another. The key takeaway is that there is no traditional “model” beyond the data and its inherent uncertainties; the system queries the data directly using these uncertainties, making every computation traceable and understandable.
Beyond Predictions: Causal Discovery and More
The framework extends beyond simple predictions. It can determine how individual features contribute to predictions without the biases often found in traditional feature importance techniques. More profoundly, it can uncover causal relationships by analyzing how uncertainty in one variable changes when another is considered. Unlike correlational methods that only find associations, this system detects the asymmetry inherent in cause-and-effect relationships.
Other capabilities include identifying anomalies by finding data records that are unusually uninformative to their neighbors, generating new data by drawing from probability distributions, and implementing reinforcement learning by selecting actions that maximize rewards based on informative data. It also handles time series data by incorporating lagged values and rates of change, enhancing forecasting and causal analysis.
Practical Implementation and Results
The technology is implemented in the “Howso Engine,” primarily using a custom language called Amalgam. It offers a user-friendly API with verbs like `train` (for adding, editing, removing data), `analyze` (for characterizing data uncertainty and reducing data redundancy), and `react` (for performing inferences). The system has shown competitive performance against established machine learning algorithms like XGBoost and LightGBM in supervised learning tasks. It also excels in anomaly detection, outperforming many benchmarks, and has demonstrated effectiveness in real-time learning from demonstration scenarios, such as navigating drones and cars in simulations. In reinforcement learning, it achieves competitive results on tasks like CartPole and Wafer-Thin-Mints, notably without explicit learning rates, offering rich interpretability back to the training data. Furthermore, it shows promise in causal discovery, data synthesis, data reduction, and adversarial robustness.
Also Read:
- Unlocking Causal Understanding: Energy-Structured Models for Self-Correcting AI
- Causal Deep Q Networks: A New Path to Intelligent Reinforcement Learning
Future Horizons
The researchers envision expanding this framework to unstructured text, treating text sequences as time series to improve context windows and potentially offer human-readable alternatives to embeddings. Applications to image data, such as handwritten digit recognition, are also being explored. Perhaps most intriguingly, the system’s ability to dynamically balance exploration and exploitation in learning opens doors for measuring and governing creativity, defining it as a balance between novelty and value relative to a body of knowledge. For more technical details, you can refer to the full research paper here.
This work represents a significant step towards creating machine learning systems that are not only powerful and scalable but also deeply understandable by humans, offering a new path to discover the fundamental “physics of information.”
