TLDR: The Robust Regression Transformer (R2T) is a new hybrid neural-symbolic AI model designed to accurately extract physiological parameters from noisy and irregular wearable device data. Unlike traditional methods that struggle with non-Gaussian noise, R2T uses a transformer encoder and a symbolic decoder, trained on synthetic corrupted data, to robustly recover underlying symbolic parameters. It significantly outperforms conventional least-squares and robust regression techniques, offering 10-300 times improvement in accuracy for temperature, heart rate, and SpO2 signals, making it highly effective for real-world wearable data analysis.
Wearable devices like smartwatches and rings are becoming increasingly popular for monitoring health and fitness. However, the data collected from these devices is often far from perfect. It can be porous, noisy, and irregular, containing random drops, spikes, and other motion artifacts. Traditional methods for analyzing this data, which rely on least-squares optimization, struggle significantly when faced with such ‘non-Gaussian’ or asymmetric structured noise. This limitation often forces researchers to analyze data only from sleep periods or average data over entire days, losing valuable, actionable insights.
A new approach, the Robust Regression Transformer (R2T), aims to overcome these challenges. Developed by Roman Gutierrez, Tony Kai Tang, and Isabel Gutierrez from Vistendo Inc., R2T is a novel hybrid neural-symbolic architecture designed for robust regression. Unlike traditional symbolic regression that tries to infer both the structure and constants of equations, R2T focuses on accurately recovering symbolic parameters for known physiological models (like the sinusoidal function for circadian rhythm) even when the data is heavily corrupted by real-world wearable noise. You can read the full research paper, Beyond Least Squares: Robust Regression Transformer (R2T), for more technical details.
How R2T Works
The R2T model utilizes a transformer encoder, a type of neural network known for processing sequences. This encoder processes numerical physiological signal sequences (like heart rate, temperature, and blood oxygen saturation) along with steps data. The output of the transformer encoder is then highly compressed by a ‘Lossy Compression NN’ into a small set of symbolic parameters, typically nine constants. These constants have direct physiological interpretations and can serve as biomarkers for health and wellness assessments.
A crucial component is the symbolic decoder. This decoder uses fixed symbolic equations that describe how the human body’s physiological signals respond to factors like time and physical activity (steps). For instance, a specific equation is used for heart rate, incorporating resting heart rate, circadian rhythm amplitude and phase, and sensitivity to steps (both immediate and delayed). The steps sequence, which indicates physical activity, is fed directly into the symbolic decoder to help reconcile changes in physiological signals during movement, allowing the transformer to focus solely on predicting the symbolic parameters.
Training with Synthetic Data
One of R2T’s innovative aspects is its training methodology. The model is trained using synthetic time-series data that is intentionally corrupted with realistic wearable-device noise, such as patterned disruptions and spikes. The training objective is to recover the original, clean sequence and its underlying symbolic parameters. This unsupervised training approach is highly advantageous because it provides noiseless targets, enables robust regression regardless of noise structure, allows for easy generation of vast amounts of training data, and can recursively improve performance on real-world data once deployed.
During training, the input sequences are averaged into 15-minute intervals, normalized, and then masked (10% of values are randomly set to 0.0, the same value used for missing data). This masking encourages the model to learn to fill in missing information. The training loss is a weighted combination of the mean squared error (MSE) for sequence reconstruction (temperature, heart rate, SpO2) and the MSE for symbolic parameter recovery (circadian phase, maximum temperature, resting heart rate).
Performance and Results
Experiments demonstrate that R2T consistently outperforms traditional robust regression techniques like Ordinary Least Squares (OLS), Huber loss, and SoftL1, especially when dealing with high corruption and asymmetric noise. For example, in synthetic wearable data, R2T achieved a median regression MSE of 6e-6 to 3.5e-5, representing a 10-300 times improvement compared to OLS and other robust regression methods.
When noise is small and exclusively Gaussian, R2T performs similarly to least squares. However, as soon as the magnitude of noise increases or anomalies are introduced, OLS fits deviate significantly, while R2T continues to generate valid regressions. For heart rate sequences with anomalies, R2T accurately fits spikes corresponding to steps, while OLS is significantly pushed off. Similarly, for SpO2 sequences with large asymmetric noise, R2T’s predictions are highly accurate, whereas OLS predictions are substantially off.
The errors in R2T’s fit and symbolic parameter predictions are within acceptable measurement error for wearable devices, even in worst-case scenarios. In contrast, OLS often fails to produce a good fit for sequences with large noise or asymmetric anomalies, leading to errors far beyond acceptable limits for wearable device measurements.
Also Read:
- PRISM: A Resource-Efficient Method for Analyzing Complex Time Data
- PrivCLIP: A New Approach to User-Controlled Privacy in Sensor Data
Conclusion
The Robust Regression Transformer (R2T) represents a significant advancement in analyzing noisy and porous non-linear functions, particularly for wearable sensor data. By leveraging a hybrid neural-symbolic architecture and training on synthetically corrupted data, R2T provides a reliable method for extracting useful physiological information that traditional least-squares-based methods cannot. This breakthrough has myriad potential applications in data analysis wherever least-squares fit is currently applied, paving the way for more precise and actionable insights from continuous physiological monitoring.
