TLDR: A new method called eMargin introduces an adaptive margin to contrastive learning for time series data. While it significantly improves unsupervised clustering metrics by creating tightly separated clusters, it surprisingly performs poorly in downstream classification tasks. This highlights a critical disconnect between clustering quality and the practical utility of learned representations for specific applications, suggesting that unsupervised metrics alone may not be sufficient for evaluating representation learning models.
Self-supervised learning (SSL) is a powerful approach that leverages the inherent structure of unlabeled data to create rich feature representations. Among SSL techniques, contrastive learning (CL) is prominent, aiming to pull similar data samples closer together in a representation space while pushing dissimilar ones apart. This is typically achieved using a contrastive loss function like InfoNCE.
While effective, applying contrastive learning to time series data presents unique challenges. Unlike images, where simple augmentations like cropping don’t change meaning, time series data is highly dependent on structure and order. Naively contrasting adjacent timestamps can lead to suboptimal representations that overfit to local continuity rather than capturing high-level structures.
A new research paper, titled “eMargin: Revisiting Contrastive Learning with Margin-Based Separation,” explores the introduction of an adaptive margin, called eMargin, into the contrastive loss function specifically for time series representation learning. The authors, Abdul-Kazeem Shambaa, Kerstin Bacha, and Gavin Taylor, investigated whether this adaptive margin, adjusted based on a predefined similarity threshold, could improve the separation between adjacent but dissimilar time steps and lead to better performance in subsequent tasks.
The core idea behind eMargin is to dynamically adjust the separation of features in the representation space. It does this by defining a “pseudo-label” based on the similarity of consecutive time steps in the raw data. If two adjacent time steps are highly similar (above a threshold), they are treated as strongly correlated, and no margin adjustment is applied. However, if they are dissimilar, a penalty is introduced to enforce a minimum separation, thereby increasing their distinction. This modification reshapes the similarity distribution before the final loss calculation, aiming to prevent trivial solutions that rely on local smoothness.
The researchers evaluated eMargin on three real-world time series datasets: HARTH, ECG, and SLEEP EEG. They used a straightforward convolutional neural network (CNN) as the feature extractor and compared eMargin’s performance against eight state-of-the-art baselines.
Key Findings: A Surprising Disconnect
The study yielded intriguing results, highlighting a critical disconnect between different evaluation metrics. On unsupervised clustering metrics, such as the Davies-Bouldin Index (DBI) and Silhouette Score (SS), eMargin consistently outperformed all baselines. A lower DBI indicates better separation between clusters, and eMargin achieved the lowest values across all datasets. Similarly, higher SS values reflect better compactness and separation, and eMargin showed significant improvements, for instance, achieving a DBI of 0.59 and a Silhouette score of 0.80 on the ECG dataset, outperforming all competitors.
This strong performance in clustering suggested that eMargin was indeed creating more compact and well-separated clusters in the embedding space. However, when the learned representations were assessed on downstream classification tasks using a linear classifier (a common way to evaluate the utility of learned features), eMargin struggled significantly. Across all three datasets, InfoNCE + eMargin showed consistently lower or middling scores on accuracy, F1 score, precision, and recall compared to many baselines. For example, on the SleepEeg dataset, its precision and recall were among the lowest, despite its excellent clustering scores.
To understand this discrepancy, the researchers visualized the learned representations using t-SNE plots. These visualizations revealed that eMargin produced distinct, tight, spiral-like structures that were visually well-separated but internally continuous and elongated. While this geometric arrangement contributed to high unsupervised clustering scores, it did not necessarily correspond to meaningful class separability for classification tasks. In contrast, other methods like TS2Vec and TNC, which performed better in classification, exhibited more discrete and semantically coherent clusters aligned with the ground-truth class labels.
Also Read:
- Extending Evidential C-Means for Complex Data with Soft-ECM
- Boosting Phenotype Prediction: Self-Supervised Learning for Gene Expression
Implications for Representation Learning
The findings from this research underscore a crucial point: improvements in unsupervised clustering metrics do not automatically translate to better performance in supervised downstream tasks. While eMargin effectively promotes geometric compactness and separation in the embedding space, these clusters may not align with the actual class-relevant decision boundaries needed for classification. This suggests that unsupervised clustering scores, while useful, can sometimes be misleading and may not accurately predict the utility of learned representations for specific applications.
The paper advocates for a more critical perspective on unsupervised evaluation protocols, emphasizing the need to assess representation quality not only through unsupervised structure but also through its effectiveness in downstream tasks. Future work, as suggested by the authors, should focus on explicitly aligning these learned structures with task-relevant semantics, potentially through semi-supervised objectives or improved inductive biases. You can read the full paper for more details here: eMargin: Revisiting Contrastive Learning with Margin-Based Separation.
