TLDR: A new loss function, Hopkins loss, is introduced to actively control and modify the organization of samples in machine learning feature spaces. Unlike existing methods that preserve topology, Hopkins loss uses the Hopkins statistic to enforce desired structures (regularly-spaced, randomly-spaced, or clustered). Experiments on speech, text, and image data show it effectively modifies feature topology with minimal impact on classification performance, and a more significant impact in dimensionality reduction, offering a valuable tool for various ML applications.
In the realm of machine learning, the way data samples are organized within a feature space, known as feature space topology, plays a crucial role. Traditionally, methods have focused on preserving the existing topology of input features. However, a groundbreaking new paper introduces a novel approach: actively controlling and modifying this topology to achieve desired structures. This innovative method, detailed in the paper “Feature Space Topology Control via Hopkins Loss,” presents a new loss function called Hopkins loss, which leverages the Hopkins statistic to shape feature spaces.
The authors, Einari Vaaras from Tampere University and Manu Airaksinen from the University of Helsinki, highlight several potential benefits of modifying feature space topology. These include improving generalization in machine learning models, enhancing dimensionality reduction for visualization and data compression, controlling feature distributions in generative models, aiding bioinformatics applications, facilitating transfer learning by aligning feature spaces, and increasing robustness against adversarial attacks.
Understanding Hopkins Loss
At the heart of this new method is the Hopkins statistic (H), a statistical test used to measure the clustering tendency of a dataset. Values of H range from 0 to 1, where values near 0.5 suggest randomly-spaced data, values between 0.01 and 0.3 indicate regularly-spaced data, and values from 0.7 to 0.99 point to clustered data. The Hopkins loss function, L_H, is defined as the absolute difference between the calculated Hopkins statistic and a pre-defined target value (H_T). By minimizing this loss during model training, a neural network can be guided to transform the feature topology of input features towards the desired structure, whether that’s regularly-spaced, randomly-spaced, or clustered.
The researchers found that using Chebyshev distance as the distance metric within the Hopkins statistic computation was most effective across various data dimensions. They also optimized the sampling process for calculating H, ensuring robust and consistent results.
Experimental Validation
To evaluate the effectiveness of Hopkins loss, experiments were conducted across diverse data types: speech (Ryerson Audio-Visual Database of Emotional Speech and Song – RAVDESS), text (IMDB movie review dataset), and images (Fashion-MNIST). Two primary scenarios were explored: classification and dimensionality reduction using autoencoders.
Classification Experiments
In classification tasks, Hopkins loss was integrated with categorical cross-entropy loss. The results showed that for speech and image data, incorporating Hopkins loss generally did not degrade model performance, while successfully modifying features towards the desired topology. For text data, adding Hopkins loss even outperformed the baseline for all targeted feature topologies. On average, the inclusion of Hopkins loss in classification experiments shifted the H value towards the target by approximately 0.09 for speech, 0.11 for text, and 0.12 for image data.
Autoencoder Experiments
For dimensionality reduction using autoencoders, Hopkins loss was combined with mean squared error loss. While the use of Hopkins loss often resulted in slightly lower classification performance compared to the baseline, there were instances where performance was similar or even better. Crucially, the modification in feature space topology was significantly more pronounced in these experiments. On average, Hopkins loss shifted the H value towards the target by about 0.19 for speech, 0.18 for text, and 0.22 for image data. This demonstrates its utility in applications where feature topology is prioritized over a minor drop in classification accuracy.
Interestingly, baseline models in both classification and autoencoder experiments exhibited a natural bias towards a clustered topology, suggesting that original features often tend to be clustered. While Hopkins loss could adjust this clustered topology towards being more regularly spaced, achieving a truly regularly-spaced topology (H_T = 0.01) proved more challenging, possibly indicating an inherent difficulty in uniformly structuring features.
Computational Overhead
The computational cost of integrating Hopkins loss was found to be modest, introducing an average increase in epoch duration of about 10-13% across all experimental conditions. The overhead was marginal for smaller datasets like RAVDESS but more noticeable for larger datasets, indicating that the cost scales with dataset size and feature dimensionality.
Also Read:
- AI Models Learn to Adapt by Deceiving Distribution Shift Detectors
- Guiding Acoustic Scene Classification with Entropy for Better Generalization
Conclusion and Future Directions
This research introduces a powerful new tool for machine learning practitioners. Hopkins loss provides a mechanism to actively shape the organization of samples in feature space, offering benefits across various applications from improving model generalization to enhancing data visualization. While the current study used simple MLPs, future work could explore its effects with more complex models, apply it to all neural network layers simultaneously, and investigate additional distance metrics and use cases beyond classification and autoencoders. The code for Hopkins loss is freely available at https://github.com/SPEECHCOG/hopkins_loss. This paper has been accepted for publication in Proc. IEEE ICTAI 2025, Athens, Greece. You can find the full research paper here: https://arxiv.org/pdf/2509.11154.
