TLDR: This research paper investigates linear dimensionality reduction techniques for word embeddings used in tabular data classification, specifically for an Engineers’ Salary Prediction Challenge. The challenge involves classifying salary categories from high-dimensional tabular data, where job descriptions are represented as 300-dimensional word embeddings, and training samples are limited. The study explores Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA). It finds that PCA with appropriate subspace dimensions can outperform raw embeddings. LDA initially performs poorly due to covariance estimation errors but significantly improves with shrinkage. The paper proposes a novel method, Partitioned-LDA, which splits embeddings into blocks and performs LDA on each, reducing covariance matrix size. Partitioned-LDA, especially when combined with shrinkage, achieves top-10 accuracy on the competition’s public leaderboard, demonstrating its effectiveness in enhancing word embedding performance for tabular data classification with limited training samples.
The world of data science often presents unique challenges, especially when combining different types of information. A recent study delves into one such complex scenario: predicting engineers’ salaries using tabular data that includes high-dimensional word embeddings from job descriptions. This research, titled “Linear Dimensionality Reduction for Word Embeddings in Tabular Data Classification,” addresses the difficulties posed by high dimensionality and limited training samples in such datasets.
The Engineers’ Salary Prediction Challenge, which inspired this research, aims to classify salary categories (low, medium, or high) based on various tabular features, including job title, state, and a vectorized job description. The job description, initially a 300-dimensional word embedding, significantly increases the dataset’s complexity, transforming 15 initial features into 315. This high dimensionality, coupled with a limited number of training samples, makes accurate classification a considerable hurdle.
The paper explores the application of linear dimensionality reduction techniques, specifically Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA), to these word embeddings. The goal is to see how reducing the dimensions of these embeddings impacts the overall performance of downstream classification tasks.
Exploring Traditional Methods
Initially, the researchers investigated PCA, a method that projects data onto a lower-dimensional subspace while maximizing variance. They found that PCA, when applied with an appropriate subspace dimension, could actually outperform using the raw, high-dimensional embeddings. This suggests that while word embeddings are inherently informative, reducing noise by focusing on key principal components can be beneficial.
Linear Discriminant Analysis (LDA) was also examined. Unlike PCA, LDA incorporates class label information, aiming to maximize the separation between different classes. However, without regularization, LDA performed poorly. This was attributed to issues with covariance estimation, which becomes numerically ill-conditioned with high-dimensional data and few training samples. To address this, a technique called shrinkage was applied. Shrinkage adjusts extreme values in the covariance matrix estimation, significantly improving LDA’s performance. With shrinkage, LDA, even when reducing dimensions to just two, showed a notable improvement, surpassing the baseline accuracy by 1%.
Introducing Partitioned-LDA
To further tackle the challenges of high-dimensional covariance estimation in LDA, the researchers proposed a novel method called Partitioned-LDA. This approach involves splitting the 300-dimensional job description embeddings into several equal-sized, non-overlapping blocks. LDA is then performed separately on each block. This strategy effectively reduces the size of the covariance matrices that need to be estimated, thereby improving the robustness of the estimation process.
The advantage of Partitioned-LDA is that it enhances the ratio of training samples to features within each block, leading to more reliable covariance estimations. While this method focuses on local patterns within the blocks, potentially losing some global interactions, selecting the right number of blocks is crucial for optimal performance. The study found that an optimal configuration for Partitioned-LDA significantly improved accuracy.
Also Read:
- Shaping Feature Spaces: A New Loss Function for Machine Learning Topology Control
- HARP: Unveiling LLM Hallucinations Through Reasoning Subspace Projection
Key Findings and Impact
The experiments, evaluated on a public leaderboard using a Gradient Boosting Decision Tree (HGBClassifier), demonstrated the effectiveness of these dimensionality reduction techniques. While the Postprocessing Algorithm (PPA) decreased accuracy, and raw LDA performed poorly, PCA showed a slight improvement over the baseline. LDA with shrinkage, however, improved the baseline accuracy by 1% while drastically reducing dimensionality to just two dimensions.
The most significant breakthrough came with Partitioned-LDA. It initially performed better than raw LDA, and when combined with shrinkage, it achieved an impressive accuracy of 77.04%. This result placed the method in the top 10 on the competition’s public leaderboard, highlighting its potential to enhance word embedding performance in tabular data classification, especially with limited training samples.
In conclusion, this research demonstrates that carefully applied linear dimensionality reduction can significantly improve the performance of word embeddings in tabular data classification. The proposed Partitioned-LDA, particularly when combined with shrinkage, offers a robust solution to the challenges of high dimensionality and sparse data, making it a valuable tool for similar predictive tasks. For more details, you can read the full paper here.
