TLDR: GEM-T is a novel generative model for synthetic tabular data based on the principle of maximum entropy. It captures n-th order interactions among data columns and, through extensive testing, matches or exceeds state-of-the-art deep neural network approaches on 23 of 34 benchmark datasets. GEM-T uses significantly fewer parameters, handles heterogeneous data types effectively, and maintains strong privacy, demonstrating a promising direction for lightweight, high-performance generative models for structured data.
In the rapidly evolving landscape of artificial intelligence, generative models have made significant strides in areas like text and images. However, generating synthetic tabular data—the kind of structured information found in spreadsheets and databases across industries like medicine, finance, and agriculture—has remained a persistent challenge. This is particularly true when dealing with limited or sensitive datasets, where privacy and data availability are crucial concerns.
Traditional deep neural network (DNN) approaches, while powerful, often require vast amounts of training data and carry the risk of overfitting, potentially compromising privacy by memorizing original data points. Other methods like copulas can struggle with high-dimensional data or complex tail behaviors. Addressing these limitations, a new research paper introduces a novel approach called GEM-T, short for “generative entropy maximization for tables.”
Authored by Miao Li, Phuc Nguyen, Christopher Tam, Alexandra Morgan, Kenneth Ge, Rahul Bansal, Linzi Yu, Rima Arnaout, and Ramy Arnaout, the paper details GEM-T as a lightweight, high-performance generative model. It operates on the principle of maximum entropy (MaxEnt), which essentially means it constructs the least-biased model possible given a specific set of known statistical properties, or “moments,” of the training data. This approach allows GEM-T to directly capture complex interactions—from simple pairwise relationships to higher-order dependencies—among the columns of a dataset.
The GEM-T process involves four key steps. First, the raw tabular data undergoes careful preprocessing, which includes integer-encoding categorical columns, dropping uninformative columns, normalizing data using a quantile transformation with a clever ‘jittering’ technique to handle identical values, and scaling to ensure a mean of zero. This preprocessing step is crucial for transforming diverse data types into a more manageable format for the model.
Next, GEM-T fits a probability distribution to these transformed features. It can perform a quick, analytical second-order fit (capturing means and variances) or a more precise, iterative higher-order fit (up to fourth-order, including skewness and kurtosis) using gradient descent. The model then intelligently selects the better of these two fits. After fitting, synthetic samples are drawn from the learned distribution. Finally, an inverse transformation is applied to these samples, converting them back into the original data space, and any necessary constraints (like rounding to integers) are enforced.
The results of extensive testing are compelling. GEM-T was benchmarked against state-of-the-art deep neural network models like CTGAN and TabularARGN (TARGN) across 34 publicly available datasets from diverse domains. GEM-T either matched or surpassed these advanced models in 23 of these datasets (68%). Notably, GEM-T achieves this superior performance with orders-of-magnitude fewer trainable parameters, suggesting that much of the valuable information in real-world data lies in simpler, low-dimensional correlations, especially when the input data is appropriately prepared.
One of GEM-T’s significant strengths is its ability to handle heterogeneous data types—continuous, discrete, and categorical—and the inherent lack of local structure often found in tabular data. The paper highlights how GEM-T’s higher-order moments effectively capture challenging data features like long tails and hard edges in distributions, which other models struggle with. Furthermore, GEM-T demonstrates strong privacy preservation, with its synthetic data being statistically indistinguishable from a random half of the original training data, as measured by the distance-to-closest-record (DCR) metric.
The research also points out that GEM-T performs consistently well across both small and large datasets, a crucial advantage for scenarios where data is scarce. This contrasts with some deep learning models that tend to struggle with smaller datasets. The paper acknowledges that while GEM-T excels in statistical fidelity, future work will explore its performance in specific downstream tasks like classification and regression.
Also Read:
- Latent Zoning Network: A Unified Approach to Generative AI, Representation Learning, and Classification
- Generative AI Enhances Wireless Sensing Capabilities
In conclusion, GEM-T represents a significant advancement in generative modeling for tabular data. By leveraging the elegance of maximum entropy and incorporating thoughtful preprocessing, it offers a robust, efficient, and privacy-preserving solution for creating high-quality synthetic tabular data. This work underscores the power of moment-driven, energy-based models in preventing the memorization of training data while accurately reflecting its underlying structure. You can read the full research paper here.
