TLDR: This research investigates how different Large Language Model (LLM) configurations and tools (GReaT and REaLTabFormer) impact the quality and computational performance of synthetic tabular data. It finds that while shallower LLMs reduce runtime, REaLTabFormer consistently produces higher quality synthetic data (in terms of ML utility and similarity to real data) across various dataset sizes, especially larger ones, even with lightweight LLMs, offering a better balance despite higher overall runtime compared to GReaT.
Synthetic tabular data is becoming increasingly vital for sharing information while protecting privacy and for developing new machine learning models. However, the effectiveness of this synthetic data heavily relies on the tools used to create it. Recent studies have highlighted that advanced Transformer-based models excel in generating high-quality synthetic data, often outperforming other methods like Generative Adversarial Networks (GANs) and Diffusion models. The challenge, though, is their significant computational cost, which can make them impractical for many users.
A recent study, titled MEASURING LLM SENSITIVITY IN TRANSFORMER-BASED TABULAR DATA SYNTHESIS, delves into this very issue. Researchers Maria F. Davila R., Azizjon Turaev, and Wolfram Wingerath from the Carl von Ossietzky University in Oldenburg, Germany, conducted a sensitivity assessment to understand how different choices in model hyperparameters—such as the number of layers or the hidden dimension within a Large Language Model (LLM)—impact both the quality of the generated synthetic data and the computational performance.
Exploring Key Tools and Metrics
The study focused on two prominent Transformer-based tabular data synthesis (TDS) tools: GReaT and REaLTabFormer. They evaluated ten different model setups, varying in their architectural type and depth, across four real-world datasets. The assessment was structured around three critical dimensions:
- Runtime: The total time taken for training and generating synthetic data.
- Machine Learning (ML) Utility: How well models trained on synthetic data perform compared to those trained on real data.
- Similarity to Real Data Distributions: How closely the synthetic data mimics the statistical properties of the original data.
Key Findings on Performance and Quality
The research revealed several important insights:
Runtime Efficiency: As expected, runtime was directly proportional to the number of hyperparameters, meaning shallower (fewer layers) configurations completed tasks faster. GReaT generally achieved lower runtimes than REaLTabFormer, especially on smaller datasets. However, this runtime advantage narrowed significantly on the largest dataset, where both tools had comparable performance. This suggests GReaT’s efficiency is more pronounced for small to medium-sized datasets.
Machine Learning Utility: For smaller datasets, both GReaT and REaLTabFormer produced synthetic data with high utility, sometimes even outperforming models trained on the original data. This is a crucial benefit for scenarios where real datasets are limited. However, for larger datasets, GReaT’s synthetic data consistently showed a noticeable drop in utility compared to the real data. In contrast, REaLTabFormer managed to sustain strong utility even with larger datasets, indicating its robustness in preserving predictive performance.
Similarity to Real Data: When it came to how closely synthetic data resembled real data, both tools performed optimally for small datasets, generating data that was difficult to distinguish from the original. This ideal behavior continued for REaLTabFormer across all datasets, maintaining high similarity regardless of the LLM configuration or dataset size. GReaT, however, showed a divergence on larger datasets, where its synthetic data became more easily identifiable as artificial, indicating a weaker resemblance to the original distribution.
Also Read:
- TABFAIRGDT: Generating Fair Tabular Data with Efficient Decision Trees
- Boosting Speech Generation Efficiency with Frame-Stacked Local Transformers
Balancing Act: REaLTabFormer’s Advantage
The study concludes that REaLTabFormer, particularly when used with lightweight LLMs, offers the best balance between preserving data quality and reducing computational requirements. While its runtime might still be higher than GReaT and some other TDS tools, it consistently maintains high ML utility and similarity to real data, even with less resource-intensive models. This suggests that while efficiency gains are possible, there’s a certain level beyond which quality might be compromised.
The researchers acknowledge limitations, including the focus on only two Transformer-based tools and a specific set of datasets. Future work will aim to broaden the evaluation to more tools, larger and more diverse datasets, and explicitly assess privacy preservation, which is a core motivation for using synthetic data.
