TLDR: FASTGEN is a novel method for generating synthetic tabular data that uses LLMs to infer data distributions and create reusable sampling scripts, rather than generating records individually. This approach significantly reduces the time and cost of synthetic data generation while producing more diverse and realistic datasets compared to traditional direct LLM-based methods.
Generating large volumes of synthetic tabular data has become increasingly important, especially when real-world data is scarce or expensive to collect. Large Language Models (LLMs) have shown great promise in creating realistic data, but using them to generate each data record individually can be incredibly slow and costly, making it impractical for large-scale needs.
Introducing FASTGEN: A Smarter Approach to Synthetic Data
A new method called FASTGEN (Fast Automated Synthetic Tabular GENeration) offers a solution to this challenge. Instead of having an LLM create every single data record, FASTGEN leverages the LLM to understand the underlying patterns and distributions of the data fields. Once these patterns are identified, the LLM generates reusable ‘sampling scripts’ (for example, in Python) that can then efficiently produce vast amounts of diverse and realistic synthetic data without needing continuous LLM inference.
This innovative approach significantly reduces the time and cost associated with synthetic data generation. FASTGEN automatically categorizes data fields into three main types: numerical (like age or amount), categorical (like gender or product type), and free-text (like descriptions or comments). For each type, the LLM infers the most suitable distribution and generates a specific script. For numerical fields, it might estimate a normal or uniform distribution; for categorical, it identifies frequent categories and their probabilities; and for free-text, it generates random text aligned with the field’s metadata.
Key Benefits and Performance
Experiments comparing FASTGEN with traditional direct LLM generation (direct-gen) show impressive results. FASTGEN consistently produces data with richer vocabulary and greater diversity, meaning the generated data is less repetitive and more varied. In terms of realism, FASTGEN’s numerical and categorical data distributions are generally closer to real-world data than those generated by direct methods.
The most striking advantage of FASTGEN is its efficiency. It requires significantly fewer tokens from the LLM to produce the same amount of data. For instance, generating 10,000 records with FASTGEN might require 60 times fewer tokens than direct-gen. This translates into massive savings in both time and computational cost. For example, generating 100,000 records for a dataset could take direct-gen 40 hours and cost $55, while FASTGEN could do it in just 0.07 hours for $0.096.
Also Read:
- Synthetic Emotions: How AI is Creating Diverse Text for Emotion Recognition
- VerilogDB: A New Standard for Training AI in Hardware Design
Future Directions
While FASTGEN marks a significant leap forward, the researchers acknowledge areas for further improvement, such as enhancing the LLM’s ability to capture complex relationships between different data fields. However, the current methodology is already being planned for use at Trillion Technology Solutions to accelerate testing in production pipelines, thereby shortening development cycles and improving overall system efficiency.
For more technical details, you can read the full research paper here.
