TLDR: StructText is a new framework that automatically generates high-quality benchmarks for extracting structured information (key-value pairs) from text. It starts with tabular data, uses LLMs to create synthetic natural language text, and then evaluates this text using multi-dimensional criteria (factuality, hallucination, coherence, numeric/temporal accuracy). The research found that while LLMs generate factually accurate text, they struggle with narrative coherence and making information easily extractable by machines, highlighting a critical area for future research in text-to-table conversion.
In the rapidly evolving landscape of artificial intelligence, Large Language Models (LLMs) are transforming how we interact with information. From generating human-like text to automating complex data tasks, their capabilities are vast. However, a significant challenge remains: how do we accurately measure their performance, especially when it comes to extracting structured information like key-value pairs from natural language text?
Building robust benchmarks for evaluating these extraction capabilities is crucial, but traditionally, this has been a labor-intensive process involving manual annotations, which limits the scale and diversity of available datasets. This is where a new framework called StructText steps in, offering an innovative solution to automatically generate high-quality benchmarks.
Introducing StructText: A New Approach to Benchmarking
Developed by researchers Satyananda Kashyap, Sola Shirai, Nandana Mihindukulasooriya, and Horst Samulowitz from IBM Research, StructText is an end-to-end system designed to create synthetic benchmarks for key-value extraction. It leverages existing tabular data as a source of ground truth and employs a clever two-stage “plan-then-execute” pipeline to generate corresponding natural language text.
The core idea is to start with structured data (like a spreadsheet) and then use LLMs to generate text that accurately reflects that data. This generated text then serves as a benchmark for evaluating how well other LLMs can extract the original structured information back out. To ensure the quality and accuracy of this synthetic text, StructText introduces a multi-dimensional evaluation strategy.
How StructText Works: From Table to Text and Back
The process begins with “Synthetic Text Generation.” In the “planning stage,” an LLM analyzes a sample of rows from an input table. It identifies natural groupings of related information, such as financial metrics or geographic data, and determines how these should be presented in a narrative. This is more than just filling in templates; the LLM intelligently recognizes relationships within the data.
Once the plan is set, the “generation stage” kicks in. The LLM produces coherent narratives for each row, strictly adhering to the identified columns. This design helps prevent the model from “hallucinating” or inventing information not present in the original data, ensuring factual accuracy.
After the text is generated, StructText moves to “Multi-Dimensional Quality Assessment.” This is where the framework truly shines. It uses LLMs themselves as “judges” to evaluate the generated text across three critical dimensions:
- Factuality: Is the information in the text correct and traceable to the source data?
- Hallucination: Does the text contain any invented details not present in the original table?
- Coherence: Does the text flow naturally and logically?
Beyond LLM-based judgments, StructText also employs objective metrics for “Numerical and Temporal Accuracy.” This ensures that numbers, dates, and other specific values are preserved correctly in the generated text, which is vital for many real-world applications like financial reporting.
Finally, a “Filtering Low Quality Text” step allows for the removal of any generated reports that don’t meet high-quality standards, ensuring that the final benchmark dataset is reliable.
Evaluating Extraction: The Baseline Approach
To demonstrate the utility of the generated benchmarks, the researchers also implemented a “Text-to-Table Extraction Baseline.” This baseline system takes the synthetically generated natural language text and attempts to convert it back into a tabular format. It first identifies the likely columns that can be extracted and then proceeds to extract the corresponding values.
The evaluation of this extraction process considers not just exact matches for column names but also semantic similarity, allowing for minor variations. This comprehensive evaluation helps assess how challenging the generated text is for LLMs to parse back into structured data.
Also Read:
- Enhancing AI Reliability: A Framework for Smarter Information Handling in Language Models
- Evaluating Retrieval-Augmented Generation in Diverse Knowledge Environments
Key Findings and Future Implications
The evaluation of StructText across diverse datasets, including SEC financial filings and WikiDB tables, yielded fascinating insights. While LLMs demonstrated exceptional performance in maintaining factual accuracy and avoiding hallucinations in the generated text, they struggled more with narrative coherence. This suggests that while models can produce correct information, organizing it into a naturally flowing and easily extractable narrative remains a challenge.
Furthermore, despite high accuracy in preserving numerical and temporal details in the generated text, the baseline extraction system showed significant room for improvement in extracting these key-value pairs back out. This indicates that even when facts are present, they can be embedded in ways that make automated extraction difficult. This highlights a critical gap: models can generate accurate text, but maintaining information accessibility for machine processing is a key requirement for practical deployment.
StructText provides a valuable open-source framework, including datasets, evaluation tools, and baseline extraction systems, to support continued research in this area. Its extensibility means it can be applied to any new tabular dataset, greatly expanding the scope of benchmark creation for text-to-table tasks. This work is a significant step towards developing LLMs that not only generate accurate text but also ensure that the information within is readily usable for downstream applications. You can find more details about this research paper here.
