TLDR: This paper introduces a system for text-to-table generation using Large Language Models (LLMs) that overcomes common challenges. It proposes two main strategies: breaking down the complex task into guided sub-tasks and iteratively refining the generated tables using fine-grained, cell-level self-feedback. This approach significantly improves accuracy on complex datasets without costly fine-tuning, demonstrating the power of structured prompting and targeted self-correction for LLMs in generating structured data.
Transforming unstructured text into organized, structured tables is a crucial task in today’s data-driven world. Imagine taking a long news article or a sports game summary and automatically converting key information into a clear table. While Large Language Models (LLMs) show great promise in this area, they often face significant hurdles. These challenges include accurately handling ambiguous or specialized data, maintaining the correct table structure, managing very long input texts, and performing precise numerical reasoning.
A recent research paper, “LLM driven Text-to-Table Generation through Sub-Tasks Guidance and Iterative Refinement”, introduces an innovative system designed to make LLMs much more effective at text-to-table generation. Authored by Rajmohan C and Arvind Agarwal from IBM Research, and Sarthak Harne from IIIT-Bangalore, this paper proposes a two-pronged approach: breaking down the complex task into smaller, guided sub-tasks and then refining the generated tables through iterative self-feedback.
Breaking Down the Task: Sub-Tasks Guidance
The first core strategy involves guiding LLMs through a series of predefined sub-tasks. This decomposition helps the LLM better understand both the input data and the overall goal, leading to clearer reasoning and improved performance. The sub-tasks include:
- Header Explanation: The LLM first interprets and explains the meaning of column headers in the context of the given text.
- Abbreviation Expansion: It expands any abbreviations found in the passage to ensure full comprehension and prevent misinterpretation.
- Data Format Resolution: The model resolves custom or non-standard data formats, like hyphenated scores, to ensure consistency in the final table.
- Entity Extraction and Grouping: This crucial step involves identifying all row header entities (e.g., player names, team names) and grouping all related data for each entity. This effectively summarizes what will become individual rows in the table.
- Table Generation: Finally, the LLM generates the table in a structured, machine-readable, and human-interpretable format.
Refining the Output: Iterative Self-Feedback
The second key strategy is to iteratively refine the generated table using the LLM’s own self-feedback. After an initial table is created, the LLM is prompted to review and critique its own output based on several criteria: Data Completeness (is all relevant info included?), Data Relevance (is only grounded info present, no hallucinations?), Table Structure Consistency (does it follow the expected format?), and Domain-Specific Feedback. The researchers explored different levels of feedback:
- Table-level feedback: The LLM reviews the entire table. This was found to be less effective, as LLMs struggled to critique their whole output accurately.
- Row-level feedback: The LLM provides feedback for one row at a time. This showed some improvement.
- Cell-level feedback: The LLM focuses on one cell at a time, verifying its accuracy. This proved to be the most effective method, especially for tasks requiring complex calculations or precise data extraction, as it allowed for more accurate corrections.
While iterative cell-level feedback significantly boosts accuracy, it’s important to note the trade-off with computational cost. Each iteration and each cell-level check requires additional LLM invocations, which can increase expenses. However, for applications where high accuracy is paramount, this refinement is highly recommended.
Also Read:
- LyS System: Zero-Shot Code Generation for Answering Questions from Tables
- Streamlining Database Interaction: An End-to-End Text-to-SQL Framework with Automated Database Selection
Performance and Impact
The system was evaluated using Llama-3-70B-Instruct, a powerful open-license LLM, on two complex text-to-table datasets: RotoWire (NBA game summaries) and LiveSum (football game commentaries). The results demonstrated that the intermediate sub-tasks guidance significantly enhanced the LLM’s reasoning capabilities, leading to better performance compared to standard prompting methods. Furthermore, the fine-grained cell-level feedback, particularly over two iterations, achieved state-of-the-art performance on these datasets.
This research highlights that by providing structured guidance and enabling targeted self-correction, LLMs can overcome their inherent limitations in generating complex structured data. This approach offers a powerful alternative to resource-intensive fine-tuning, making high-quality text-to-table generation more accessible and efficient for various real-world applications.
