TLDR: TEN is a new neurosymbolic system that extracts tables from messy text by combining a large language model (LLM) with a symbolic checker. The LLM generates an initial table, which the checker validates for errors like hallucinations. A “critique LLM” then provides actionable feedback to the original LLM, allowing it to refine the table iteratively. This approach significantly outperforms purely neural methods, producing more accurate tables that users prefer, especially from challenging, unstructured inputs.
Extracting structured information, such as tables, from unstructured or semi-structured text remains a significant challenge in the world of artificial intelligence. While large language models (LLMs) have made incredible strides in understanding and generating human-like text, they often struggle with the nuances of tabular data embedded within messy documents like PDFs, emails, or reports. This difficulty arises because the original formatting, clear delimiters, and cell structures are frequently lost during processes like copy-pasting or optical character recognition (OCR).
Traditional methods for table extraction often fall into two categories: symbolic (rule-based) or purely neural (deep learning). Symbolic methods are precise and interpretable but rigid, failing when faced with irregular or unexpected formats. Neural models, while flexible, are prone to ‘hallucinations’ – generating plausible but incorrect data or structures – and lack the ability to enforce strict logical constraints. This leads to unreliable outputs that require extensive manual correction, a process known to have high error rates (ranging from 1% to 5%, and even up to 50% for complex documents), costing businesses significant time and resources.
To address these limitations, researchers have developed a novel neurosymbolic approach called TEN: Table Explicitization, Neurosymbolically. TEN combines the powerful generative capabilities of LLMs with the rigorous validation of symbolic logic, creating a robust system for extracting tabular data from even the most challenging text inputs. You can read the full research paper here.
How TEN Works: A Self-Correcting Loop
The core of TEN is an iterative, self-debugging feedback loop. It begins with a Large Language Model, referred to as the Table Generator LLM, which takes the unstructured input text and attempts to create an initial table. This LLM uses a specialized technique called “Structural Decomposition prompting.” Unlike generic prompting, this method guides the LLM to first identify table-like regions and potential row delimiters within the text before attempting to structure the data. This mimics how a human might approach the task, breaking down the problem into logical steps.
Once the initial table is generated, it’s passed to a “Symbolic Sanity Checker.” This component acts as a strict validator, applying a set of hand-crafted, domain-agnostic rules to assess the table’s quality. It checks for common extraction errors such as merged cells, inconsistent data types within columns, incorrect delimiters, mismatched parentheses, and empty rows. The checker doesn’t just give a pass/fail; it provides detailed feedback, including measures of how much of the original text is covered, the rate of hallucinated content, and scores indicating the table’s structural goodness or badness.
The output of the Symbolic Sanity Checker, along with the initial table, is then sent to a secondary “Critique LLM.” This LLM is specifically designed to interpret the symbolic feedback contextually. It analyzes the identified issues and translates them into precise, actionable natural language guidance for fixing the table. This critique is then fed back to the original Table Generator LLM, prompting it to refine and regenerate the table. This iterative process continues until the table meets predefined quality criteria or a maximum number of refinement attempts is reached.
Key Advantages and Performance
Extensive experiments demonstrate that TEN significantly outperforms purely neural baselines across various datasets and metrics. The “Structural Decomposition prompting” strategy consistently yields better results than generic or simple chain-of-thought prompting, showing improvements of 10-40 percentage points in exact match accuracy. This specialized prompting particularly benefits smaller LLMs, effectively narrowing the performance gap between them and larger, more capable models.
TEN also shows superior performance against existing neurosymbolic and symbolic table reconstruction methods like Tabularis Revilio and SplitText, especially in content-fidelity metrics (Exact Match, Cell Value Match, and Column Value Match). While some structural metrics might show mixed results due to TEN’s ability to normalize layouts in ways that differ from a reference, its core strength lies in accurately preserving and reconstructing the actual data content.
A user study involving 21 spreadsheet users further validated TEN’s practical utility. Participants rated tables generated by TEN as significantly more accurate and consistently preferred them for ease of verification and correction. This indicates that TEN not only produces technically superior tables but also tables that are more usable and require less manual effort to fix in real-world scenarios.
Also Read:
- Enhancing LLM Table Creation: Sub-Tasks and Self-Correction
- LyS System: Zero-Shot Code Generation for Answering Questions from Tables
The Impact of Each Component
An ablation study, which involved testing TEN with different components removed, confirmed that both the symbolic feedback and the critique LLM are crucial for the system’s success. While symbolic feedback alone improves structural and content metrics, and LLM feedback alone can provide significant gains on simpler datasets, the full TEN system, combining both, achieves the best overall performance. This integrated approach ensures that feedback is both targeted (from symbolic rules) and contextualized (from the critique LLM), leading to robust and high-fidelity table outputs.
In conclusion, TEN represents a significant step forward in automated table extraction. By intelligently combining the flexibility of large language models with the precision of symbolic validation in a self-correcting loop, it offers a powerful solution for transforming messy, unstructured text into clean, reliable tabular data, thereby addressing a critical bottleneck for knowledge workers and data analysis.
