TLDR: A new research paper introduces ‘computing on unstructured data,’ a paradigm that uses a three-stage Extract-Transform-Project (XTP) pipeline to convert unstructured data (text, images) into structured formats for computation, then projects it back. This bidirectional approach, supported by a new system called MXFlow, combines AI models with symbolic methods to enable full read/write manipulation and reasoning over diverse data, moving beyond simple querying to unlock the analytical power of previously inaccessible information.
In an era where the vast majority of information exists in unstructured formats like text, images, audio, and video, traditional data systems have struggled to keep pace. These systems are built for structured data, leaving a wealth of information effectively locked away from powerful computational tools. However, recent advancements in Artificial Intelligence, particularly large language models (LLMs) and vision models, are beginning to bridge this gap, revealing the rich, latent semantic structure within seemingly unstructured data.
A new research paper, “A Case for Computing on Unstructured Data,” by Mushtari Sadia, Amrita Roy Chowdhury, and Ang Chen from the University of Michigan, proposes a groundbreaking paradigm shift: computing directly on unstructured data. This vision extends beyond mere querying, aiming for a full read/write capability that allows manipulation, interaction, and reasoning over diverse content. The core of this paradigm is a holistic three-stage pipeline: eXtract, Transform, and Project (XTP).
The XTP Pipeline: A Bidirectional Approach
The XTP pipeline is designed to be bidirectional, mirroring how humans process information. It involves:
- Extract: This initial phase ingests raw unstructured data and converts it into a structured internal representation. For example, text could be transformed into tables, images into symbolic programs, or audio into event timelines.
- Transform: Once data is in a structured format, traditional data processing techniques can be applied. This includes operations like optimization, imputation (filling in missing data), and validation.
- Project: The final phase translates the transformed results back into unstructured formats. This could be the original format or a different one, making the outputs easily consumable by humans and AI agents.
This closed-loop system allows unstructured data to leverage the analytical power of structured computation while retaining its original richness and accessibility. The intermediate structured representation acts as a computational foundation, enabling verifiable transformations, while the projection step ensures results are practical for real-world consumption.
Real-World Applications: Text and Diagrams
The researchers illustrate the XTP paradigm with two compelling use cases:
- Medical Text: Doctors often work with free-form medical prescriptions. An XTP pipeline could extract patient information and diagnoses to build a structured database. Beyond simple queries, the system could then transform this data to adjust dosages, identify at-risk patients, or generate revised prescriptions and reports.
- Engineering Diagrams: Engineers use complex visual artifacts like datacenter layouts. XTP could extract formal representations (ee.g., a symbolic program describing components and connections). This allows for transformations like validating redundancy, optimizing fault tolerance, or converting designs to different regional standards, finally projecting an updated visual diagram.
Introducing MXFlow: A New Data System
To realize this ambitious vision, the paper introduces MXFlow, a new data system specifically designed for unstructured data management. MXFlow allows users to express tasks as a dataflow graph, where operators (nodes) process data (edges). Key features of MXFlow include:
- Hybrid Tools: It seamlessly integrates both neural (AI models like LLMs) and symbolic (deterministic, rule-based) operators, leveraging the flexibility of AI and the correctness guarantees of traditional methods.
- Dynamic Dataflow: Unlike traditional systems with fixed paths, MXFlow supports dynamic routing. The system can adapt its processing path based on data characteristics or intermediate results, even looping back to earlier stages if needed.
- Multi-modality Computation: Each stage can operate across different data modalities, and projections can mix output modalities (e.g., text and charts) while maintaining consistency.
MXFlow addresses challenges such as the inherent ambiguity of unstructured data, the non-deterministic nature of LLMs, and the need for adaptive schema generation and entity mapping throughout the pipeline.
Also Read:
- Dynamic AI Routing for Better Table Comprehension
- Automating Data Pipelines: A Hybrid Approach for LLM-Powered Workflow Generation
The Future of Data Computation
The researchers conducted preliminary experiments demonstrating MXFlow’s capabilities, including processing clinical notes to generate anonymized reports and analyzing circuit diagrams for structural optimizations. These experiments highlight the system’s dynamic dataflow, the effective mix of neural and symbolic operators, and its ability to handle multimodal outputs.
In essence, MXFlow offers a foundation for structured reasoning over unstructured data, moving beyond retrieval-focused systems to enable full read-write computation. This represents a significant step towards unlocking the analytical power of the vast amount of unstructured information that surrounds us. You can read the full research paper here.
