TLDR: A new study evaluates two strategies for using Large Language Models (LLMs) to enable systems to interoperate autonomously. Researchers tested 13 open-source LLMs on an agricultural data conversion task, finding that `qwen2.5-coder:32b` was the most effective. While direct data conversion (DIRECT strategy) struggled with complex tasks like unit conversion, generating code for conversion (CODEGEN strategy) proved more robust. The findings suggest LLMs can significantly reduce the effort required for system interoperability, with CODEGEN showing particular promise for creating reusable, deterministic solutions.
In today’s rapidly evolving digital landscape, systems are becoming increasingly complex, dynamic, and diverse. This trend amplifies a long-standing challenge in software engineering: interoperability. Interoperability refers to the ability of different systems to exchange and mutually use information. Beyond its technical intricacies, it also carries a significant economic burden, as considerable development time and effort are typically required to build the necessary components for systems to communicate effectively.
A recent research paper, titled “Evaluating the effectiveness of LLM-based interoperability,” explores how large language models (LLMs) might offer a groundbreaking solution to this persistent problem. The authors, Rodrigo Falcão, Stefan Schweitzer, Julien Siebert, Emily Calvet, and Frank Elberzhager, investigate whether LLMs can enable systems to interoperate autonomously, at runtime, without human intervention.
The Interoperability Challenge
Traditionally, achieving interoperability involves engineers understanding different data formats, implementing custom data adapters, and then testing and deploying these solutions. This process is often specific to each pair of systems, making it resource-intensive and slow. While technical standards and protocols exist, the challenge persists due to the continuous emergence of new technologies, organizational difficulties in agreeing on common data formats, and the sheer economic cost of manual adaptation.
Two LLM-Based Strategies
The researchers proposed and evaluated two distinct LLM-based strategies to tackle this challenge:
1. Direct Conversion (DIRECT): In this strategy, an LLM is directly prompted to convert input data from an unknown representation into a desired target format. The service dynamically constructs a prompt that includes the input data and an example of the target representation. The LLM then attempts to generate the converted data directly.
2. Conversion Module Generation (CODEGEN): This strategy involves prompting the LLM to generate actual source code (a conversion algorithm) that can transform the unknown input data into the target representation. This generated code is then cleaned, tested, and deployed in an execution environment. Once deployed, this module can be reused for subsequent data conversions without needing to call the LLM again.
Experimental Setup and Key Findings
To evaluate these strategies, the team selected 13 open-source LLMs and curated four versions of a dataset based on an agricultural interoperability use case. The task involved converting field boundary data from a proprietary John Deere API format to the widely used GeoJSON format. The dataset versions increased in complexity, with the fourth version (v4) requiring not only structural changes but also a unit conversion (hectares to acres).
The experiments involved three runs for each model with each dataset version, using both strategies. The effectiveness was measured using `pass@1`, a metric commonly used for evaluating code generation models, which indicates the proportion of correct top-1 predictions.
The results highlighted `qwen2.5-coder:32b` as the most effective model across both strategies and most dataset versions. For the DIRECT strategy, `qwen2.5-coder:32b` achieved an average `pass@1` of over 0.99 for datasets v1, v2, and v3. However, all models, including `qwen2.5-coder:32b`, failed when using DIRECT on dataset v4, which required unit conversion.
In contrast, using the CODEGEN strategy, `qwen2.5-coder:32b` maintained strong performance, with an average `pass@1` of 0.75 for dataset v4. This indicates that generating code for complex transformations, like unit conversions, was more successful than direct conversion. Other models generally performed significantly worse, with many not reaching a `pass@1` of 0.5.
Interestingly, the study also found that for some models using the DIRECT strategy, increasing dataset complexity (from v1 to v3) actually led to better performance. This was attributed to the models avoiding the addition of extra, unwanted key-value pairs from the input when the target schema became more defined.
Also Read:
- AI Teams Automate C4 Software Architecture Design
- A Unified Data Approach for Training Advanced AI Agents
Implications and Future Outlook
The research suggests that certain LLMs possess the capability to enable autonomous system interoperability, potentially saving significant development time. However, the task is not trivial for most models, and the study points to a need for better benchmarks specifically designed for data conversion tasks, as the performance observed in this experiment differed considerably from general code generation benchmarks like EvalPlus.
Choosing between DIRECT and CODEGEN involves trade-offs. DIRECT is simpler to implement but offers a non-deterministic solution, requiring an LLM call for every conversion. CODEGEN, while demanding more initial effort for implementation, yields a deterministic and reusable solution once the code is generated and deployed. The authors view CODEGEN as the more promising strategy for advancing LLM-based interoperability.
The paper concludes by recommending further evaluation in different domains and additional research into reliability strategies, especially given that an effectiveness below 1 might not be acceptable in many real-world applications. The full research paper can be accessed here: Evaluating the effectiveness of LLM-based interoperability.
