TLDR: A new open-source conversational AI assistant integrates multimodal retrieval and large language models to enable non-expert users, like farmers, to interact with complex agricultural and geospatial data using natural language. It combines orthophotos, satellite imagery (vegetation indices), and textual documents to provide contextualized answers about agricultural plots. Evaluated using an LLM-as-a-judge methodology, the system demonstrates high-quality, relevant, and clear responses, effectively lowering the technical barrier to accessing specialized agricultural information.
In an era where vast amounts of Earth Observation (EO) and agricultural data are freely available, a significant challenge remains: how to make this complex information accessible to everyday users, especially farmers and local stakeholders, who lack specialized technical skills. A new research paper introduces an innovative solution: a multimodal conversational assistant designed to bridge this gap, enabling natural language interaction with diverse agricultural and geospatial datasets.
The core problem addressed by this research is the high technical barrier associated with accessing, processing, and interpreting geospatial data. While initiatives like the Copernicus program provide extensive satellite data, and numerous open-access documents exist for agriculture, their utility is limited for non-experts. This new assistant aims to democratize access to this valuable information, transforming how agricultural plots are characterized and managed.
How the Assistant Works
The proposed system is an open-source conversational assistant that integrates multimodal retrieval and large language models (LLMs). It allows users to query specific parcels of land using natural language and receive contextualized, multi-source answers. The system combines three main types of data:
- Orthophotos: High-resolution aerial images of the terrain, processed by multimodal LLMs to generate detailed descriptions.
- Sentinel-2 Imagery: Multispectral satellite data used to derive vegetation indices like NDVI (Normalized Difference Vegetation Index) and EVI (Enhanced Vegetation Index), which reflect the biophysical conditions of the land.
- User-Provided Documents: Integration of domain-specific textual knowledge, including public repositories and user-uploaded PDFs, through a retrieval-augmented generation (RAG) mechanism.
This architecture allows the system to flexibly determine whether to rely on visual evidence, textual knowledge, or both when formulating a response. The user interacts through a conversational interface, providing a query and a plot identifier (e.g., “Is plot 0:0:107:161:1 adequate land for planting apple trees?”). The backend then retrieves plot information from registries like SIGPAC (Geographic Information System for Agricultural Plots), gathers context from orthophotos and satellite data, and queries a document repository. All this multi-domain data is then fused and converted into natural language, preparing a rich context for the final LLM to generate an informed answer.
Under the Hood: Key Technologies
The system leverages several advanced AI technologies. For the final conversational responses, the Qwen3-32B model was chosen due to its superior performance among evaluated open-source LLMs. The inference engine vLLM manages the model, orchestrated through the liteLLM proxy. Beyond the main LLMs, the system also incorporates embedding models, rerankers, and a tool-triage model to efficiently process and prioritize information.
Evaluating Performance
Assessing the quality of responses in specialized domains like geospatial and agriculture is challenging due to a lack of standard benchmarks. The researchers adopted an “LLM-as-a-judge” methodology, using Qwen3-32B in a zero-shot, unsupervised setting. Responses were evaluated across four dimensions: correctness, relevance, clarity, and completeness, each scored on a 1-5 scale.
Preliminary results from 45 experiments showed that the system consistently produces high-quality responses. Both multimodal retrieval-based queries and RAG-based queries yielded strong results, with multimodal queries generally achieving slightly higher scores across all dimensions. The system demonstrated its ability to respond accurately and contextually to diverse agricultural queries, such as assessing land suitability for planting apple trees or maximizing economic and environmental sustainability for a specific parcel.
Also Read:
- Advancing Conversational Search with Intelligent Query Reformulation and Result Merging
- See&Trek: Boosting AI’s Spatial Awareness Without Extensive Training
Impact and Future Directions
This work represents a significant step towards making highly specialized Earth Observation and agricultural data more accessible to non-expert users. By providing an open, reproducible, and scalable design, the system has the potential to support decision-making in agriculture and land management across various geographic regions. The primary contributions include a novel architecture for fusing multimodal EO and textual knowledge, a practical demonstration of lowering data access barriers through natural language, and an open design that encourages further development.
The research highlights that conversational interfaces can effectively bridge the gap between complex technical datasets and end-users without requiring specialized training. Future work will focus on expanding context retrieval sources, optimizing computational efficiency, and exploring other state-of-the-art open-source models. You can read the full research paper here: A Multimodal Conversational Assistant for the Characterization of Agricultural Plots from Geospatial Open Data.
