TLDR: GeoGPT.RAG is an open large language model system designed to enhance geoscience research. It integrates Retrieval-Augmented Generation (RAG) to provide accurate, context-specific answers by leveraging a specialized GeoGPT Library and allowing users to create personalized knowledge bases. Key innovations include the domain-optimized GeoEmbedding and GeoReranker models, a synthetic data generation pipeline, and the GeoQA benchmark. The system demonstrates significant performance improvements in factual grounding and answer completeness, making it a powerful and accessible AI tool for geoscientists.
A new open large language model system, GeoGPT, is set to significantly advance research in the geosciences. Developed by the GeoGPT Team at Zhejiang Lab, this system integrates Retrieval-Augmented Generation (RAG) to enhance its domain-specific capabilities, allowing it to provide accurate and context-specific answers by drawing from specialized knowledge sources.
Large language models (LLMs) are powerful, but they often struggle with factual accuracy and domain-specific nuances in specialized fields. GeoGPT addresses this by using RAG, which grounds the model’s outputs in external knowledge. This is particularly crucial for geosciences, where precision and interpretability are paramount. RAG not only improves accuracy but also allows for dynamic updates to the knowledge base and promotes transparency by making the information retrieval process traceable.
Core Innovations for Geoscience AI
The GeoGPT project introduces several key contributions to the field:
- GeoEmbedding: This is the first decoder-based embedding model specifically optimized for geoscience. Built on a modern decoder-only architecture (Mistral-7B) and trained with synthetic domain-specific data, GeoEmbedding significantly improves retrieval accuracy for geoscience queries while maintaining strong general performance.
- GeoReranker: A domain-augmented reranking model, GeoReranker, is designed to improve the precision of search results in geoscience applications. It also performs well on general information retrieval tasks.
- LLM-based Synthetic Data Generation: To overcome the scarcity of high-quality labeled data in geosciences, the team developed a novel pipeline powered by LLaMA3-70B. This pipeline generates realistic and diverse training data by abstracting and reformatting scholarly geoscience articles.
- Data Quality Enhancement: Comprehensive strategies, including hard negative mining and refinement of positive and negative samples, were implemented to improve the effectiveness and robustness of the retrieval models.
- GeoQA Benchmark: GeoGPT also introduces GeoQA, the first large-scale benchmark for evaluating geoscience retrieval systems. It includes open-access paper-based documents and 938 expert-reviewed query-document pairs, providing a standardized tool for future research.
How GeoGPT.RAG Works
GeoGPT utilizes a curated, domain-tailored corpus called the GeoGPT Library, which includes content from open-access geoscience papers. Users can access GeoGPT for free at geogpt.zero2x.org. Beyond this shared library, GeoGPT offers a ‘My Documents’ feature, allowing users to upload their own academic papers or technical documents to create a personalized knowledge base. This effectively creates a customized version of GeoGPT for each user, tailored to their specific research needs and interests.
Performance and Evaluation
The system’s retrieval and generation capabilities were rigorously evaluated using a custom test set named GeoQA, comprising 1,000 questions derived from public geoscience papers. The integration of RAG significantly boosted answer completeness, showing a 13.7% absolute gain in Answer Recall compared to the base model. Expert evaluations on real-world queries related to petroleum exploration confirmed an 85.7% accuracy, demonstrating the practical effectiveness of RAG-enhanced GeoGPT in specialized domains.
Further improvements were achieved through Retrieval-Augmented Fine-Tuning (RAFT), where task-specific RAG examples were incorporated into the model’s training. This led to substantial performance gains in both in-domain (geoscience) and out-of-domain tasks, enhancing the model’s ability to integrate retrieved information effectively.
Optimizing RAG for Geoscience
The GeoGPT team implemented a multi-stage optimization process focusing on text segmentation, embedding model fine-tuning, and reranker model fine-tuning. Text segmentation ensures that lengthy documents are divided into semantically coherent chunks for effective retrieval. The GeoEmbedding model was fine-tuned using synthetic geoscientific data, while the GeoReranker model was trained with a large corpus of open-source and domain-specific ranking pairs to maximize its performance.
Also Read:
- HANRAG: Enhancing AI’s Question Answering Through Adaptive Retrieval and Noise Filtering
- A New AI System to Support Goat Health and Farm Management
Robust RAG Engineering
The RAG system leverages Zilliz Cloud for its vector database, offering high performance, scalability, and enterprise-grade data security. It maintains both a public library (approximately 15 million vector entries) and a user library (around 3 million entries), with user data stored in dedicated partitions for efficient querying. The system also includes a robust workflow for batch processing PDF files, encompassing text extraction, layout analysis, metadata extraction, cleaning, and vectorization for semantic indexing. For online chat, user queries are vectorized, relevant chunks are retrieved and ranked, and then combined with the original query to construct a prompt for the LLM to generate accurate and grounded responses.
The GeoGPT project represents a strong commitment to open science, with its core RAG components—GeoEmbedding and GeoReranker—open-sourced to support geoscientists and researchers worldwide. For more technical details, you can refer to the GeoGPT.RAG Technical Report.
