TLDR: The BMGQ framework introduces an automated, four-stage pipeline for generating high-difficulty, training-ready multi-hop reasoning questions from semi-structured data. It addresses the scarcity of suitable datasets for training large language models by transforming raw knowledge into structured evidence clusters, building diverse logical reasoning paths using Natural Language Inference (NLI), and constructing complex questions through a bottom-up, reverse reasoning strategy. A robust Data Quality Evaluation System ensures that generated questions are challenging, uniquely solvable, and verifiable, significantly reducing manual curation effort and enabling scalable production of high-quality training data for advanced AI reasoning.
Creating advanced AI models that can answer complex questions requiring multiple steps of reasoning and information retrieval is a significant challenge. While many datasets exist for training these models, most fall short in truly testing an AI’s ability to dig deep, connect obscure clues, and reason across different knowledge domains. These existing datasets often feature shallow reasoning chains or are designed purely for evaluation, making them unsuitable for the large-scale training needed to build highly capable AI agents.
Manual creation of such complex questions is prohibitively expensive and doesn’t scale. This creates a critical bottleneck for developing AI models that can handle real-world, intricate information retrieval and reasoning tasks. To address this, a new research paper titled “BMGQ: A Bottom-up Method for Generating Complex Multi-hop Reasoning Questions from Semi-structured Data” introduces an automated framework for generating high-difficulty, training-ready multi-hop questions from semi-structured knowledge sources.
Authored by Bingsen Qiu, Zijian Liu, Xiao Liu, Haoshen Yang, Zeren Gao, Bingjie Wang, Feier Zhang, Yixuan Qin, and Chunyan Li from ByteDance DMC, the BMGQ framework offers a scalable solution to this data scarcity problem. You can find the full research paper here: BMGQ Research Paper.
The BMGQ Framework: A Four-Stage Approach
The BMGQ methodology is structured into a four-stage pipeline, designed to transform raw knowledge into challenging, verifiable questions:
1. Data Sources & Adaptation: The process begins by taking raw data, such as information from Wikipedia and Wikidata, and converting it into a lightweight, high-performance relational database. This structured format allows for efficient querying and forms a robust foundation for subsequent reasoning tasks.
2. Node Information Construction: In this stage, the system identifies high-quality candidate entities and their supporting evidence from the prepared data. A key challenge here is to prevent “semantic drift,” where the reasoning path loses its relevance by expanding into generic or weakly related terms. BMGQ tackles this using a transformer-based BERT Named Entity Recognition (NER) model, which reliably filters out irrelevant concepts and ensures that only semantically grounded entities are considered.
3. Evidence Chain Construction: This is where the multi-hop reasoning paths are built. Instead of relying on simple similarity, which can lead to repetitive or shallow connections, BMGQ employs a Natural Language Inference (NLI) framework. This framework classifies relationships between entities based on whether an evidence passage logically supports a hypothesized connection. Six logical relation types are used (causes, part of, is a, has attribute, requires, used for), ensuring diverse and logically interpretable links. The system uses a controlled breadth-first expansion strategy, incorporating diversity constraints to create a rich, multi-layered graph of interconnected entities.
4. Question Construction & Optimization: The final stage transforms these evidence clusters into complex multi-hop questions. BMGQ uses a “bottom-up, reverse reasoning” strategy, starting from the most distant pieces of evidence and working backward to the main answer. This approach ensures that questions require deep reasoning rather than simple lookups. The questions undergo an “obfuscation” process, where explicit terms like exact years or names are generalized to increase retrieval difficulty. An iterative refinement loop further optimizes questions, increasing their complexity while rigorously preserving the uniqueness of the correct answer.
Ensuring Quality: The Data Quality Evaluation System
A crucial aspect of BMGQ is its robust Data Quality Evaluation System, which acts as a filtering layer to ensure that only high-quality, solvable, and unique questions are included in the final dataset. This system has two main components:
1. Graph-Based Textual Structure: Before formal evaluation, questions are converted into a structured graph representation, explicitly mapping subjects, objects, attributes, and their linguistic relations. This allows for early structural screening, discarding questions that don’t form coherent or solvable reasoning graphs based on criteria like the absence of orphan nodes, sufficient attribute count, edge count, and graph diameter.
2. Data Quality Evaluation Workflow: This two-step workflow rigorously validates questions. First, multiple AI models attempt to answer the generated question; if a majority agree on the correct answer, the question is accepted. If not, it proceeds to a more detailed verification. Here, the question is decomposed into atomic, verifiable constraints (predicates). These predicates are then screened against explicit conditions (time, location, entity type) and matched against an evidence pack. Only questions where the seed answer is uniquely and verifiably supported by evidence are retained.
Also Read:
- FAIR-RAG: Enhancing LLM Accuracy with Evidence-Driven Iterative Refinement
- FunReason-MT: Enhancing AI’s Ability to Use Tools in Complex Conversations
Impact and Future Directions
By automating the creation of multi-hop datasets that match the difficulty of advanced evaluation benchmarks like BrowseComp, BMGQ significantly reduces the cost of manual curation. This framework provides a scalable way to produce challenging, high-quality training data, which is essential for advancing research in reasoning-centric large language models. The authors plan to extend this pipeline to incorporate multimodal evidence, explore cross-lingual dataset construction, and integrate it with reinforcement learning workflows to further enhance AI reasoning capabilities.
