TLDR: This research introduces HiCBench, a new benchmark designed to effectively evaluate document chunking methods in Retrieval-Augmented Generation (RAG) systems, specifically addressing the issue of evidence sparsity in existing benchmarks. It also proposes the HiChunk framework, which uses fine-tuned LLMs to create multi-level hierarchical document structures. Combined with the Auto-Merge retrieval algorithm, HiChunk significantly enhances chunking quality, retrieval accuracy, and overall RAG system response quality, demonstrating its effectiveness through comprehensive experiments and maintaining reasonable processing times.
Retrieval-Augmented Generation, or RAG, has become a cornerstone for enhancing how language models interact with external knowledge. By pulling in relevant information, RAG helps these models provide more accurate and up-to-date responses, reducing common issues like hallucinations. However, a critical yet often overlooked aspect of RAG systems is document chunking – how a document is broken down into smaller, manageable pieces for retrieval. The quality of these chunks directly impacts the relevance of the retrieved information and, consequently, the overall quality of the generated response.
Existing RAG evaluation benchmarks often fall short in truly assessing the effectiveness of different chunking methods. Researchers have found that many current benchmarks suffer from ‘evidence sparsity,’ meaning only a few sentences in a document are relevant to a given query. This makes it difficult to distinguish between good and bad chunking strategies, as even poor chunking might still retrieve the few relevant sentences. In real-world scenarios, users often need answers that require dense, continuous fragments of information, such as when summarizing or enumerating facts.
Introducing HiCBench: A New Standard for Chunking Evaluation
To address this gap, a new benchmark called HiCBench has been developed. HiCBench is specifically designed for document Question-Answering (QA) and aims to effectively evaluate how chunking methods influence various parts of the RAG system, including the chunker itself, the retriever, and the response model. It achieves this by providing manually annotated multi-level document chunking points, along with synthesized ‘evidence-dense’ QA pairs and their corresponding evidence sources. This focus on dense evidence ensures that chunking methods are rigorously tested on their ability to segment semantically continuous fragments accurately and completely.
The HiChunk Framework: Hierarchical Structuring for Better Retrieval
Beyond evaluation, the paper also introduces the HiChunk framework. Unlike traditional chunking methods that often treat documents as linear sequences, HiChunk recognizes that documents have inherent hierarchical structures (sections, subsections, paragraphs). It uses fine-tuned large language models (LLMs) to create these multi-level document structures. This hierarchical approach allows RAG systems to dynamically adjust the semantic granularity of retrieved chunks based on the user’s query.
A key innovation within the HiChunk framework is the Auto-Merge retrieval algorithm. This algorithm works by adaptively adjusting the granularity of retrieval chunks. When relevant child nodes (smaller chunks) are identified, the algorithm can intelligently merge them upward into their parent nodes (larger, more semantically complete chunks) if certain conditions are met, such as having enough related child nodes and sufficient token budget. This ensures that the retrieved context is both semantically rich and complete, bridging fragmented knowledge gaps.
Also Read:
- HANRAG: Enhancing AI’s Question Answering Through Adaptive Retrieval and Noise Filtering
- MMORE: Bridging the Gap Between Diverse Data and Large Language Models
Experimental Validation and Impact
Experiments conducted on HiCBench and other datasets demonstrate the effectiveness of the HiChunk framework. HiCBench proved to be a superior tool for evaluating chunking methods, especially with its evidence-dense QA tasks, which highlighted the performance differences between various chunking strategies. The HiChunk method, particularly when combined with the Auto-Merge algorithm (HC200+AM), consistently showed improved chunking accuracy and enhanced performance across the entire RAG pipeline, leading to better evidence retrieval and higher-quality model responses.
Furthermore, the research explored the influence of the retrieval token budget and the maximum hierarchical level. It was found that a larger token budget generally improves response quality, and HiChunk+AM maintained superior performance across different budget settings. The importance of hierarchical structure was also underscored, with performance improving as more hierarchical levels were considered. Crucially, HiChunk achieves these quality improvements within an acceptable time cost, making it practical for real-world RAG system implementations.
In conclusion, this research provides valuable insights into the often-underestimated role of document chunking in RAG systems. By introducing HiCBench for robust evaluation and the HiChunk framework with its Auto-Merge algorithm for enhanced hierarchical structuring, the paper offers significant advancements for improving the overall effectiveness and efficiency of Retrieval-Augmented Generation. For more details, you can read the full paper here.
