TLDR: DeepWriter is a novel AI writing assistant designed for specialized domains like finance, medicine, and law. It addresses the limitations of traditional Large Language Models (LLMs), such as hallucination and lack of domain knowledge, by operating on a curated, offline knowledge base. The system employs a sophisticated pipeline involving task decomposition, outline generation, multimodal retrieval, and section-by-section content composition with reflection. A key innovation is its fine-grained citation system, which allows for precise source attribution down to the paragraph or sentence level, ensuring factual accuracy and verifiability. Experiments on financial report generation demonstrate that DeepWriter produces high-quality, verifiable articles that surpass existing baselines in factual accuracy and content quality, even when using more compact models.
Large Language Models (LLMs) have shown impressive abilities in many areas, but they often fall short when used as writing assistants in specialized fields like finance, medicine, or law. This is because they lack deep, domain-specific knowledge and can sometimes make up information, a problem known as hallucination. Existing solutions, such as Retrieval-Augmented Generation (RAG), can struggle with consistency over long documents, while methods relying on online searches might produce lower quality content due to unreliable web information.
To tackle these issues, researchers have introduced DeepWriter, a new kind of writing assistant. DeepWriter is designed to be customizable, can handle different types of content (multimodal), and is built for generating long documents. Crucially, it operates using a carefully selected, offline knowledge base, meaning it doesn’t rely on potentially unreliable internet searches.
How DeepWriter Works
DeepWriter uses a unique process that involves several key steps. First, it breaks down a complex writing task into smaller, manageable parts. Then, it generates an outline for the document. After that, it retrieves relevant information from its offline knowledge base, which includes both text and visual elements like images and charts. Finally, it composes the document section by section, with a built-in reflection mechanism to ensure quality.
One of DeepWriter’s strengths is its ability to deeply extract information from unstructured sources like PDF documents. It uses advanced tools to organize text, tables, and images along with their associated details. This structured information is then stored in a hierarchical database, making retrieval efficient and accurate. This database has three levels: a ‘document’ level for broad concepts, a ‘page’ level for intermediate detail, and a ‘chunk’ level for raw source information.
When generating content, DeepWriter first refines the user’s initial request to make it more precise. It then breaks down the task into specific sub-queries focusing on facts, data, and key points. It uses a multimodal embedding model to find relevant text and visual content from its knowledge base. The retrieved information is then grouped by section titles, which helps in organizing the writing process.
The writing process itself is done section by section. DeepWriter creates a draft for each section, combining facts, data, and viewpoints, and then refines it. It also summarizes previously written sections to maintain coherence and avoid repetition. A notable feature is its approach to integrating multimodal content. It calculates how relevant each image or chart is to different parts of the text and then intelligently places them to ensure they appear near their most relevant descriptions, maintaining the document’s logical flow.
Grounded Citations for Reliability
A critical feature of DeepWriter is its precise citation system. Unlike traditional methods that might cite an entire document, DeepWriter can point to specific pages or even paragraphs within the source material. For multimodal content, it can even include coordinates for images and tables. This fine-grained citation ensures that every factual claim, number, or visual element can be easily traced back to its original source, significantly improving the document’s verifiability and trustworthiness.
Also Read:
- Enhancing Language Model Accuracy with Knowledge Graph Integration
- Building Specialized AI Expertise: A Knowledge Graph Approach to Domain-Specific Superintelligence
Performance and Future Directions
DeepWriter was tested on generating financial reports using the World Trade Report (WTR) dataset. The experiments showed that DeepWriter performs comparably to leading open-source models, even those using much larger foundation models like GPT-4o. It particularly excels in factual accuracy and the overall quality of generated content, demonstrating the effectiveness of its controlled, offline approach.
While DeepWriter shows strong performance, there are areas for improvement. For instance, it currently lacks sophisticated temporal reasoning, meaning it might struggle with queries involving time-sensitive information. It also faces challenges with very complex visual elements and its precise citation system can sometimes lead to very verbose citations. Future work aims to address these limitations by incorporating better temporal understanding, more advanced visual processing, and exploring hybrid approaches that combine offline reliability with timely online information, while maintaining strict fact-checking.
DeepWriter represents a significant step forward in creating AI writing assistants that produce high-quality, fact-grounded, and verifiable documents for specialized domains. You can learn more about this research in the full paper available here.
