TLDR: Innovator is a new large language model that successfully integrates extensive scientific knowledge with general capabilities, avoiding catastrophic forgetting. It achieves this by upcycling a pre-trained dense LLM into a fine-grained Mixture-of-Experts (MoE) model. Different experts specialize in various scientific disciplines, while a shared expert handles general tasks. This four-stage training paradigm, combined with meticulous data preparation, results in a model that significantly improves scientific performance (25% average improvement) while retaining 99% of its general task abilities. A post-trained version, Innovator-Reason, further boosts scientific reasoning by over 30%.
In the rapidly evolving landscape of artificial intelligence, Large Language Models (LLMs) have shown remarkable capabilities, from engaging in conversations to acting as intelligent agents. However, integrating vast amounts of scientific knowledge into these models without compromising their general abilities has been a significant hurdle. Directly training LLMs on scientific data often leads to “catastrophic forgetting,” where they lose their general task performance.
A groundbreaking new development, named Innovator, addresses this challenge head-on. Innovator is designed to create a powerful LLM that excels in both general and diverse scientific tasks, laying the foundation for what researchers call “science general intelligence.”
What is Innovator?
Innovator is a specialized type of LLM known as a Mixtures-of-Experts (MoE) model. Unlike traditional LLMs that use a single, large network, MoE models employ multiple smaller “experts,” each capable of learning different aspects or domains of knowledge. Innovator takes a pre-trained general LLM (specifically, Qwen2.5-7B) and “upcycles” it into this fine-grained MoE structure during continued training. This means it transforms an existing model rather than building one from scratch, making the process more efficient.
The core idea behind Innovator is to have different experts specialize in various scientific disciplines, such as physics, chemistry, or biology, while a dedicated “shared expert” maintains the model’s general knowledge. This separation helps prevent the negative interference between different knowledge domains, which is a common problem in multi-domain learning.
How Innovator Learns: A Four-Stage Journey
Innovator employs a unique four-stage training process to achieve its dual generalist-scientist capabilities:
1. Scientific Expert Induction: In this initial stage, the model identifies and creates “coarse-grained” scientific experts by training on data specific to individual scientific disciplines. The original general model’s knowledge is preserved during this step.
2. Fine-grained Expert Splitting: The coarse scientific experts are then further divided into multiple “finer-grained” experts. This step enhances the model’s ability to decouple knowledge across different scientific domains, allowing for more flexible and diverse combinations of expertise when solving complex problems.
3. Science-Aware Routing Warmup: A crucial component of MoE models is the “router,” which directs incoming information to the most relevant experts. In this stage, Innovator’s router is specifically trained to accurately dispatch scientific data to the appropriate specialized experts. This ensures that, for example, a chemistry problem goes to the chemistry experts.
4. Generalist-Scientist Integration: Finally, the entire Innovator model, comprising the shared general expert and the newly specialized scientific experts, is trained together on a vast dataset that includes both general and scientific information. This stage integrates all the learned knowledge, allowing the model to seamlessly switch between general and scientific tasks.
Data Quality: A Foundation for Success
The success of Innovator heavily relies on high-quality training data. The researchers developed a sophisticated “tri-level aligned framework” for general pre-training data, involving human experts, advanced LLMs, and smaller models to ensure cleanliness and coherence. For scientific data, an “intelligent cleaning and restoration system” was created to process complex scientific literature, handling everything from formula reconstruction to chart metadata maintenance. This meticulous data preparation ensures the model learns from accurate and relevant information.
Impressive Performance
Innovator demonstrates significant advancements. With 53.3 billion total parameters and 13.3 billion activated for each task, it was trained on 300 billion tokens. The results are compelling:
- It achieves an average 25% improvement across 30 scientific tasks, with a 70% win rate against other models.
- Crucially, it retains 99% of its performance in general tasks, effectively solving the catastrophic forgetting problem.
- Innovator even outperforms larger models like Qwen2.5-14B in scientific tasks by 16%, despite having fewer activated parameters.
Furthermore, a version called Innovator-Reason, which is post-trained from Innovator specifically for reasoning tasks, shows over 30% improvement in solving complex scientific problems, alongside enhanced non-scientific reasoning. This highlights Innovator’s robust foundation for advanced problem-solving.
Also Read:
- Multi-TAG: A New Framework for Advanced Mathematical Reasoning in AI
- PrismRAG: A New Approach to Enhance AI’s Factual Accuracy in Question Answering
The Future of Scientific AI
Innovator represents a significant step towards building LLMs that can truly act as scientific innovators. By effectively decoupling and integrating general and scientific knowledge, it paves the way for AI systems that can assist in understanding complex scientific literature, extracting knowledge, and solving domain-specific problems across a wide range of disciplines without sacrificing their broad utility. You can read the full research paper for more details here.
