TLDR: Apertus is a new suite of fully open large language models (LLMs) developed by the Swiss AI Initiative. It addresses critical shortcomings in data compliance and multilingual representation by training on 15 trillion tokens from over 1800 languages, retroactively respecting data privacy, and using a novel Goldfish objective to prevent memorization. Released at 8B and 70B scales, Apertus models achieve state-of-the-art performance among fully open models on multilingual benchmarks. The project emphasizes transparency by releasing all development artifacts and incorporates a ‘Swiss AI Charter’ for ethical alignment on controversial topics, aiming to democratize trustworthy and globally relevant AI.
A new era for large language models (LLMs) is dawning with the introduction of Apertus, a groundbreaking suite of models developed by the Swiss AI Initiative. This project aims to tackle two significant challenges in today’s AI landscape: ensuring data compliance and vastly improving multilingual representation. Unlike many existing models that release only their weights, Apertus is a fully open model, offering complete transparency across its development cycle.
The core philosophy behind Apertus is built on rigorous data compliance. The models are exclusively pretrained on openly available data, meticulously filtering out content that does not respect content-owner rights. This includes retroactively applying `robots.txt` exclusions, and filtering for non-permissive, toxic, and personally identifiable information (PII). To further enhance privacy and prevent unintended data leakage, Apertus employs a unique “Goldfish objective” during pretraining. This method significantly suppresses the verbatim recall of training data while maintaining strong performance on various tasks.
Another standout feature of Apertus is its expansive multilingual coverage. While many LLMs primarily focus on English or a small subset of high-resource languages, Apertus has been trained on an impressive 15 trillion tokens from over 1800 languages. Approximately 40% of its pretraining data is dedicated to non-English content, a substantial allocation that aims to democratize LLMs for a broader global audience. This commitment extends to post-training, where data from 149 languages is used to adapt Apertus for user interaction, including a specialized dataset for Romansh, Switzerland’s fourth national language.
The Apertus suite includes models at two scales: 8 billion and 70 billion parameters. These models have demonstrated strong performance on multilingual benchmarks, rivaling or even surpassing many open-weight counterparts. The development of Apertus involved several architectural and training innovations, such as the xIELU activation function and the AdEMAMix optimizer, designed to stabilize large-scale training and improve efficiency.
Beyond the model weights, the Swiss AI Initiative is releasing all scientific artifacts from the Apertus development cycle under a permissive license. This includes data preparation scripts, intermediate checkpoints, evaluation suites, and training code. This level of transparency allows for independent audits and extensions, fostering a truly open and collaborative AI ecosystem. The name “Apertus” itself, Latin for “open,” reflects this core commitment to transparency.
The post-training phase of Apertus involves supervised finetuning to adapt the models to structured conversational formats, followed by an alignment stage using the Quantile Reward Policy Optimization (QRPO) algorithm. This alignment process refines the model’s behavior based on human preferences for helpfulness, harmlessness, and honesty. For controversial topics, Apertus incorporates the principles of the Swiss AI Charter, a set of precepts informed by Switzerland’s constitutional values, ensuring a balanced and ethically guided response.
The training of Apertus was a monumental engineering feat, utilizing the Alps Research Infrastructure at the Swiss National Supercomputing Centre (CSCS). This supercomputing system, equipped with NVIDIA Grace-Hopper GPUs, provided the stable and scalable environment necessary for months of continuous, large-scale LLM training. The project consumed around 6 million GPU hours, highlighting the immense computational resources required for such an endeavor.
Also Read:
- Exploring Fault Tolerance in Large Language Models with GoldenTransformer
- Enhancing Private In-Context Learning Through Public Information
While Apertus represents a significant leap forward, the team acknowledges its current limitations, including the potential for hallucinations, toxic outputs, and its text-only modality. The models are intended for deployment only after extensive use-case alignment and additional testing. Future directions for the project include training even larger models, distilling the 70B model into smaller versions, and expanding into multimodality while maintaining the same high standards of compliance and transparency. For more detailed information, you can refer to the Apertus V1 Technical Report.
