Swiss-AI Releases Apertus-v1.5 with 262k Context and Thinking Mode
Swiss-AI has released Apertus-v1.5, a family of 8B and 70B open-source language models featuring a 262k context window, fully open training data, and a new thinking mode for reasoning. These models now support multimodal inputs, allowing them to process images, audio, and text to generate text. It advances the open-source AI ecosystem by providing fully transparent models (open weights, data, and training recipes) while respecting data owners' opt-out consent. The addition of native multimodal capabilities and a toggleable reasoning mode makes it highly competitive with proprietary alternatives. The models are built on a decoder-only transformer architecture using the xIELU activation function and trained with the AdEMAMix optimizer. Apertus-v1.5 was created by continuing the pretraining of Apertus 1.0, adding 4 trillion tokens to the 8B model and 2 trillion tokens to the 70B model.
## BACKGROUND
The AdEMAMix optimizer is an improvement over the standard AdamW optimizer, utilizing two Exponential Moving Averages (EMAs) of past gradients to accelerate convergence and lower training loss. The xIELU (Expanded Integral of the Exponential Linear Unit) is a trainable piecewise activation function designed to improve neural network training dynamics.