Mistral AI Releases Shieldstral, a 3B Multimodal Content Moderation Model
Mistral AI has released Shieldstral, a 3B-parameter open-source multimodal content moderation model under the Apache 2.0 license. The model supports 12 languages, runs on a single 16GB GPU, and achieves state-of-the-art performance in safety classification. Unlike traditional guard models with fixed safety policies, Shieldstral allows developers to dynamically input custom moderation rules directly into the prompt. This flexibility, combined with its low hardware requirements, makes advanced AI safety guardrails highly accessible and adaptable for diverse applications. The model structures moderation tasks as binary questions using three input fields: `<Instruct>`, `<Query>`, and `<Document>`, which can include text or images. During inference, it calculates a safety score by normalizing the logits of the "Yes" and "No" tokens using a softmax function.
## BACKGROUND
Content moderation in AI, often implemented via "guardrails," is crucial for preventing large language models from generating harmful, toxic, or policy-violating outputs. Traditionally, these guardrail models have rigid safety policies baked into their weights, requiring costly retraining or fine-tuning whenever a company's moderation guidelines change.