Giving DeepSeek V4 Flash Vision via a 40M-Parameter Connector
An open-source developer successfully added basic vision capabilities to the text-only DeepSeek V4 Flash model by training a 40.1-million-parameter MLP connector on 100,000 image-text examples. Both the large language model and the 417-million-parameter MoonViT image encoder remained frozen during the training process. This project demonstrates a cost-effective method (costing around $2,000) to convert a massive text-only Mixture-of-Experts (MoE) model into a multimodal model without expensive full-parameter retraining. It provides the open-source community with a practical blueprint and weights for building lightweight vision-language connectors. The model was quantized to NVFP4 format and served using the SGLang framework across four NVIDIA B200 GPUs. While the pilot model successfully performs basic OCR and UI grounding, it is not yet production-quality and can still hallucinate details or miss small text due to the limited training scale.
## BACKGROUND
Multimodal models often align visual features with text embeddings using a connector (like a projection layer or MLP) to bridge an image encoder and a language model. SGLang is a high-performance open-source serving framework designed for low-latency and high-throughput LLM and multimodal inference, while NVFP4 is a 4-bit floating-point format optimized for efficient inference on modern NVIDIA hardware.