nanoVLM: PyTorch VLM Training Toolkit

AI Impact Summary

The nanoVLM repository provides a simplified toolkit for training Vision Language Models (VLMs) using PyTorch, leveraging Google’s SigLIP vision encoder and HuggingFaceTB’s SmolLM2 language model. This project is designed for beginners and researchers seeking a lightweight and understandable approach to VLM development, particularly focusing on Visual Question Answering. The training pipeline utilizes a split optimizer with different learning rates for the modality projector and the pre-trained backbones, and incorporates mixed precision training and logging to the Weights & Biases platform.