BigCode releases StarCoder2 and The Stack v2 — 3B, 7B, and 15B models

AI Impact Summary

BigCode is releasing StarCoder2, a family of open code LLMs with varying parameter sizes (3B, 7B, and 15B) built upon The Stack v2, a new, high-quality code dataset. The 15B model, trained by NVIDIA on NeMo infrastructure, achieves performance comparable to 33B+ models. The Stack v2 leverages Software Heritage's archive of software source code, enabling contextualized model training and offering a significant expansion in training data compared to its predecessor.