OpenAI DSGym: New Framework for Evaluating Data Science Agents

AI Impact Summary

OpenAI has released DSGym, a comprehensive framework for evaluating and training data science agents, particularly LLMs. This release introduces a vast collection of benchmarks, including 90 bioinformatics tasks and 92 Kaggle competitions, alongside synthetic trajectory generation capabilities. The framework's focus on addressing limitations in existing benchmarks—such as reliance on incompatible evaluation interfaces and isolated skills—positions it as a significant advancement for LLM-based data science agent development, particularly highlighting the need for robust domain-grounding capabilities.