GPT-OSS Agentic RL Training: Log-Probability Mismatch Fix

AI Impact Summary

OpenAI's GPT-OSS model is being explored for agentic reinforcement learning training, a technique that optimizes decision-making through direct interaction with environments. This retrospective details a critical fix to the verl training framework used for GPT-OSS, addressing a log-probability mismatch stemming from the model's Mixture of Experts architecture. The core issue was that subtle differences in forward passes during MoE routing caused the importance sampling ratio to deviate from 1, leading to PPO instability and preventing effective training.