IPO Implementation Fix: IPO now matches DPO and KTO results

AI Impact Summary

The document details a critical fix to the implementation of Identity Preference Optimization (IPO) for LLMs, revealing that the original approach incorrectly averaged loss over log-likelihoods instead of summing them. This correction, implemented in a recent PR, now aligns with the original IPO paper’s results, demonstrating that IPO performs comparably to DPO and outperforms KTO in paired preference settings. This update is crucial for teams utilizing TRL and its associated alignment methods.