MoEs in Transformers: WeightConverter enables faster loading of Qwen1.5-110B-Chat

AI Impact Summary

Mixture of Experts (MoE) models in Transformers are leveraging a WeightConverter-driven refactor to pack all expert weights into a single tensor and enable kernel-optimized MoE execution. This approach, coupled with dynamic weight loading and async materialization, reduces memory peaks and eliminates repeated scans during checkpoint loading. Benchmarks on Qwen/Qwen1.5-110B-Chat show dramatic load-time improvements when upgrading from v4 to v5: Async device_map loading falls from about 66–67 seconds to 20.71 seconds, with tensor-parallel variants reaching as low as ~10 seconds. These changes unlock faster onboarding of large MoE models and more predictable deployment on single-GPU or small-scale multi-GPU clusters, improving iteration speed and time-to-market for MoE-based services.