GaLore: Training Llama 7B on RTX 4090 with 8-bit Optimizers

AI Impact Summary

GaLore enables training of large language models like Llama 7B on consumer-grade hardware (e.g., NVIDIA RTX 4090) by significantly reducing memory requirements through gradient projection and subspace switching. This approach leverages the low-rank structure of gradients and 8-bit precision optimizers to achieve over 82% reduction in optimizer state memory, unlocking LLM training for a wider range of users. The integration with Hugging Face Transformers and techniques like layer-wise updates further enhances the practicality of this approach.