Assisted Generation: Low-Latency Text Generation via Caching

AI Impact Summary

This change introduces a new decoding method, "assisted generation," focused on reducing text generation latency by revisiting the bottlenecks in autoregressive text generation. The core technique involves caching model forward pass results to avoid redundant computations, particularly addressing the memory bandwidth limitations of running large language models. This approach leverages techniques like Flash Attention and batching to improve throughput, alongside hardware optimizations and distributed computation across multiple devices.