Q8-Chat: 8-bit LLM inference on Intel Xeon with SmoothQuant and Optimum Intel

AI Impact Summary

Intel-led quantization techniques, including 8-bit quantization with SmoothQuant, enable mid-sized LLMs to run on a single Intel Xeon socket (Sapphire Rapids) with low latency. The post highlights quantizing OPT-13B/2.7B/6.7B, LLaMA 7B, Alpaca 7B, Vicuna 7B, BloomZ 7.1B, and MPT-7B-chat using Hugging Face Optimum Intel and Intel Neural Compressor to shrink models without major accuracy loss. This creates a viable on-prem CPU inference path for chat workloads, reducing GPU dependency and overall TCO, though performance and quality will vary by model and quantization approach. Migration involves integrating the Optimum Intel tooling and SmoothQuant into production pipelines to replicate the demo results.