NEW
AWQ in Practice: Quantize LLMs for Faster AI Inference Without Killing Output Quality
AWQ cuts 70B model serving costs from $0.757 to $0.253 per million tokens by moving from two GPUs to one. Llama 3.3 70B quantized to INT4 with AWQ drops 2 MMLU points and 3 HumanEval points versus full precision. AWQ INT4 brings a 70B model's VRAM down from 140GB to roughly 40GB, fitting it on a…
