NEW
AWQ Quantization vs 8-Bit and 4-Bit: What Actually Cuts LLM Inference Cost Without Breaking Quality
Inference cost stops being a notebook problem the moment you serve a model on hardware someone pays for every month. In FP16 (16-bit floating point, the default weight format), a 32B model needs 60+ GB of GPU memory. A 70B model needs about 140 GB for weights alone—more than a single 80 GB GPU…