quantization to INT8 doubles throughput

tensor cores are

Tensor cores are specialized hardware for matrix multiply-accumulate on GPU

AWQ does differently

AWQ selectively retains weights crucial for model performance, unlike traditional quantization

tensor cores do 4x4 matrix multiply in one clock cycle

Tensor cores perform 4x4 matrix multiply using optimized GEMM (General Matrix Multiply) instructions in one clock cycle

GPTQ quantization does

Post-training quantization using second-order information for model compression

TensorRT does: NVIDIA's inference optimizer that quantizes and fuses operations

TensorRT optimizes deep learning inference by quantizing and fusing operations for NVIDIA GPUs

batch size affects generalization: larger batches find sharper minima

Larger batch sizes lead to sharper minima, enhancing generalization by providing more accurate gradient estimates