quantization

LLM量化

https://zhuanlan.zhihu.com/p/616969812

• SmoothQuant
• Outlier Suppression
  • Outlier Suppression
  • Outlier Suppression+
• AWQ
  基于激活与参数大小的缩放保护
• LLM.int8
  • LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
  • https://timdettmers.com/2022/08/17/llm-int8-and-emergent-features/
  • https://github.com/timdettmers/bitsandbytes
• GPTQ
  • GPTQ: 模型量化，穷鬼救星
    不使用贪心算法，对W进行优化时，固定位置挑选Q
    W不同列间权重更新互相独立，使用批处理更新（Lazy Batch更新，group_size参数控制）
    数值稳定性优化
  • 4-bit LLM Quantization with GPTQ
• ZeroQuant
  • https://github.com/microsoft/DeepSpeed/pull/2217
  • ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation
• LUT-GEMM
  • LUT-GEMM: 基于lut的量化矩阵乘法在大规模生成语言模型中的高效推理
• SparseGPT
• weight only
  • Who Says Elephants Can’t Run: Bringing Large Scale MoE Models into Cloud Scale Production
  • TVM weight-only
  • cutlass_fpA_intB_gemm