Engineering guide

LLM compression and efficient inference

A source-backed map of the fundamentally different ways to shrink checkpoints, lower runtime memory, reduce active compute, or decode more tokens per target-model step. Each choice changes a different bottleneck.

Data as of: Aug 3, 2026

Techniques

Choose the resource you need to change

Quantization

Store or compute weights and activations with fewer bits.

Pruning

Remove or skip weights, channels, blocks, or other structures.

Distillation

Train a smaller student to reproduce a larger teacher.

Sparse / MoE

Route each token through only part of a much larger parameter set.

Runtime memory

Move, map, stream, or evict weights without changing checkpoint size.

Architecture

Change attention or model structure to reduce compute and memory movement.

KV cache

Reduce, page, quantize, or selectively retain autoregressive attention state.

Speculative decoding

Propose several tokens cheaply and verify them with the target model.

Emerging

Adapter merging and adjacent deployment techniques with different tradeoffs.

Interactive comparison

What each technique actually changes

Rows use source-scoped ranges or explicitly state when no authoritative cross-hardware benchmark exists.

								Sources
AWQQuantization	Yes	Yes	Often with fused kernels	Low to moderate	Calibration only	4-bit weights; total varies	Widely deployed	6 7
BF16Quantization	Yes	Yes	Usually on native hardware	Low	No	~2x weight bytes vs FP32	Mature	1 3
BitNetQuantization	Yes	Yes	Kernel-dependent	Training recipe-dependent	Yes	1.58-bit ternary weights	Emerging	10
Dynamic Expert CachingRuntime memory	No	Bounded cache	Locality-dependent	None if misses load exactly	No	Cache-budget-dependent	Specialized	22 25
Expert StreamingRuntime memory	No	Yes	Usually slower	None if lossless	No	Kimi example: 1.45 TB experts nonresident	Experimental	22 25
FP8Quantization	Yes	Yes	Hardware-dependent	Low to moderate	No for PTQ; yes for QAT	Up to ~2x tensor bytes vs FP16	Production on supported GPUs	3
FP16Quantization	Yes	Yes	Usually on native hardware	Low	No	~2x weight bytes vs FP32	Mature	1 3
GPTQQuantization	Yes	Yes	Often with fused kernels	Low to moderate	Calibration only	Up to ~4x INT4 weight memory	Widely deployed	4 5
HQQQuantization	Yes	Yes	Backend-dependent	Bit-width-dependent	No calibration	1-8 bit weights; total varies	Supported in Transformers	9
INT2Quantization	Yes	Yes	Not guaranteed	High risk	Method-dependent	No authoritative end-to-end value	Emerging	1 33
INT4Quantization	Yes	Yes	Kernel-dependent	Moderate	No	~4x memory in HF 4-bit path	Widely deployed	2 1
INT8Quantization	Yes	Yes	Kernel-dependent	Low to moderate	No	~2x memory in HF 8-bit path	Mature	2 8
Knowledge DistillationDistillation	Yes	Yes	Yes for smaller student	Student-dependent	Yes	Student architecture-dependent	Mature concept	14
KV Cache QuantizationKV cache	No	Yes during generation	Can hurt short-context latency	Bit-width-dependent	No	KIVI: 2.6x peak memory in tests	Available in runtimes	32 33
Layer StreamingRuntime memory	No	Yes	Usually slower	None if lossless	No	Runtime budget-dependent	Specialized	22 23
LoRA MergeEmerging	No meaningful base reduction	Adapter overhead only	Can remove adapter overhead	None from exact merge	Adapter already trained	Base size unchanged	Supported by PEFT	15
Memory MappingRuntime memory	No	Yes	Workload-dependent	None	No	Only touched pages resident	Mature OS primitive	24 40
Mixture of ExpertsSparse / MoE	No	Not automatically	Vs same total-parameter dense model	Architecture-dependent	Yes	Active parameters are top-k subset	Production at scale	16 17 18
Multi-Token PredictionSpeculative	No	Adds heads	Up to 3x in Meta paper	Can improve trained model	Yes	Serial decode steps reduced	Growing	38 18
PagedAttentionArchitecture	No	Less KV waste	2-4x serving throughput in paper	None	No	Near-zero KV waste in paper	Widely deployed	30 41
PruningPruning	Sometimes	Sometimes	Only with usable sparsity	Sparsity-dependent	Optional recovery	No authoritative universal value	Pattern-dependent	12 13
SmoothQuantQuantization	Yes	Yes	Up to 1.56x in paper	Negligible in tested models	Calibration only	Up to 2x memory in paper	Production implementations	8
SparseGPTPruning	Sparse representation	With sparse storage	Needs sparse kernels	Minimal at reported sparsity	No retraining	50-60% sparsity in paper	Research / specialized	12
Speculative DecodingSpeculative	No	Usually adds draft memory	2-3x in cited papers	None with exact acceptance	No for classic draft pair	Serial target steps reduced	Widely supported	36 37
SSD-backed inferenceRuntime memory	No	Yes	Much slower than resident weights	None if lossless	No	Kimi example: 8.24 GB RSS	Experimental / specialized	22
Structured PruningPruning	Yes when re-exported	Yes	More likely than unstructured	Structure-dependent	Often recovery	Pattern-dependent	Supported patterns only	13

Mechanisms, tradeoffs and hardware

Decision guide

Start with the limiting resource

Need to run locally?

Which hardware tier?

Datacenter GPU

Maximum quality

FP16 or BF16

Use FP8 only after engine-specific validation.

Consumer GPU

Fit matters

AWQ INT4

GPTQ is another mature path; choose the format your runtime accelerates.

6 4

CPU only

Local portability

GGUF

Pick a quantization that fits RAM and quality needs.

40 11

Model exceeds RAM

Capacity first

Expert / layer streaming

Expect storage-bound, very low throughput.

22 25

Comparison matrix

Deployment characteristics at a glance

Technique	Compression ratio	Accuracy loss	Hardware compatibility	Inference speed	Implementation complexity	Popularity	Maturity	Sources
AWQ	Commonly ~4x vs FP16 weights	Low to moderate	Broad serving support	Often with fused kernels	Medium	Very high	Mature	6 7
BF16	~2x vs FP32 weights	Low	Datacenter / newer CPU	Usually on native hardware	Low	Very high	Mature	1 3
BitNet	1.58 bits/weight representation	Training recipe-dependent	Specialized runtimes	Kernel-dependent	High	High research interest	Research / emerging	10
Dynamic Expert Caching	Checkpoint unchanged	None if misses load exactly	MoE + tiered memory	Locality-dependent	Very high	Emerging	Specialized	22 25
Expert Streaming	Checkpoint unchanged	None if lossless	MoE + fast storage	Usually slower	Very high	Emerging	Experimental	22 25
FP8	Up to ~2x vs FP16 tensors	Low to moderate	Hopper/newer and runtime-specific	Hardware-dependent	Medium	High	Production on supported hardware	3
FP16	~2x vs FP32 weights	Low	Broad	Usually on native hardware	Low	Very high	Mature	1 3
GPTQ	Commonly ~4x vs FP16 weights	Low to moderate	Broad via GPTQModel	Often with fused kernels	Medium	Very high	Mature	4 5
HQQ	Configurable 1-8 bit	Bit-width-dependent	CPU/CUDA/XPU; backend-specific	Backend-dependent	Low-medium	Medium	Growing	9
INT2	2-bit payload; overhead varies	High risk	Specialized	Not guaranteed	High	Low	Research / emerging	1 33
INT4	~4x vs FP16 weights	Moderate	Broad but format-specific	Kernel-dependent	Medium	Very high	Mature formats; varied kernels	2 1
INT8	~2x vs FP16 weights	Low to moderate	Broad	Kernel-dependent	Low-medium	Very high	Mature	2 8
Knowledge Distillation	Teacher/student-dependent	Student-dependent	Broad	Yes for smaller student	High	Very high	Mature	14
KV Cache Quantization	Cache-only, configurable bits	Bit-width-dependent	Backend-specific	Can hurt short-context latency	Medium	High	Growing	32 33
Layer Streaming	Checkpoint unchanged	None if lossless	Sequential layer execution	Usually slower	High	Low-medium	Specialized	22 23
LoRA Merge	~1x base model	None from exact merge	PEFT-supported architectures	Can remove adapter overhead	Low	Very high	Mature deployment step	15
Memory Mapping	Checkpoint unchanged	None	File-backed runtimes	Workload-dependent	Low-medium	High in CPU runtimes	Mature primitive	24 40
Mixture of Experts	Stored size unchanged	Architecture-dependent	MoE serving stack	Vs same total-parameter dense model	Very high	Very high	Production at scale	16 17 18
Multi-Token Prediction	Not compression	Can improve trained model	MTP-aware model/runtime	Up to 3x in Meta paper	High	High	Growing	38 18
PagedAttention	No value compression	None	Paged serving engines	2-4x serving throughput in paper	High	Very high	Mature	30 41
Pruning	Method-specific	Sparsity-dependent	Sparse-kernel-dependent	Only with usable sparsity	Medium	High research interest	Mixed	12 13
SmoothQuant	W8A8	Negligible in tested models	INT8 accelerators	Up to 1.56x in paper	Medium	High	Production-ready with supported stacks	8
SparseGPT	50-60% sparsity demonstrated	Minimal at reported sparsity	Sparse-kernel-dependent	Needs sparse kernels	High	High research interest	Research / specialized	12
Speculative Decoding	Not compression	None with exact acceptance	Draft-target support	2-3x in cited papers	High	Very high	Production	36 37
SSD-backed inference	Checkpoint unchanged	None if lossless	Fast local SSD	Much slower than resident weights	Very high	Emerging	Experimental	22
Structured Pruning	Pattern-dependent	Structure-dependent	Best for fixed N:M or removed channels	More likely than unstructured	High	Medium	Production for selected patterns	13

Real-world implementations

Where these techniques ship

llama.cpp

Portable local inference

CPU, Metal, CUDA and other backends

GGUFlow-bit quantizationmemory mappingCPU/GPU offloadKV cache options

vLLM

High-throughput serving

NVIDIA, AMD, Intel, TPU and CPU paths vary by feature

paged KV cachequantizationspeculative decodingcontinuous batchingexpert parallelism

TensorRT-LLM

NVIDIA-optimized serving

Supported NVIDIA GPUs

FP8/INT8/INT4paged attentionKV cache managementspeculative decodingMTP/Medusa/EAGLE

MLX

Apple Silicon inference

Apple Silicon unified memory

4-bit conversionrotating KV cacheprompt cachingdistributed inference

ExLlamaV2

Consumer NVIDIA GPU inference

CUDA GPUs; project is archived in favor of ExLlamaV3

GPTQEXL2 mixed bit-widthQ4 KV cachepaged attentionspeculative decoding

SGLang

Production model serving

GPU, CPU, TPU and NPU support varies

paged attentionRadixAttentionFP4/FP8/INT4/AWQ/GPTQspeculative decodingexpert parallelism

DeepSpeed

Distributed inference and compression

Multi-GPU and supported accelerator stacks

tensor parallelismkernel injectionINT8 MoQMoE inferenceNVMe offload tooling

KTransformers

CPU/GPU heterogeneous MoE inference

CPU plus GPU, with architecture-specific kernels

heterogeneous expert placementINT4/INT8 CPU kernelsGPTQ GPU kernelsCPU-GPU expert schedulingtiered prefix cache

kimi-k3-in-c

Storage-backed Kimi K3 CPU inference

Linux x86-64 CPU with AVX2/FMA and fast local storage

MXFP4 expert executionexpert streaminglayer streamingLRU expert cacheincremental decoding

References

Primary evidence index

46 papers and official project sources