Technical guide 06 · Open model inference
How Much Hardware Do Open Models Actually Need?
Across 16 current open models and seven deployment modes, minimum credible capacity ranges from a few gigabytes on a notebook to eight-GPU servers, multi-node clusters, and SSD-streamed expert weights.
Open models now span an extraordinary range. Some run on a notebook with a few gigabytes of memory. Others require an eight-GPU server or a multi-node cluster.
Parameter count alone no longer tells us where a model can run.
The Open Model Deployment Precision Map compares current model families across their available precision and compression formats. It shows the approximate model or memory footprint, expected quality loss where paired results are available, and minimum plausible hardware class.
Interactive deployment map
Open models by format and minimum hardware class
Memory is approximate. Quality delta is shown only where a paired result was published.
Four-bit formats pull dense 27B models into consumer-GPU range; native low-bit frontier models still need server nodes unless their experts are streamed from SSD.
| Model / series | ReferenceBF16 / native | 8-bitINT8 / FP8 | 4-bitINT4 / Q4 | FP4NVFP4 / MXFP4 | 2-bitINT2 / ternary | 1-bitretrained | StreamSSD load |
|---|---|---|---|---|---|---|---|
| Gemma 4 E2BSmall multimodalEdge | |||||||
| Gemma 4 E4BSmall multimodalEdge | |||||||
| Gemma 4 12BDense 12BLocal | |||||||
| Bonsai 8BNative low-bit · Qwen3 baseLocal | |||||||
| Qwen3.6 27BDense 27BLocal | |||||||
| Bonsai 27BNative low-bit · Qwen3.6 baseLocal | |||||||
| gpt-oss-20b3.6B active / 21B totalLocal | |||||||
| Gemma 4 26B-A4B4B active / 26B totalLocal | |||||||
| gpt-oss-120b5.1B active / 117B totalAdvanced | |||||||
| Llama 4 Scout17B active / 109B totalAdvanced | |||||||
| Qwen3 235B-A22B22B active / 235B totalAdvanced | |||||||
| Mistral Large 341B active / 675B totalFrontier | |||||||
| DeepSeek V4 Flash13B active / 284B totalFrontier | |||||||
| MiniMax M2.710B active / 230B totalFrontier | |||||||
| GLM-5.240B active / 753B totalFrontier | |||||||
| Kimi K3104B active / 2.78T totalFrontier |
Qwen3.6 27B
17.6 GB · Q4 K_V
Hardware
Gaming GPU
8-32 GB VRAM
Method
Post-training quantization
How this deployment point was produced
Published delta
0.09%
Against the available reference
How to read this map
NR means no paired percentage was published; an empty cell means no credible public deployment point was added.
Native identifies the released low-precision baseline, not a post-training compression result.
Hardware is a minimum capacity class. Runtime overhead, context, KV cache, concurrency, and kernel support can raise it.
Practical center
Four-bit remains the practical sweet spot
For conventional post-training quantization, four-bit weight-only formats such as GPTQ, AWQ and GGUF Q4 generally offer the strongest balance between memory reduction, model quality and runtime support.
A 27B dense model that requires roughly 54 GB in BF16 can fit into approximately 18 GB using a good four-bit representation. This brings capable models into the range of consumer GPUs and high-memory notebooks.
Going below four bits can reduce memory further, but quality degradation becomes less predictable. Reasoning, coding and tool-use benchmarks often decline before simpler knowledge benchmarks reveal a problem.
Released baselines
Some models are already compressed
Not every low-precision checkpoint is a compressed version of a conventional BF16 model.
- gpt-oss uses native MXFP4 expert weights and was evaluated in that format.
- DeepSeek V4 Flash is released with a mixture of FP4 and FP8 weights.
- Mistral Large 3 is primarily distributed as an FP8 checkpoint.
- Kimi K3 ships with MXFP4 routed experts.
- Bonsai models are retrained specifically for binary or ternary weights.
For these models, the released low-precision checkpoint is the real baseline. Reporting a quality loss against a theoretical BF16 version would be misleading if no paired evaluation exists.
The map therefore distinguishes native low-bit models from post-training quantization and quantization-aware training.
Capacity is a floor
Model size is not the same as required hardware
A checkpoint fitting into 24 GB does not guarantee that the model runs comfortably on a 24 GB GPU. Inference also requires memory for runtime and temporary buffers, activations, KV cache, context length, and concurrent requests.
Runtime state
Kernels, temporary buffers, and the execution graph consume capacity beyond the weights.
KV cache
Context length, cache precision, and concurrent requests can materially change peak memory.
Kernel support
NVFP4 targets NVIDIA Blackwell. A fitting checkpoint does not guarantee accelerated execution.
Throughput target
Minimum fit, interactive latency, and production concurrency describe different hardware plans.
The hardware classes in the map should therefore be read as minimum credible capacity classes, not guaranteed latency or production-throughput recommendations.
Memory placement
Streaming changes the memory equation
Mixture-of-Experts models create another option: keeping only frequently used weights in memory and streaming routed experts from SSD.
Kimi K3 demonstrates the extreme case. Its native checkpoint occupies approximately 1.56 TB, but an experimental CPU engine measured only 8.24 GB of resident RAM by streaming weights from NVMe.
The trade-off is latency. The smallest-memory configuration required roughly 33 seconds per generated token. The model fits, but it is not an interactive deployment.
Streaming is therefore not another bit width. It is a memory-placement technique that exchanges RAM and VRAM requirements for SSD capacity and latency.
Deployment checklist
The main conclusion
The question is no longer simply, “How many parameters does this model have?”
A useful deployment assessment must ask:
- What is the model’s native precision?
- Are weights, activations or the KV cache quantized?
- Was the model quantized after training, trained with quantization or retrained as a low-bit model?
- Does the target runtime provide optimized kernels?
- Does the memory figure represent weights or peak runtime usage?
- Is the deployment optimized for interactive use, throughput or merely technical feasibility?
Quantization is becoming one of the main mechanisms for broadening access to capable open models. But bit width alone is not enough to predict quality, speed or required hardware.
The combination of model architecture, numeric format, runtime, memory placement and hardware determines what is actually deployable.
Selected sources