Practical guide 05 · Local inference
What is the most capable LLM you can run on a Mac mini?
Apple's 64 GB M4 Pro configuration has 20.9 GB of nominal capacity beyond llama.cpp's documented 43.1 GB Llama 3.1 70B Q4_K_M conversion size; the 20.9 GB figure is a capacity calculation without measured residency validation. [1] [4] DS4 documents a larger DeepSeek V4 Flash experiment through expert-aware Q2 quantization and SSD streaming. [8] [19]
Apple specifies up to 64 GB unified memory in the current M4 Pro Mac mini. [1] llama.cpp documents a 43.1 GB Llama 3.1 70B Q4_K_M conversion size, and the pinned DS4 artifact page lists an 80.8 GiB DeepSeek V4 Flash quant. [4] [14] Those values define the capacity boundary examined here.
Parameter count tells only one part of the hardware story. Precision controls weight size. Architecture controls how many weights participate in each token. Context controls KV-cache growth. Runtime quality controls how efficiently the machine moves those bytes. Those four variables now matter as much as the number printed in a model name.
Evidence key
Publisher fact · Runtime measurement · Derived estimate · Editorial recommendation
Numbers identify their class in context. Bracketed links open the exact source used for the adjacent claim.
The short answer
Two ceilings: a daily driver and a capacity experiment
Editorial daily-driver choice
Qwen3.5-35B-A3B
Q5/Q6 target on 48 GB; Q6/Q8 target on 64 GB
Dense capacity example
Llama 3.1 70B Q4
43.1 GB documented conversion size; M4 Pro residency not benchmarked here
64 GB streaming experiment
DeepSeek V4 Flash
284B total; 80.8 GiB DS4 Q2 with MacBook SSD streaming
Why models suddenly fit
Parameter count split into three different questions
Model size used to work as a rough hardware proxy because most well-known transformers were dense and served in FP16. The weight-only calculation for 70B parameters at two bytes each is 140 GB before the KV cache and execution graph. llama.cpp documents 43.1 GB for its Llama 3.1 70B Q4_K_M conversion example. [4] New models break the old shortcut in several ways.
Ask three questions instead. How many total parameters need storage? That sets the weight footprint. How many parameters activate for one token? That sets much of the decode compute. How much state does the requested context create? That sets the KV-cache and attention overhead. Quantization, MoE routing and modern attention architectures affect different parts of this budget.
Dense transformer
70B active per generated token
Every feed-forward block participates in every forward pass.
Sparse Mixture of Experts
13B active per token in DeepSeek V4 Flash
A router selects expert blocks while attention and shared layers also participate.
DeepSeek V3 has 671B main-model parameters with 37B active for each token. [13] DeepSeek V4 Flash uses 284B total and 13B active. [8] Qwen3.5-35B-A3B uses 35B total and 3B active. [5] Gemma 4 26B A4B uses 25.2B total and 3.8B active. [6]
Expert routing avoids evaluating every routed expert for each token; attention, routing and shared layers still run. All expert weights still need a home. A conventional runtime keeps every expert in RAM or VRAM. DS4 adds a capacity mode for routed models: retain an expert cache in memory and load other experts from SSD on demand. [19]
Compression techniques
Quantization maps weights into lower-precision values
Training usually keeps weights in 16-bit or higher precision. Inference can represent those fixed values with fewer bits. The ideal weight-only estimate is parameter count multiplied by bits per weight, divided by eight. PyTorch defines the scale and zero-point mapping and the resulting quantization error. [15] Real GGUF formats add block metadata and may mix tensor precisions; llama.cpp reports 4.8944 effective bits per weight for its Llama 3.1 8B Q4_K_M example. [4]
llama.cpp reports 43.1 GB for a 70B Llama model in Q4_K_M, compared with 280.9 GB for the original 32-bit checkpoint. [4] The FP16 weight-only value is a separate calculation: 70 billion weights multiplied by two bytes equals 140 GB. Both values exclude runtime state.
Chart 1 · Same 70B model
Weight memory falls with effective bits per weight
Derived 70B weight-only extrapolations from llama.cpp's Llama 3.1 8B effective bits per weight; runtime, multimodal components and KV cache are extra [4]
The weight-only calculation falls from 140 GB at 16-bit to about 43 GB at 4.8944 effective bits per weight. Precision labels do not predict task quality; benchmark the exact file.
| Technique | Effective bits | Memory reduction | Quality impact | Main tradeoff |
|---|---|---|---|---|
| F16 | 16.0005 | 1.00x | Reference quality | Highest weight memory and bandwidth demand |
| Q8_0 | 8.5008 | 1.88x | Editorial: conservative baseline | Large files; hardware support determines speed |
| Q6_K | 6.5633 | 2.44x | Editorial: quality-first target | Validate the exact model and task |
| Q5_K_M | 5.7036 | 2.81x | Editorial: balanced target | Validate the exact model and task |
| Q4_K_M | 4.8944 | 3.27x | Editorial: common starting point | Quantization error remains task-dependent |
| Q3_K_M | 3.9960 | 4.00x | Editorial: capacity-first | Requires workload evaluation |
| IQ2_XXS | 2.3824 | 6.72x | Editorial: model-specific only | Use published calibration or expert-only evidence |
| Mixed precision | Tensor-specific | Model-specific | Preserves selected tensors | Needs calibration and runtime support |
| Mixture of Experts | Architecture | Compute, not weight bytes | Designed into training | Every expert still needs memory or storage |
| Specialized runtime | Execution layer | Overhead and traffic | Output equivalence must be tested | Support stays model and backend specific |
Effective-bit and reduction values are derived from llama.cpp's published quantization data. Quality descriptions beginning “Editorial” are evaluation starting points, not measured accuracy claims. [4] Mixed-precision behavior is illustrated by the DS4 tensor recipe. [14]
Symmetric and asymmetric quantization
Symmetric quantization centers the range around zero and removes the zero-point offset. PyTorch notes that skewed signals can waste resolution because the range includes values that do not occur. [15]
Affine, or asymmetric, quantization maps the observed minimum and maximum with a scale and zero point. PyTorch says this can provide a tighter clipping range, while weight inference can be more computationally expensive. [15]
How mixed precision preserves selected tensors
Quantization schemes can assign parameters per tensor or per channel, and sensitivity varies across a network. PyTorch recommends empirical validation and sensitivity analysis; llama.cpp exposes importance matrices and tensor-specific quantization controls. [15] [4]
DS4 provides a concrete mixed-precision example. Its 80.8 GiB DeepSeek V4 Flash file uses IQ2_XXS for routed-expert gate/up tensors, Q2_K for routed-expert down tensors, Q8_0 for attention, shared experts and output, plus F16 or F32 for routing and other control tensors. [14]
Why Apple silicon matters
Unified memory turns system RAM into a large GPU pool
A conventional workstation gives the CPU system RAM and a discrete GPU its own VRAM. Apple silicon instead exposes unified memory to its processors. MLX states that arrays live in shared memory and CPU or GPU operations can use them without transferring data. [2]
Apple offers the M4 Pro Mac mini with up to 64 GB unified memory and 273 GB/s bandwidth. [1] That capacity is shared by the operating system, applications, model, execution graph and cache. MLX LM warns that models large relative to total RAM can be slow and may require wired-memory tuning even when they fit. [3]
Discrete GPU topology
Fast inference favors a model that fits inside the dedicated VRAM boundary.
Apple silicon topology
One unified memory pool
Weights, KV cache, apps and the operating system all draw from the same capacity.
The bandwidth boundary
Apple lists 120 GB/s memory bandwidth for M4 and 273 GB/s for M4 Pro. [1] NVIDIA describes autoregressive decode as memory-bound. [12] Dividing 273 GB/s by llama.cpp's documented 43.1 GB Llama 3.1 70B Q4_K_M conversion size produces a 6.3 tok/s bandwidth-only ceiling. [4] The resulting figure is a transparent calculation with no measured Mac mini rate behind it, and actual throughput must be lower.
Sparse MoE changes that arithmetic because each token routes through selected experts while attention and shared layers still execute. Active parameter count alone does not predict observed speed. [5]
Where NVIDIA publishes higher bandwidth and accelerator support
NVIDIA lists 3.35 TB/s for H100 SXM and 3.9 TB/s for H100 NVL, alongside FP8 and INT8 Tensor Core support. [16] NVIDIA's inference guide documents batching, quantization, KV caching and speculative inference as serving optimizations. [12]
Editorial recommendation: favor a high-memory Mac for a private, single-user workstation; benchmark an accelerator server when concurrency or a strict latency target controls the decision. The hardware specifications alone do not establish workload performance.
Chart 2 · Mac mini recommendation
Editorial recommendation: 48 GB for balance, 64 GB for 70B Q4
Planning targets for model files, with unspecified headroom for macOS, cache and runtime state
The recommendations combine Apple memory tiers with published model dimensions and file sizes. They require validation on the exact runtime and context.
16 GB
M4, 120 GB/sEditorial model-file target: up to 10 GB
- Gemma 4 E4B
- Phi-4 14B Q4 target
- gpt-oss-20b native MXFP4
Learn, prototype and ship a focused single-user tool.
24 GB
M4 or M4 ProEditorial target: 18 GB; Qwen Q4_K_M is tight at 18.6 GB
- Qwen3 30B A3B Q4_K_M (tight)
- Gemma 4 26B A4B Q4 target
- gpt-oss-20b native MXFP4
Entry point for sparse models with constrained headroom.
48 GB
M4 Pro, 273 GB/sEditorial model-file target: up to 40 GB
- Qwen3.5 35B A3B Q5/Q6 target
- Gemma 4 31B Q6 target
- Llama 3.1 70B Q3_K_M estimate
Best overall value for serious local development.
Chart 3 · Tradeoff matrix
Editorial starting points by quantization level
Qualitative task risk versus relative weight memory
Higher-bit formats allocate more weight precision; their task-quality impact is unmeasured here. Q3 and IQ2_XXS require evaluation on the exact model and workload.
Memory shares extrapolate llama.cpp's published Llama 3.1 8B effective bits per weight. They are not measured 70B file sizes. Quality labels are editorial starting points; validate the exact quantized file on your tasks. [4]
Chart 4 · Four mechanisms
Each optimization attacks a different bottleneck
Capacity, active compute, addressable memory and execution overhead
The largest local runs combine all four mechanisms. No single technique explains the change in consumer inference.
Quantization
Shrinks each weight
140 GB calculated at 16-bit; 43.1 GB published for 70B Q4_K_M
Tradeoff: Rounding error grows as bits fall [4]
Mixture of Experts
Routes each token to selected experts
Qwen3.5-35B-A3B activates 3B per token
Tradeoff: Attention, shared layers and all stored experts still matter [5]
Unified memory
Expands GPU-addressable pool
CPU and GPU share one 16-64 GB pool on Mac mini
Tradeoff: The OS, model and cache share the same bytes [1][2]
Specialized runtime
Changes kernels and data movement
Fused kernels, cache control and expert streaming
Tradeoff: Performance and output equivalence require testing [19]
Apple also offers 32 GB on the base M4 at 120 GB/s. [1] Editorially, treat it as an intermediate capacity tier and validate the exact model file, cache and runtime peak before purchase.
Mac mini model matrix
What realistically runs at 16, 24, 48 and 64 GB
Publisher model dimensions and Apple memory configurations are cited row by row. Fit labels and quantization choices are editorial planning recommendations for one interactive session; they are not vendor guarantees. Unmeasured models say so explicitly. The separate performance table contains the runtime measurements and derived ceiling used by this guide.
| Memory | Model | Parameters | Quantization | Fits? | Generation | Source | Recommendation |
|---|---|---|---|---|---|---|---|
| 16 GB | Gemma 4 E4B | 8B total / 4.5B effective | Q6 or Q8 target | Comfortable | No published Mac mini rate | [6] | Editorial: fast multimodal starter |
| 16 GB | Phi-4 reasoning-plus | 14B dense | Q4_K_M | Comfortable | No published Mac mini rate | [10] | Editorial: math-focused compact reasoner |
| 16 GB | gpt-oss-20b | 21B total / 3.6B active | Native MXFP4 | Publisher: 16 GB | No official Apple speed result | [7] | Editorial: close other apps and cap context |
| 24 GB | Gemma 4 26B A4B | 25.2B total / 3.8B active | Q4_K_M | Comfortable | No published Mac mini rate | [6] | Editorial: sparse multimodal option |
| 24 GB | Qwen3-30B-A3B | 30.5B total / 3.3B active | Q4_K_M | Tight | No published Mac mini rate | [17][20] | 18.6 GB file leaves 5.4 GB nominal margin before OS, runtime and KV cache |
| 24 GB | Qwen3.5-35B-A3B | 35B total / 3B active | Q3_K_M | Tight | No published Mac mini rate | [5] | Editorial: use 48 GB for higher precision and context |
| 48 GB | Qwen3.5-35B-A3B | 35B total / 3B active | Q5_K_M or Q6_K | Comfortable | No published Mac mini rate | [5] | Editorial: balanced capacity and headroom |
| 48 GB | Gemma 4 31B | 30.7B dense | Q6_K | Comfortable | No published Mac mini rate | [6] | Editorial: dense multimodal option |
| 48 GB | Llama 3.1 70B | 70B dense | Q3_K_M estimate | Editorial estimate | No published Mac mini rate | [18][4] | Editorial capacity estimate: 35.0 GB weight-only; runtime and cache are extra |
| 64 GB | Qwen3.5-35B-A3B | 35B total / 3B active | Q8 or Q6_K | Comfortable | No published Mac mini rate | [5] | Editorial: higher-precision daily-driver candidate |
| 64 GB | Llama 3.1 70B | 70B dense | Q4_K_M | Capacity estimate | Derived ceiling: 273 GB/s ÷ 43.1 GB = 6.3 tok/s; actual throughput is lower | [18][4][1] | Capacity calculation: documented 43.1 GB conversion size leaves 20.9 GB nominal margin before OS, runtime and KV cache |
| 64 GB | gpt-oss-120b | 117B total / 5.1B active | Native MXFP4 | Does not fit natively | No official Apple speed result | [7] | Publisher target: one 80 GB accelerator; smaller community quants need validation |
| 64 GB | DeepSeek V4 Flash | 284B total / 13B active | DS4 routed Q2 | SSD streamed | No published 64 GB rate; SSD-cache dependent | [8][14][19] | Editorial capacity experiment; 80.8 GiB weight file |
“Comfortable,” “tight,” “editorial estimate” and “capacity estimate” are editorial fit judgments. “Publisher: 16 GB” repeats OpenAI's stated gpt-oss-20b requirement. The Llama 3.1 70B row shows a derived upper bound; every other unmeasured rate is marked as unpublished.
Performance expectations
Prompt processing and generation are different workloads
Prompt processing, also called prefill, sees the full input at once and turns large chunks into matrix-matrix operations. GPUs parallelize that work well. Generation, or decode, produces one token after another. Each new token waits for the previous token and repeatedly moves weights plus attention state through memory. NVIDIA's inference guide describes decode as memory-bound for this reason. [12]
DS4's published M3 Max 128 GB Q2 run illustrates the gap: its 11,709-token case reports 250.11 prompt tokens per second and 21.47 generated tokens per second. [11] Chat responsiveness depends on time to first token after prefill and the steady decode rate that follows.
| Configuration | Hardware | Prompt | Prefill | Decode | Evidence class | Source |
|---|---|---|---|---|---|---|
| M3 Max 128 GB, DS4 Q2 | MacBook Pro M3 Max | Short prompt | 58.52 t/s | 26.68 t/s | Runtime measurement | [11] |
| M3 Max 128 GB, DS4 Q2 | MacBook Pro M3 Max | 11,709 tokens | 250.11 t/s | 21.47 t/s | Runtime measurement | [11] |
| M5 Max 128 GB, DS4 Q2 | MacBook Pro M5 Max | Short prompt | 87.25 t/s | 34.27 t/s | Runtime measurement | [11] |
| M3 Ultra 512 GB, DS4 Q4 | Mac Studio M3 Ultra | Short prompt | 78.95 t/s | 35.50 t/s | Runtime measurement | [11] |
| M4 Pro 70B Q4 ceiling | Mac mini M4 Pro, 273 GB/s | 43.1 GB documented conversion size | Not derived | At most 6.3 t/s | Derived estimate; actual lower | [1][4] |
| 64 GB DeepSeek V4 Flash SSD streaming | MacBook with 64 GB | DS4 example configuration | Not published | No published 64 GB rate | MacBook mode; Mac mini transfer unmeasured | [19][14] |
The four DS4 measurements are community-reported single-run Metal CLI results at commit efdadd4 using --ctx 32768, --nothink, greedy decoding and -n 256. The short prompt is a small Italian story prompt. The source identifies the tested variants only as q2 and q4, not by an immutable GGUF filename. DS4 warns that the table may lag later runtime optimizations. [11]
Context and KV cache
Every cached token adds keys and values across layers. GQA, sliding attention and compressed attention reduce the slope, while 128K or 1M context still consumes real memory. [12][6][8]
SSD offloading
Streaming adds capacity and trades predictable RAM access for cache misses and storage latency. Fast internal SSDs help; resident weights remain the preferred path. [19]
Concurrency
Each live conversation needs its own KV state. A model that fits for one chat can run out of memory with several long sessions, even when weights are shared. [12]
Speculative decode
A draft model or multi-token head proposes future tokens. The main model verifies them in parallel, improving speed when acceptance is high. [12]
Quality versus hosted OpenAI models
Use named models as reference points, then test your workload
Hosted products combine a base model with system prompts, tools, retrieval, safety layers and serving code. Quantized local weights can change results again. The table therefore repeats only named publisher comparisons or exact benchmark cells; it does not turn them into a broad equivalence claim.
| Local model | Publisher comparison or exact cells | Scope and source |
|---|---|---|
| Phi-4 reasoning-plus 14B | Phi / o3-mini: AIME 2025 78.0 / 78.0; GPQA Diamond 68.9 / 77.7 | Microsoft-published evaluation. Microsoft describes Phi as math-focused and primarily English. [10] |
| gpt-oss-20b | OpenAI says it delivers similar results to o3-mini on common benchmarks | Publisher summary, not an independent one-to-one evaluation. OpenAI says the native model requires 16 GB. [7] |
| Qwen3.5-35B-A3B | Qwen / GPT-5-mini 2025-08-07: MMLU-Pro 85.3 / 83.7; GPQA Diamond 84.2 / 82.8; SWE-bench Verified 69.2 / 72.0 | Qwen-published results under its stated harness, before a community local quantization. [5] |
| Llama 3.3 70B | No one-to-one OpenAI comparison used in this guide | Meta publishes benchmark rows, but an era-style product analogy would hide setup and quantization differences. [9] |
| gpt-oss-120b | OpenAI says near-parity with o4-mini on core reasoning benchmarks | Publisher summary for native MXFP4. OpenAI specifies one 80 GB GPU; the claim provides no measured result from a 64 GB Mac. [7] |
| DeepSeek V4 Flash High/Max | No one-to-one OpenAI result; Flash Max: GPQA Diamond 88.1, LiveCodeBench Pass@1 91.6, SWE-bench Verified 79.0 | DeepSeek-published scores for the official FP4-expert plus FP8 mixed release. A community routed-Q2 file needs separate quality validation. [8] |
Hardware decision
Buy memory for the model class, then buy bandwidth for patience
16 GB
M4, 120 GB/sLearn, prototype and ship a focused single-user tool.
Editorial model-file target: up to 10 GB. This planning target leaves unspecified headroom for the operating system, execution graph and cache; verify peak memory on the exact runtime before standardizing.
24 GB
M4 or M4 ProEntry point for sparse models with constrained headroom.
Editorial target: 18 GB; Qwen Q4_K_M is tight at 18.6 GB. This planning target leaves unspecified headroom for the operating system, execution graph and cache; verify peak memory on the exact runtime before standardizing.
48 GB
M4 Pro, 273 GB/sBest overall value for serious local development.
Editorial model-file target: up to 40 GB. This planning target leaves unspecified headroom for the operating system, execution graph and cache; verify peak memory on the exact runtime before standardizing.
64 GB
M4 Pro, 273 GB/sChoose for 70B dense models and capacity experiments.
Editorial model-file target: up to 52 GB. This planning target leaves unspecified headroom for the operating system, execution graph and cache; verify peak memory on the exact runtime before standardizing.
Choose the runtime
MLX LM provides Apple-silicon generation, conversion, quantization, fine-tuning and prompt-cache tools. [3] llama.cpp documents Apple Silicon Metal support, integer quantization and CPU+GPU hybrid inference. [21] DS4 is explicitly a narrow DeepSeek V4 and GLM runtime with expert streaming. [19]
Leave operational headroom
Do not treat model-file size as total memory use. MLX LM identifies model and cache memory separately, and llama.cpp notes that multimodal encoders or projectors may require another file. [3] [4] Editorial recommendation: measure peak memory with the intended context and retain enough headroom for normal desktop use.
Benchmark before standardizing
Record runtime commit, model file, quant, context, prompt tokens, generated tokens, time to first token, decode rate and peak memory. Repeat warm and cold runs. A model that wins a short chat can lose a 40K-document workflow through prefill time or cache pressure.
What comes next
Planning beyond 64 GB with published model-file sizes
High-memory Apple laptops show the next tier in one model-specific runtime. DS4 reports 26.68 t/s decode on an M3 Max 128 GB and 34.27 t/s on an M5 Max 128 GB for short-prompt resident q2 runs. [11] Separately, the pinned community artifact page lists a routed-Q2 file at 80.8 GiB. [14] DS4 documents SSD streaming on a 64 GB MacBook but does not publish a matching rate or an M4 Pro Mac mini result. [19]
Planning scenario, not a product forecast: 128 GB exceeds OpenAI's stated 80 GB requirement for native gpt-oss-120b and the 80.8 GiB DS4 Q2 file, before runtime and cache allowances. [7] [14] A 256 GB pool exceeds DS4's published 153.3 GiB routed-Q4 file. [14] Actual fit and speed would still depend on the released hardware, runtime and context.
Hardware is only half the trajectory. New architectures use sparse experts, grouped or compressed attention, quantization-aware training and multi-token prediction. Runtimes increasingly specialize kernels, cache layouts and prompt formats for one model family. The likely future is a small set of highly optimized local stacks, each extracting more capability from the same memory bandwidth.
Decision FAQ
The questions that change a hardware purchase
What is the most capable LLM that fits fully in a Mac mini?
Our capacity recommendation for the current 64 GB M4 Pro is a 70B dense model at Q4 or a 35B-class sparse MoE at higher precision. The cited Llama Q4_K_M file is 43.1 GB, and Qwen3.5-35B-A3B has 35B total parameters with 3B active per token. This guide has no direct M4 Pro speed comparison between them. [1][4][5][18]
Can a 64 GB Mac mini run DeepSeek V4 Flash?
DS4 documents a 64 GB MacBook configuration that streams its 80.8 GiB routed-Q2 file from SSD while caching experts in unified memory. Applying that mode to the M4 Pro Mac mini is an unmeasured editorial transfer; DS4 publishes no matching 64 GB throughput result. [14][19]
Does an MoE model only need memory for its active parameters?
Every expert weight still needs a home in memory or storage. Routing evaluates selected expert blocks instead of every expert, while attention and shared layers still run. Total parameter count and precision continue to determine stored weight bytes unless a runtime streams experts from SSD or another machine. [5][8][13][19]
Is 16 GB enough for local LLM development?
Yes for small-model application development. OpenAI explicitly states that native gpt-oss-20b requires 16 GB, while Gemma 4 E4B has 8B total and 4.5B effective parameters. Context, multimodal components and runtime state also consume the shared pool, so measure peak memory on the intended workload. [7][6][3][4]
Source register
Publisher sources and community runtime artifacts
Hardware and model specifications were checked against publisher documentation on 3 August 2026. Community runtime measurements and quantization artifacts are labeled and pinned separately. Calculations show their inputs, and planning judgments are labeled editorial recommendations.
- 01Apple Mac mini technical specifications
Publisher specification · Apple Support
M4 and M4 Pro memory configurations and 120 GB/s versus 273 GB/s memory bandwidth.
- 02MLX framework repository
Official framework documentation · Apple Machine Learning Research
Shared-memory arrays and CPU/GPU execution on Apple silicon.
- 03MLX LM documentation
Official framework documentation · Apple Machine Learning Research
Quantization, rotating KV caches, prompt caching and large-model memory guidance.
- 04llama.cpp quantization documentation
Community project documentation · pinned commit · ggml-org
Llama 3.1 8B effective bits per weight and the separate documented 43.1 GB Llama 3.1 70B Q4_K_M conversion size at commit ad1b88c.
- 05Qwen3.5-35B-A3B model card
Publisher model card · Qwen
35B total, 3B active, 256 experts and a native 262K context window.
- 06Gemma 4 model card
Publisher model card · Google DeepMind
Dense and MoE variants, including Gemma 4 26B A4B with 25.2B total and 3.8B active parameters.
- 07OpenAI gpt-oss release
Publisher release · OpenAI
117B/5.1B-active and 21B/3.6B-active models, MXFP4 precision and official memory guidance.
- 08DeepSeek V4 model card
Publisher model card · DeepSeek
DeepSeek V4 Flash has 284B total, 13B active and a one-million-token architecture.
- 09Llama 3.3 70B model card
Publisher model card · gated weights · Meta
70B dense architecture, grouped-query attention and 128K context.
- 10Phi-4 reasoning-plus model card
Publisher model card · Microsoft Research
14B dense reasoning model, 32K context and task-specific evaluation results.
- 11DwarfStar DS4 runtime
Community beta runtime · pinned commit · Salvatore Sanfilippo and contributors
Single-run Metal benchmarks, routed-expert SSD streaming and the beta-status warning at commit efdadd4.
- 12NVIDIA LLM inference optimization guide
Official vendor guide · NVIDIA
Prefill, memory-bound decode, KV caching, batching, quantization and speculative inference.
- 13DeepSeek V3 model card
Publisher model card · DeepSeek
671B total parameters, 37B activated per token and 685B including the MTP module.
- 14DeepSeek V4 Flash DS4 GGUF files
Community quantization artifact · pinned revision · Salvatore Sanfilippo
80.8 GiB routed-Q2 and 153.3 GiB routed-Q4 files with tensor-specific precision at revision 1cd7b56.
- 15Practical quantization in PyTorch
Official framework guide · PyTorch
Scale and zero-point mapping, quantization error, symmetric and affine schemes, calibration and per-channel quantization.
- 16NVIDIA H100 specifications
Publisher specification · NVIDIA
H100 SXM and NVL memory capacity, HBM bandwidth and low-precision Tensor Core specifications.
- 17Qwen3-30B-A3B model card
Publisher model card · Qwen
30.5B total parameters, 3.3B activated, 128 experts and native 32K context.
- 18Llama 3.1 70B model card
Publisher model card · gated weights · Meta
70B model class, grouped-query attention and a 128K context window.
- 19DwarfStar DS4 SSD streaming documentation
Community beta runtime · pinned commit · Salvatore Sanfilippo and contributors
64 GB MacBook SSD-streaming example, expert-cache accounting and non-resident performance caveats at commit efdadd4.
- 20Qwen3-30B-A3B Q4_K_M GGUF
Publisher quantization artifact · pinned revision · Qwen
Official 18.6 GB Q4_K_M GGUF file at revision e4d4baf.
- 21llama.cpp runtime documentation
Community project documentation · pinned commit · ggml-org
Apple Silicon Metal support, integer quantization and CPU+GPU hybrid inference at commit ad1b88c.