Quantization and MoE: model efficiency, explained

4 August 2026 · 11 min · Engineering

TL;DR GGUF Q4_K_M quantization remains the practical sweet spot, reducing parameter memory footprint to approximately 4.5 bits per weight with minimal loss in benchmark accuracy. Mixture-of-Experts architectures route tokens to active sub-networks (such as 3B active out of 35B total), drastically lowering per-token compute FLOPS compared to dense equivalents. Memory management techniques like PagedAttention reduce key-value cache memory fragmentation from 30% to 50% waste down to under 4%. Quantization reduces 14B parameter models to approximately 7 GB VRAM, allowing 27B to 35B models to fit comfortably on a single 24 GB GPU card.

Executing high-capability artificial intelligence models on localized consumer hardware relies on two architectural developments: low-bit parameter quantization and sparse Mixture-of-Experts (MoE) routing. Without these techniques, running a 35B or 70B parameter model at 16-bit precision requires server-grade multi-GPU arrays. Quantization and MoE routing reduce memory footprints and per-token compute demands while preserving task accuracy (P1).

These two techniques address different bottlenecks. Quantization reduces the memory footprint of the model weights. MoE routing reduces the compute cost of generating each token. Together, they make it possible to run models that would otherwise require data center infrastructure on a single workstation. Understanding how each works, and what they cost in accuracy and complexity, is necessary for any team making a local deployment decision.

Quantization formats: the GGUF Q4_K_M benchmark

Quantization maps floating-point model weights (16-bit FP16 or 32-bit FP32) down to lower bit representations (4-bit or 8-bit integers). The goal is to reduce memory occupancy without degrading model output quality beyond what the application can tolerate.

The [GGUF Q4_K_M format](https://github.com/ggerganov/llama.cpp) (4-bit quantization with medium k-quant block structures) is the established engineering standard for local self-hosting (P1). It compresses weight memory to approximately 4.5 bits per weight while maintaining near-baseline perplexity scores across coding, reasoning, and document retrieval tasks. The format uses block-wise quantization, where groups of weights share a scaling factor, which preserves more precision than naive uniform quantization.

The memory savings are significant. A 14B parameter model in FP16 occupies approximately 28 GB of VRAM. At Q4_K_M, the same model occupies approximately 7 GB. This is the difference between requiring a multi-GPU server and fitting on a single consumer GPU. A 27B model drops from 54 GB in FP16 to approximately 16 GB at Q4_K_M, which fits on a standard 24 GB workstation card with headroom for context.

Extremely low quantization levels (Q2_K or Q3_K_S) reduce VRAM requirements further but introduce noticeable accuracy degradation in complex reasoning workflows (P2). The tradeoff is not linear. Below 4 bits, the perplexity increase accelerates, and the model begins to lose coherence on multi-step reasoning tasks. For applications where reasoning quality matters, Q4_K_M is the floor. For applications where the model is used for simple classification or text completion, lower quantization may be acceptable.

Higher precision formats (Q5_K_M or Q8_0) preserve marginal precision at the cost of significantly higher memory consumption. A 27B model at Q8_0 occupies approximately 28 GB, which exceeds a 24 GB card. The accuracy gain over Q4_K_M is measurable but small, typically less than 1% on standard benchmarks. For most practical deployments, the memory cost of higher precision is not justified by the accuracy gain.

Emerging low-precision formats, such as [NVFP4](https://developer.nvidia.com) (NVIDIA 4-bit floating point), provide hardware-accelerated matrix multiplication on specialized Blackwell architectures while maintaining numerical stability (P1). Unlike integer quantization (INT4), floating-point quantization preserves a wider dynamic range, which can improve accuracy at the same bit width. However, NVFP4 requires hardware that supports the format, which limits its portability compared to GGUF.

Mixture-of-Experts: sparse routing efficiency

Dense models execute 100% of their parameters for every processed token. This means a 35B parameter model performs 35 billion multiply-accumulate operations per token. The compute cost scales linearly with total parameter count, and so does the latency.

In contrast, Mixture-of-Experts (MoE) architectures divide network layers into specialized sub-networks (experts) and route tokens dynamically. A gating network evaluates each token and selects a small subset of experts to process it. The remaining experts are inactive for that token.

For example, a 35B parameter MoE model might route each token through a gating network to activate only 3B parameters per forward pass (P1). This means the per-token compute requirement equals a small 3B model, delivering fast generation speeds. The total 35B weights preserve reasoning capability and factual knowledge, because the full parameter set is available across the expert pool. The model has the knowledge of a 35B system and the latency of a 3B system.

The tradeoff is in memory, not compute. Because any expert might be needed for any given token, all 35B total weights must remain loaded in VRAM. Memory capacity requirements are determined by the total parameter count, not the active parameter count. This is the core tension of MoE architectures: they trade compute efficiency for memory pressure.

For teams evaluating MoE models, the selection criteria are different from dense models. A dense 27B model at Q4_K_M occupies 16 GB and performs 27B operations per token. An MoE 35B-A3B model at Q4_K_M occupies 20 GB but performs only 3B operations per token. The MoE model is slightly larger in memory but substantially faster in inference. For interactive applications where latency matters, the MoE architecture is the better choice. For batch processing where throughput matters more than latency, the dense model may be simpler to deploy.

Key-value cache management: PagedAttention

In long-context RAG and multi-turn agentic workflows, memory bottlenecks often occur in the key-value (KV) cache rather than model weight storage alone. The KV cache stores the intermediate attention states for all tokens in the context window, and it grows linearly with context length.

Traditional inference engines allocate contiguous physical memory blocks for context windows, resulting in 30% to 50% memory fragmentation waste (P2). When a request finishes and frees its memory block, the freed space may be too small for the next request's context window, leaving gaps that cannot be reused. This fragmentation effectively reduces usable VRAM by up to half.

The [PagedAttention algorithm](https://docs.vllm.ai) (implemented in the vLLM serving engine) manages KV cache memory like virtual memory pages in operating systems (P1). By partitioning attention keys and values into non-contiguous physical pages, PagedAttention reduces memory fragmentation waste to below 4%. This enables significantly higher batch sizes and longer context windows on fixed VRAM.

The practical impact is that a server with 24 GB of VRAM can serve more concurrent users with longer context windows than the same hardware running a traditional inference engine. The memory that was previously lost to fragmentation becomes available for additional requests. For teams building multi-user API endpoints, PagedAttention is not an optimization. It is a requirement for achieving acceptable concurrency on fixed hardware.

Engineering recommendations

When configuring local inference pipelines, the following guidelines help balance accuracy, memory, and speed.

Select Q4_K_M quantization as the default for optimal accuracy per gigabyte of VRAM. This format provides the best-documented tradeoff between memory savings and quality preservation, with extensive benchmark data confirming near-baseline performance on coding, reasoning, and retrieval tasks.

Evaluate models based on whether target hardware can host the full total parameter weight in VRAM. For MoE models, the total parameter count, not the active parameter count, determines the memory floor. A 35B-A3B model needs memory for 35B parameters, not 3B.

Measure model perplexity and task accuracy on your specific domain data rather than relying solely on raw parameter counts. A 27B model fine-tuned on your domain may outperform a 70B general model on your specific task, while using less memory and generating faster. The benchmark that matters is the one run on your data, on your hardware, with your quantization.

Where this leaves you

Quantization and MoE routing are the two techniques that make local AI deployment practical on consumer hardware. Quantization reduces the memory footprint of the model weights. MoE routing reduces the compute cost of generating each token. PagedAttention reduces the memory waste in the key-value cache. Together, they transform a data center problem into a workstation problem.

Before you deploy, verify your numbers. Calculate the total parameter count at your chosen quantization, add the KV cache for your required context length, and confirm the sum fits within your VRAM budget. Then measure the actual perplexity and latency on your hardware with your data. Which of these three techniques, quantization, MoE routing, or PagedAttention, is most likely to shape your next deployment decision?