Hardware tiers for running local AI

4 August 2026 · 11 min · Engineering

TL;DR Local inference hardware has standardized into three tiers: edge (1B to 7B models on mobile and single-board computers), workstation (24 GB VRAM for 27B to 70B models), and frontier (256 GB and above for large MoE architectures). Memory capacity and bandwidth, rather than raw compute FLOPS, remain the primary binding constraint for agentic workflows and long context windows. Specialized unified memory hardware, such as the NVIDIA DGX Spark with 128 GB unified memory at approximately $4,000, enables executing up to 200B quantized parameter models on a single compact system. Standard 24 GB VRAM consumer GPUs combined with 4-bit quantization represent the optimal price-to-performance target for small engineering teams.

Deploying local artificial intelligence models requires matching hardware configurations against specific parameter counts, quantization levels, and context window requirements. While cloud evaluations emphasize FLOPS per dollar, local inference sizing is constrained primarily by VRAM bandwidth and memory capacity. Memory capacity sets the ceiling on parameter size, while memory bandwidth determines single-user token generation speed (P1).

The mismatch between cloud benchmarks and local constraints is a common source of procurement errors. A model that benchmarks well on a cloud cluster with 8x80 GB GPUs may be impossible to run on the workstation a small team actually owns. The hardware decision must come before the model selection, not after.

The three hardware deployment tiers

The local inference ecosystem is organized into three distinct hardware tiers based on memory capacity and target workload (P2). Each tier serves a different deployment scenario, and the boundaries between them are set by memory capacity, not compute speed.

Edge tier: mobile, NPU, and single-board computers

The edge tier targets 1B to 7B parameter small language models. These devices rely on integrated Neural Processing Units (NPUs), system RAM (8 GB to 16 GB), or low-power ARM SoCs. The hardware is defined by constraint: limited memory, limited bandwidth, and limited power draw.

Edge hardware handles specialized tasks like offline transcription, basic classification, and lightweight local search without network connectivity. A 1B parameter model at 4-bit quantization occupies approximately 0.5 GB of memory, which fits comfortably within the RAM of a modern smartphone. A 7B model at 4-bit occupies approximately 4 GB, which fits on a single-board computer with 8 GB of RAM.

The use case for this tier is not general-purpose chat. It is targeted inference: transcribing audio offline, classifying documents, or running a small specialized model for a specific task where sending data to a cloud API is not acceptable. The constraint is power and connectivity, not capability.

Workstation tier: 24 GB VRAM to 128 GB unified memory

The workstation tier is the practical standard for small engineering teams and technical users. This tier relies on single workstation GPUs equipped with 24 GB VRAM (such as RTX 4090 or RTX 5090 class hardware) or Apple M-series chips with 48 GB to 128 GB of unified memory.

Workstation hardware executes quantized 4-bit (Q4) 70B dense models and active-parameter-efficient Mixture-of-Experts (MoE) architectures at interactive speeds. A 70B model at Q4_K_M occupies approximately 40 GB of VRAM, which fits on a 48 GB workstation card or a 64 GB Mac with unified memory. A 27B model at Q4_K_M occupies approximately 16 GB, which fits on a standard 24 GB consumer GPU with headroom for context window storage.

The workstation tier is where most local AI deployment happens in 2026. The hardware is affordable (a 24 GB GPU workstation costs $2,000 to $4,000), the models are capable (27B to 70B parameter models handle most enterprise tasks), and the deployment is simple (a single machine, no cluster management). For a team building an internal tool or a small business running a specialized model, this tier is the default starting point.

Frontier tier: 256 GB and above multi-GPU clusters

The frontier tier is designed for enterprise self-hosting of flagship open-weight models and massive MoE architectures. This tier utilizes multi-GPU setups (multiple 80 GB VRAM cards connected via NVLink) or high-end workstation clusters to maintain complete model weights in memory.

A 400B total parameter MoE model at Q4_K_M occupies approximately 230 GB of VRAM, which requires three 80 GB GPUs or four 64 GB GPUs. The cost of this configuration starts at $15,000 for the GPUs alone, before accounting for the server chassis, power supply, and cooling. This is the tier where the hardware cost begins to approach cloud API pricing for moderate workloads, and the decision to self-host becomes a question of data sovereignty and long-term cost rather than pure capability.

The frontier tier is not where most teams should start. It is where teams end up after they have validated their use case on workstation hardware and determined that the capability gap between a 70B model and a 400B model justifies the hardware investment. The procurement decision should be driven by measured performance on the actual workload, not by parameter count.

Unified memory architectures and specialized hardware

Unified memory designs have changed the economics of high-parameter local inference by combining system RAM and GPU memory into a single address space. This architectural shift eliminates the PCIe bus bottleneck that limits discrete GPU setups.

[Apple Silicon](https://developer.apple.com) hardware benefits from high-bandwidth unified memory architectures, with up to 800 GB/s on top-tier Mac configurations. This allows 64 GB and 128 GB desktop systems to host large model weights without dedicated server infrastructure (P1). The CPU and GPU share the same memory pool, so model weights loaded into memory are immediately accessible to the GPU without a copy operation.

In specialized workstation hardware, setups like the [NVIDIA DGX Spark](https://nvidia.com) (utilizing the GB10 Grace Blackwell architecture) provide 128 GB of unified memory and approximately 1 petaFLOP of sparse FP4 compute in a compact desktop envelope priced around $4,000 (P1). This configuration enables executing quantized models up to 200 billion parameters without multi-node server racks. The DGX Spark represents a new category: workstation-class hardware with server-class memory capacity, targeting teams that need to run frontier-scale models without data center infrastructure.

The significance of unified memory for local AI is that it breaks the traditional link between model size and hardware cost. A 128 GB unified memory system can hold a 200B parameter model at Q4, which would require three 80 GB discrete GPUs in a traditional architecture. The unified memory approach costs less, draws less power, and requires less physical space. The tradeoff is that unified memory bandwidth is lower than discrete GPU HBM, so token generation speed is slower. For interactive use cases where latency matters more than throughput, this tradeoff is acceptable.

Hardware selection strategy

For most organizational deployments, selecting hardware depends on context length and concurrency requirements. The decision framework has three primary paths.

For single-user and developer desktops, target the 24 GB VRAM workstation tier for the optimal balance of cost and model capability. A 24 GB card handles 27B models at Q4 with full context, which covers most enterprise tasks including document analysis, code generation, and retrieval-augmented generation.

For agentic and long-context workflows, prioritize unified memory capacity (64 GB to 128 GB) because key-value cache storage scales linearly with context length. A 27B model with a 128K context window requires additional VRAM for the KV cache on top of the base model weights. On a 24 GB card, the context window is constrained. On a 128 GB unified memory system, the context window can extend to its full length without memory pressure.

For enterprise serving with multiple concurrent users, deploy multi-GPU nodes in the frontier tier to accommodate both high total parameter sizes and parallel user requests. The concurrency requirement drives the hardware decision here, not the model size. A 27B model serving 50 concurrent users needs more GPU memory than a 70B model serving one user.

Where this leaves you

Hardware selection for local AI is a memory problem first and a compute problem second. The model you can run is determined by the VRAM you have, not the FLOPS you can deliver. Before you buy hardware, calculate the total parameter count of your target model at your chosen quantization, add the key-value cache for your required context length, and verify the sum fits within your memory budget.

The three tiers give you a framework: edge for targeted inference on constrained devices, workstation for general-purpose local AI on a single machine, and frontier for flagship models and concurrent serving. Which tier matches your deployment target, and does your current hardware fit the model you actually need to run?