Running open models privately: a deployment guide
You can run a capable model on hardware you own, with no API and no data leaving the building. This is not a research curiosity. It is a normal, supported path in 2025, and the tooling is mature enough for a small team to run it.
Two serving paths
Pick the path by who will use the model. llama.cpp loads models in the GGUF format and runs on CPU, GPU or both, with INT4 quantization that fits large models on modest hardware. It is the right choice for a single user, a developer laptop, or a classroom box. vLLM targets GPUs and serves many users at once, using PagedAttention and supporting AWQ, GPTQ, FP8 and INT4 (W4A16) weights. It is the right choice when throughput and concurrency matter.
Ollama wraps llama.cpp with a simpler interface and is a fine way to start. Open WebUI gives a chat front end. The point is that none of this phones home. The model, the server and the data sit on your network.
Quantization is the lever
A model released in BF16 or FP16 is large. Qwen3-32B in BF16 needs about 64 GB of VRAM. Quantize it to Q4_K_M and that drops to roughly 20 GB, small enough for a single consumer card. An 8B model in Q4_K_M needs about 5 GB. The trade is quality: aggressive INT4 can soften reasoning and maths, so keep INT4 for general chat and reach for AWQ or GPTQ where precision matters. AWQ preserves more accuracy at a given size than older GPTQ recipes.
Privacy and data residency
Run the server inside your own network and lock outbound traffic. With llama.cpp or a self-hosted vLLM there is no vendor in the loop, so no prompt or document leaves the building unless you send it. For schools and clinics this is the difference between a safeguarding policy and a wish.
A minimal reference stack
- Choose a model whose license fits your use. Prefer Apache 2.0 or MIT (L2) for products you will sell.
- Quantize to Q4_K_M if VRAM is the constraint; keep AWQ for precision work.
- Serve with llama.cpp for a single user or vLLM for a team behind a GPU.
- Front it with Open WebUI for a clean chat interface.
- Lock network egress so nothing leaves the host.
The 2025 reality is simple: a capable, private model is a procurement decision, not a research project. The cost is the hardware once, the license review once, and the engineering to keep the network closed.
The private-lens questions (cost, privacy, offline control) are one of three decision frames you can apply to the same research base before you choose a model. The other two are the enterprise and non-profit lenses.