Consumer GPUs now run models that were cloud-only two years ago. A 16 GB card serves quantized 27–32B models comfortably; two of them handle larger workloads and longer contexts. Everything runs offline, and no prompt ever leaves the house. The premium AI tier is included — the only metered usage is the electricity meter.
The stack
- Inference server. llama.cpp-based servers (llama-swap and friends) load GGUF models on demand and swap them by route. Q4_K_M is the everyday sweet spot; Q8_0 where VRAM allows.
- API gateway. A LiteLLM-style proxy gives every app in the house one OpenAI-compatible endpoint, per-key budgets, and logging — cloud fallback optional.
- Memory & tooling. Self-hosted memory stores, search frontends, and assistants all speak plain HTTP to the gateway.
Sizing rules of thumb
- VRAM budget: model file + KV cache + activations. A 32B Q4 model with long context wants ~20–24 GB across GPUs, or a shorter context.
- Quantization: Q4_K_M for speed, Q8_0 when quality matters and it fits. Extreme low-bit quants trade quality for capacity — test before committing.
- RAM vs VRAM: CPU offload works but is slow; keep hot layers on GPU.
Good habits
- Long-running generation and other GPU work compete for the same VRAM — sequence them or size for both.
- Set an API key on anything that listens on the LAN. "Local only" services still deserve auth.
- Quantization is a per-model decision: check the model's own quant zoo and community perplexity numbers before picking a size.
- Monitor host RAM, not just VRAM — model servers hold big mmaps and caches on the CPU side.