Sending confidential company data, source code, and customer records to commercial LLM APIs exposes organizations to privacy breaches, unexpected rate limits, and runaway monthly bills. With open-weight models like DeepSeek-R1 / V3, Llama 3...
Sending confidential company data, source code, and customer records to commercial LLM APIs exposes organizations to privacy breaches, unexpected rate limits, and runaway monthly bills.
With open-weight models like DeepSeek-R1 / V3, Llama 3.3 70B, Mistral NeMo, and Qwen 2.5, running local inference on your own hardware (or dedicated GPU cloud instances like Hetzner, RunPod, or Lambda) is faster and more cost-effective than ever.
However, selecting the right inference engine determines whether you achieve 5 tokens/sec or 120 tokens/sec per GPU.
In this guide, we break down the four leading self-hosted LLM serving engines and provide production-ready Docker Compose configurations.
1. Engine Comparison Matrix
Engine
Primary Focus
Best For
Model Formats
Concurrency & Batching
vLLM
High-throughput production serving
Multi-user API backends & enterprise
AWQ, GPTQ, FP8, SafeTensors
PagedAttention (Massive throughput)
Ollama
Developer simplicity & desktop/homelab
Single-user & small team prototyping
GGUF
Continuous batching (Basic)
LocalAI
Drop-in OpenAI API replacement
CPU + GPU fallback & multimodal
GGUF, GGML, ONNX, Diffusers
Standard queueing
TGI (Hugging Face)
Production enterprise deployment
Hugging Face ecosystem pipelines
SafeTensors, AWQ, EETQ
Dynamic batching & FlashAttention
2. Choosing Your Serving Architecture
When to Choose vLLM
You have modern NVIDIA GPUs (RTX 3090, 4090, A100, H100) or AMD ROCm hardware.
You need to serve multiple concurrent users or background agent workflows without throughput collapse.
You want state-of-the-art memory management via PagedAttention, eliminating VRAM waste from KV caching.
When to Choose Ollama
You want zero-configuration model pulling (ollama run llama3.3).
You are running on consumer hardware, Apple Silicon (M1/M2/M3/M4), or hybrid CPU/GPU setups using GGUF quantization.
You want seamless integration with Open WebUI, AnythingLLM, or Cursor.
When to Choose LocalAI
You need a unified engine capable of running text generation, text-to-speech (TTS), audio transcription (Whisper), and image generation (Stable Diffusion) inside a single container.
3. Production Deployment Configurations
Stack 1: High-Performance vLLM + Open WebUI (GPU Production Stack)
This stack exposes an OpenAI-compatible API on port 8000 and the premier ChatGPT-like interface (Open WebUI) on port 3000:
services:
vllm:
image: vllm/vllm-openai:latest
container_name: vllm_server
restart: unless-stopped
environment:
- HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
volumes:
- ~/.cache/huggingface:/root/.cache/huggingface
ports:
- "127.0.0.1:8000:8000"
ipc: host
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
command: >
--model Qwen/Qwen2.5-Coder-7B-Instruct
--gpu-memory-utilization 0.90
--max-model-len 8192
--dtype auto
--api-key ${VLLM_API_KEY:-super-secret-key-123}
open-webui:
image: ghcr.io/open-webui/open-webui:main
container_name: open_webui
restart: unless-stopped
ports:
- "127.0.0.1:3000:8080"
volumes:
- open_webui_data:/app/backend/data
environment:
- OPENAI_API_BASE_URL=http://vllm:8000/v1
- OPENAI_API_KEY=${VLLM_API_KEY:-super-secret-key-123}
- WEBUI_AUTH=true
- ENABLE_SIGNUP=false # Disable public registration
depends_on:
- vllm
volumes:
open_webui_data:
Stack 2: Ollama + Open WebUI (CPU & GPU Homelab Stack)
For mixed environments and easy GGUF model handling:
services:
ollama:
image: ollama/ollama:latest
container_name: ollama
restart: unless-stopped
ports:
- "127.0.0.1:11434:11434"
volumes:
- ollama_models:/root/.ollama
# Uncomment deploy block if NVIDIA GPU is present:
# deploy:
# resources:
# reservations:
# devices:
# - driver: nvidia
# count: all
# capabilities: [gpu]
open-webui:
image: ghcr.io/open-webui/open-webui:main
container_name: open_webui
restart: unless-stopped
ports:
- "127.0.0.1:3000:8080"
volumes:
- open_webui_data:/app/backend/data
environment:
- OLLAMA_BASE_URL=http://ollama:11434
- WEBUI_AUTH=true
depends_on:
- ollama
volumes:
ollama_models:
open_webui_data:
4. Hardware Sizing & VRAM Rules of Thumb
Model Size
Quantization
Minimum VRAM / RAM
Recommended Hardware
7B / 8B (Llama 3.1 / Qwen 2.5)
Q4_K_M (GGUF)
6 GB VRAM
RTX 3060 / Apple M1 16GB
7B / 8B
FP16 / AWQ 4-bit
8–16 GB VRAM
RTX 4070 / RTX 3090
14B / 32B (DeepSeek / Qwen)
Q4_K_M / AWQ
16–24 GB VRAM
RTX 3090 (24GB) / RTX 4090
70B (Llama 3.3)
Q4_K_M (GGUF)
48 GB RAM / VRAM
2x RTX 3090 (NVLink) or Mac Studio 64GB
70B
FP8 / AWQ
48–80 GB VRAM
2x RTX 4090 or 1x A100 (80GB)
5. Production Hardening Checklist
Authentication First: Never expose unauthenticated Ollama (11434) or vLLM (8000) ports to the public internet. Anyone finding the IP can drain your compute or inject prompts.
Reverse Proxy & SSL: Wrap your Open WebUI behind Traefik or Caddy with Let's Encrypt certificates.
Model Weights Cache: Persist ~/.cache/huggingface and ~/.ollama on fast NVMe volumes to prevent re-downloading 20GB+ weights across container restarts.
Conclusion & Architecture Roadmap
Self-hosting your AI stack provides full data governance, zero compliance risk, and fixed compute costs.
Looking for battle-tested Docker Compose templates for local AI, RAG pipelines, vector databases, and reverse proxies? Check out the complete guides on SelfHostStack or grab the Self-Hosted Starter Stack Pack ($29).