QWE AI Academy Why self hosting LLM suddenly matters
API invoices creep up. Residency rules get louder. Open-weight models keep closing the gap. Self hosting LLM puts the weights on gear you control – laptop, workstation, rented GPU – so prompts stay on your network and the meter stops ticking per token.
Sounds neat. Then someone binds the port wrong, or three teammates hit the box at once and the “private assistant” becomes either a public free-for-all or a molasses crawl. Most beginner write-ups stop at “install Ollama and chat.” Below: a working local endpoint in minutes, then the traps that actually bite.
Quick context: what you actually run
You pull open-weight checkpoints (Qwen, Gemma, Llama-family, DeepSeek, and friends), load them in an inference engine, expose HTTP. Two engines matter early:
- Ollama – one-command pull + chat + REST on 11434. Fine for one person or a tiny team. MIT-licensed. CPU or GPU.
- vLLM – throughput engine with continuous batching and PagedAttention. OpenAI-compatible on 8000. Real NVIDIA GPU, more setup.
In practice: prototype on Ollama. Past a handful of concurrent callers, move to vLLM.
Hands-on: self hosting LLM with Ollama in under 10 minutes
Linux/macOS install is the official one-liner (Ollama Linux docs):
curl -fsSL https://ollama.com/install.sh | sh
ollama -vWindows: installer from the download page, or Docker. Prefer an explicit loopback publish:
docker run -d -v ollama:/root/.ollama -p 127.0.0.1:11434:11434 --name ollama ollama/ollamaPull a small current tag that fits modest hardware – Qwen2.5 3B-class is enough to prove the pipe:
ollama pull qwen2.5:3b
ollama run qwen2.5:3bType a prompt. /bye exits. Same model over HTTP (first curl failed for me once because I typed 11443 – check the port):
curl http://localhost:11434/api/chat -H "Content-Type: application/json" -d '{
"model": "qwen2.5:3b",
"messages": [{"role": "user", "content": "Explain KV cache in one sentence."}],
"stream": false
}'OpenAI-compatible clients: point base_url at http://localhost:11434/v1. Default local listener needs no key.
Pro tip: start with the smallest tag that still passes your eval set. A sharp 3-8B Q4 often beats a half-loaded 70B that spills into system RAM all day.
Heavier GPU box later? After CUDA and a Python 3.11+ venv, vLLM looks like this (API key set on purpose – unlike stock Ollama):
pip install vllm
vllm serve Qwen/Qwen2.5-7B-Instruct --host 127.0.0.1 --port 8000 --api-key $(openssl rand -hex 16) --gpu-memory-utilization 0.90Hit /v1/chat/completions with any OpenAI SDK. Install constraints and flags live in the vLLM docs (Linux/WSL, NVIDIA compute 7.5+, not native Windows).
Common pitfalls to avoid
The catch is the bind address. Ollama defaults to loopback and ships with no authentication. Guides that say “set OLLAMA_HOST=0.0.0.0 so your phone can reach it” open the full surface – /api/generate, pull, delete. Turns out community/SentinelLABS-style scans in 2026 saw on the order of 175,000 reachable Ollama hosts; most looked accidental. Stay on 127.0.0.1, firewall the port, put an authenticating reverse proxy in front the moment a second machine needs in. Hardening write-ups such as Praesidia’s Ollama/vLLM notes walk the proxy path.
Second trap: VRAM math that only counts weights. As of 2026 sizing guides, 4-bit weights land roughly here: 7-9B → ~5-6 GB; 27-31B → ~17-22 GB; 70B → ~35-40 GB. KV cache still grows with context length × concurrent sequences. Multi-user long context eats the “free” headroom, then you OOM or offload to CPU and watch latency fall off a cliff.
Third: treating Ollama like a multi-tenant server. One interactive user feels great. A small burst of simultaneous requests and throughput folds. Continuous batching plus PagedAttention exist because of that gap – the SOSP 2023 work (arXiv:2309.06180) reported about 2-4× throughput at matched latency by cutting KV fragmentation.
Performance and results you can expect
| Setup | Best for | Rough concurrency | Notes |
|---|---|---|---|
| Ollama + small Q4 model | Laptop / single user | 1-few | Fastest path to first token |
| Ollama + 24 GB GPU | Small team chat | Low | Still not a production server |
| vLLM + same GPU | Shared internal API | Dozens (model-dependent) | PagedAttention + continuous batching |
Tokens/sec swing with quantization, context, and memory bandwidth. Measure on your prompts; blog peaks rarely survive real traffic.
Does the privacy win still feel worth the ops tax when GPU util sits at 15% overnight? No universal answer – only your token volume and compliance list.
When NOT to self-host
Skip self hosting LLM for low or spiky traffic. Skip it when you need frontier closed models (Claude/GPT weights are not public). Skip it if nobody wants drivers, updates, and monitoring on their plate.
As of 2026 cost write-ups, break-even versus APIs sprawls from tens of millions to billions of tokens per month once idle GPUs and engineering time count. Below that band, pay-per-token usually wins on pure dollars. Residency or air-gap requirements flip the call regardless of price.
FAQ
Do I need a GPU to self-host an LLM?
No. Ollama runs on CPU. Expect far fewer tokens/sec; treat ~8-16 GB system RAM as the floor for tiny models.
How much VRAM for a usable 70B model?
Picture a support bot with 8k context and two parallel sessions – weights near the mid-30s GB at 4-bit are not enough by themselves. You want a 48 GB card (or multi-GPU) plus spare for KV, or you offload and crawl. Benchmark your real max context before you buy.
Is self-hosting always cheaper than ChatGPT or Claude APIs?
No – and that myth burns budgets. Cheap only shows up at high, steady utilization after labor. Individuals and small teams usually pay more in GPU + power + babysitting than in API tokens until volume is large and predictable. Privacy or license control can still justify the box when the spreadsheet prefers the API.
Next: run the install, pull qwen2.5:3b (or the smallest Qwen/Gemma tag that fits), fire one curl at localhost. Then ss -tlnp | grep 11434 – anything other than 127.0.0.1, fix the bind before you do anything else.