Install Local LLM with Ollama v0.40.3 [2026]

The #1 mistake that wrecks your first local LLM

I watched a friend pull a 70B model the night he discovered local LLMs. His 16 GB laptop froze, fans screamed, and he declared the whole idea broken. The real problem wasn’t the software. He started with a model that needed more memory than he had free after the OS, browser, and Slack.

A local LLM is weights on disk plus a runner that keeps every token on your machine. Most people who actually ship one today use Ollama. As of mid-October 2026 the stable release is v0.40.3. Paste the commands below, then hit the traps happy-path posts skip.

System requirements before you touch the installer

Ollama itself is tiny. Models are not. Check free RAM and disk – not the sticker total.

Per the official quickstart (as of 2026), gemma4:e2b is about a 7.2 GB download and wants ~8 GB available VRAM or unified memory. Bigger context windows take more. Tight free memory? Start smaller. Close the browser zoo.

Download and install Ollama v0.40.3

Official only: ollama.com and GitHub releases. Skip random mirrors.

macOS / Linux (one-liner)

curl -fsSL https://ollama.com/install.sh | sh

Linux usually installs a systemd unit so the server starts on boot. Check it stuck:

ollama -v
# expect: ollama version is 0.40.3

Windows (PowerShell)

irm https://ollama.com/install.ps1 | iex

Or grab the installer from ollama.com/download. Tray icon shows up; ollama lands on PATH.

Docker (secondary path)

Image: ollama/ollama. Tag latest tracked 0.40.3 when this was written.

# CPU
docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

# NVIDIA (Container Toolkit first)
docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

# AMD ROCm
docker run -d --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama:rocm

Then docker exec -it ollama ollama run .... Mac Docker Desktop: no GPU passthrough – use the native app.

Upgrading an older Linux install? Docs say sudo rm -rf /usr/lib/ollama before a manual extract so leftover libs don’t fight the new binary.

First model – size it to free RAM

No config file needed for chat. Server default: http://localhost:11434. Pull something that fits.

ollama pull gemma4:e2b
ollama run gemma4:e2b

Type a prompt. Done? /bye. ollama list shows downloads. ollama ps while chatting – PROCESSOR should mention GPU if acceleration stuck.

OpenAI-compatible URL for Cursor, Continue, or a script: http://localhost:11434/v1. Base URL + any dummy key. No extra flags for basic use.

One change in the 0.40 line (see v0.40.2/v0.40.3 release notes): first run of an older downloaded model may upgrade it in the background for llama.cpp compatibility and keep the original as a backup. Disk use can double until you prune. A later release may auto-prune; until then reclaim space once the new runner looks good.

Verify the install actually works

1. ollama -v → 0.40.x
2. curl http://localhost:11434 → “Ollama is running”
3. ollama run gemma4:e2b "Reply with the single word: pong" → something sensible
4. ollama ps during chat → CPU vs GPU

Step 2 connection refused? Server isn’t up. Linux: sudo systemctl start ollama or run ollama serve in a terminal.

Common install errors and fixes that work

“listen tcp 127.0.0.1:11434: bind: address already in use”
Leftover process or tray app still owns the port. Linux/macOS: sudo lsof -i :11434 then kill -9 <PID>. Windows: netstat -ano | findstr 11434 then taskkill /PID <pid> /F. Free the port first. Reinstall does nothing.

GPU ignored – 1-3 tok/s on CPU
ollama ps stuck at 100% CPU. NVIDIA: nvidia-smi must work; driver 550+ (570+ for compute 5.0-6.2 per the GPU hardware docs). UVM flaky? sudo rmmod nvidia_uvm && sudo modprobe nvidia_uvm. AMD on Linux needs ROCm v7 – older stacks time out on discovery and fall back silent. Apple Silicon uses Metal on its own; if it’s slow you’re probably out of unified memory.

Pull fails or OOM on load
Tag bigger than free RAM/VRAM plus context. Drop to a 2B/4B/e2b-class tag, close other apps, or cut context later with a Modelfile. Partial pulls leave locks – retry or clear incomplete files under the models directory.

Linux path weirdness (Fedora and friends)
Community reports: libraries land under /usr/local/lib while the service expects another layout, so GPU discovery fails. Stop the service, re-run the official install script, or follow manual extract in the Linux docs. Then journalctl -u ollama.

Upgrade and uninstall

Same install command again:

curl -fsSL https://ollama.com/install.sh | sh # Linux/macOS
# or re-run Windows installer / PowerShell line

Docker: docker pull ollama/ollama:latest, recreate the container.

Clean Linux leave (official sequence):

sudo systemctl stop ollama
sudo systemctl disable ollama
sudo rm /etc/systemd/system/ollama.service
sudo rm -r $(which ollama | tr 'bin' 'lib')
sudo rm $(which ollama)
sudo userdel ollama
sudo groupdel ollama
sudo rm -r /usr/share/ollama
# also: rm -rf ~/.ollama if models lived in your home

macOS: quit app, rm -rf /Applications/Ollama.app, sudo rm /usr/local/bin/ollama, rm -rf ~/.ollama. Windows: Settings → Apps → Uninstall Ollama, then delete %USERPROFILE%.ollama for model space.

Once the runner answers in the terminal, wiring it into your editor or a small RAG script is the next move. Don’t skip the free-memory check next time a bigger tag looks tempting.

FAQ

Do I need a GPU for a local LLM with Ollama?

No. CPU works out of the box. It’s slower. That’s the whole answer for most first installs.

Why is my first model so slow even after install?

Check ollama ps while a prompt is running. If PROCESSOR is pure CPU, drivers or ROCm/CUDA discovery failed – fix that before blaming the model. Second common case: the tag forces swap on an 8-16 GB box; gemma4:e2b (or another small tag) is the sanity check. Third: after a 0.40 upgrade path, an old model can spend the first run converting – wait it out, then retest tokens/sec.

How do I expose Ollama to other machines on my LAN?

Default bind is localhost only. Set OLLAMA_HOST=0.0.0.0:11434 (systemd override on Linux, app env on desktop, or Docker -e), restart, call http://<lan-ip>:11434. Keep that port off the public internet unless you add auth and a reverse proxy – local weights do not mean an open unauthenticated API.