Install Ollama v0.31.1: Local AI Chatbot Setup Guide

Here’s the question I keep seeing on Reddit and in GitHub issues: “I installed Ollama on a machine with a decent GPU. Why is my local AI chatbot crawling at CPU speed?” Nine times out of ten, the install “worked” but silently fell back to CPU. This tutorial walks you through installing Ollama v0.31.1 – the current stable release as of early July 2026 – in a way that avoids that trap.

Ollama is the easiest way to run open-weight models like Gemma 4, Qwen, DeepSeek, and gpt-oss on your own hardware. One binary, one daemon, a REST API on port 11434. The install itself takes 60 seconds. Making sure it uses your GPU takes a little more attention.

Pick your version first (and why v0.31.1 matters)

Current stable: v0.31.1, with v0.31.2-rc1 out for testing as of July 6, 2026. The 0.30 line – which shipped May 13, 2026 – was the significant rewrite: it pairs the new MLX engine on Apple Silicon with continued llama.cpp support, so the same binary runs well across Mac, NVIDIA, and AMD hardware.

If you’re on Apple Silicon and care about speed, the official release notes document something worth knowing: Gemma 4 on M-series Macs is now nearly 90% faster on average, thanks to multi-token prediction (MTP) that auto-tunes as it runs. No configuration needed – it doesn’t change model output either.

Think of version choice like picking a tire compound before a race. The “latest stable” is the safe all-weather choice. The release candidate is the sticky performance tire – faster in good conditions, but you’re the test driver. For most people: v0.31.1, done.

System requirements (mid-2026 numbers)

RAM minimums from the official quickstart, as of mid-2026:

GPU side (also mid-2026): NVIDIA needs Compute Capability 5.0+ and Driver 550+. AMD needs a ROCm v7 / HIP7-capable driver stack. One thing that catches people: Ollama bundles the CUDA runtime libraries it needs – the only NVIDIA dependency you install manually is the display driver, visible via nvidia-smi. No CUDA toolkit required.

Install Ollama (Linux, macOS, Windows, Docker)

From ollama.com/download:

# Linux - official install script
curl -fsSL https://ollama.com/install.sh | sh

# Linux - manual, if you don't want to pipe curl to shell
curl -fsSL https://ollama.com/download/ollama-linux-amd64.tar.zst 
 | sudo tar x -C /usr

# Windows
# Search winget for the current Ollama package name - it may update between releases
winget search Ollama

# Docker (with NVIDIA GPU)
docker run -d --gpus all -v ollama:/root/.ollama 
 -p 11434:11434 --name ollama ollama/ollama

# Docker (CPU only)
docker run -d -v ollama:/root/.ollama 
 -p 11434:11434 --name ollama ollama/ollama

macOS: grab the .zip from the download page and drag Ollama.app to Applications. The Metal backend for Apple Silicon is baked in – no driver install.

Need to pin an older version? The install script accepts a version override: curl -fsSL https://ollama.com/install.sh | OLLAMA_VERSION=0.5.7 sh. Useful for reproducing bugs or matching a team environment. Documented in the official troubleshooting guide – omitted by essentially every third-party tutorial.

First-time configuration and verification

On macOS and Windows, the desktop app starts the server. On Linux, the installer registers a systemd unit. Confirm it’s alive:

# Version check
ollama --version

# Server health check
curl http://localhost:11434/api/version

# Pull and run a model (~5 GB download for gemma4)
ollama run gemma4

# In another terminal - is the GPU actually being used?
ollama ps

That last one is the check nobody runs. The PROCESSOR column shows 100% GPU, 100% CPU, or a split like 40/60. Paying for a graphics card and seeing 100% CPU? Something is wrong – keep reading.

The API works on the same box, port 11434:

curl http://localhost:11434/api/chat -d '{
 "model": "gemma4",
 "messages": [{ "role": "user", "content": "Summarize quantum entanglement in one sentence." }],
 "stream": false
}'

To expose Ollama to other machines on your network, set OLLAMA_HOST=0.0.0.0 before starting the server. Skip this on public networks.

Windows-specific gotcha: Add your models directory to Windows Defender’s exclusion list. Defender will scan every multi-gigabyte GGUF blob as it downloads – turning a 3-minute pull into a 30-minute crawl. The path is %USERPROFILE%.ollamamodels.

The silent CPU-fallback traps

Every install guide points at the same install command. Almost none tell you what breaks after. Three specific traps show up repeatedly.

1. The install script misreports your GPU. GitHub issue #10883 documented this: a user with an AMD card ran the install script and saw “Nvidia GPU installed” in the output – on a machine with no NVIDIA hardware. The model refused to touch the GPU after that. If you see this, check journalctl -u ollama for the actual detection log and reinstall ROCm.

2. Your AMD card isn’t on Ollama’s supported list. Consumer cards like the RX 6600 (gfx1032) fall back to CPU silently. The fix, confirmed by community troubleshooting:

sudo systemctl edit ollama
# Add under [Service]:
# Environment="HSA_OVERRIDE_GFX_VERSION=10.3.0"
sudo systemctl restart ollama

The 10.3.0 value targets RDNA2 cards (RX 6600, 6700, 6800 series). For other architectures, check the ROCm gfx compatibility list first – the wrong value here can make things worse.

3. You pulled a :cloud variant by accident. New as of May 2026: pulling a :cloud model tag (DeepSeek V4 Flash, DeepSeek V4 Pro, and others) runs the model on Ollama’s hosted servers – not your GPU. The pull finishes in seconds because there’s nothing to download locally. If your “local” chatbot suddenly needs an internet connection, this is why.

One more annoyance: ollama stop <model> does not stop the server. It only unloads a model from memory. To restart the server itself, use systemctl restart ollama – or quit and relaunch the desktop app.

Upgrade and uninstall

The official Linux docs include a step most tutorials never mention: before re-running the install script over an older version, remove the old libraries first.

sudo rm -rf /usr/lib/ollama

Skip this and you can end up with mixed library versions that cause subtle inference failures – not a clean error, just wrong behavior. Then re-run the install command normally.

Full uninstall on Linux:

sudo systemctl stop ollama
sudo systemctl disable ollama
sudo rm /etc/systemd/system/ollama.service
sudo rm $(which ollama)
sudo rm -r /usr/share/ollama /usr/lib/ollama
sudo userdel ollama
rm -rf ~/.ollama # wipes all downloaded models - budget 10-100 GB of space

macOS: quit the app, drag it to trash, delete ~/.ollama. Windows: uninstall from Settings → Apps, then delete the Ollama data folder from your user profile (check the installer’s documentation for exact paths – they can vary by release).

Worth pausing here for a second. The uninstall step is where you actually learn how Ollama stores state. Everything – models, config, conversation history – lives in ~/.ollama. The binary is almost irrelevant. Which means if you’re migrating to a new machine, you’re really just copying that folder. That’s a more useful mental model than thinking of Ollama as software you “install.”

FAQ

Do I need to install the CUDA toolkit separately?

No. Ollama ships its own CUDA runtime. Driver only – verify with nvidia-smi.

Why is my chatbot slow even though ollama ps shows GPU?

The KV cache is probably spilling to CPU. A 14B Q4 model at 32k context can push past your VRAM ceiling – you’ll see 40%/60% CPU/GPU in the PROCESSOR column rather than 100% GPU. Reduce context length in the model settings or pick a smaller quantization. That usually fixes it. On Macs it’s a different story: GPU and system RAM are the same pool on Apple Silicon, so a 70B model on a 16GB M2 will thrash no matter how the percentage reads.

Can I run Ollama entirely offline?

Yes – once models are pulled, Ollama doesn’t phone home for inference. The one exception is :cloud model tags (introduced in 2026), which route to Ollama’s hosted infrastructure. Stick to plain tags like gemma4 or qwen3:8b and you’re fully air-gapped.

Next step: with the daemon running, pull one small model (ollama pull qwen3:8b) and one bigger one that stresses your VRAM, then run ollama ps during inference on each. Both showing 100% GPU? Your install is genuinely working – not just installed.