GGUF Inference with llama.cpp: A Real Deployment Guide (2026)

The number one mistake with GGUF inference on llama.cpp isn’t picking the wrong quantization. It’s turning on --flash-attn on with a quantized KV cache, watching the server start cleanly, and never realizing your attention just fell back to CPU. Issue #24485 documents prefill slowdowns of 25-45x when the build lacks -DGGML_CUDA_FA_ALL_QUANTS=ON. No warning. No log line. Just a slow server that looks fine.

This guide reverse-engineers a working deployment from that failure mode: build the right binary, run the right flags, and verify the fast path is actually engaged. Latest build tag as of July 27, 2026 is b10154 (releases page) – the project ships continuous build-tagged releases and there are no semantic version numbers to track.

System requirements (minimum vs recommended)

llama.cpp runs on a Raspberry Pi and on an 8×GPU server. That flexibility is the point – but GGUF inference has a floor set by your model and quantization, not by the binary itself.

The VRAM figure is where most people underestimate. Weights for a 7B Q4_K_M sit around 4.1 GB – but the KV cache and compute buffers at a large context can push total usage several GB higher. The CUDA driver also reserves a chunk that nvidia-smi doesn’t show as “used”. Plan for the context, not just the weights.

Pick your install path: Docker or source

Two paths work. Docker is faster to get running and easier to pin to a specific build. Source gives you every kernel flag and backend. Pick one – don’t mix them and expect a clean debug experience.

Docker (recommended for most). The official registry is ghcr.io/ggml-org/llama.cpp (the project moved from ggerganov/llama.cpp to ggml-org/llama.cpp in 2024). Tags include :server, :full-cuda (CUDA 12), :full-cuda13 (CUDA 13), and :server-vulkan – all documented in the official Docker docs. One caveat: GPU-enabled images are not tested by CI beyond being built. If you need a specific CUDA, ROCm, or MUSA configuration, you’ll need to build the image locally.

Source (recommended if you’re on AMD ROCm, Apple Metal, or need FA_ALL_QUANTS). Clone, configure with CMake, build.

# Clone (shallow - the full history is huge)
git clone --depth 1 https://github.com/ggml-org/llama.cpp
cd llama.cpp

# Linux + NVIDIA - FA_ALL_QUANTS is the flag most tutorials miss
cmake -B build 
 -DGGML_CUDA=ON 
 -DGGML_CUDA_FA_ALL_QUANTS=ON 
 -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j$(nproc)

# macOS Apple Silicon (Metal is default, but be explicit)
cmake -B build -DGGML_METAL=ON
cmake --build build --config Release -j$(sysctl -n hw.physicalcpu)

# CPU-only (Linux, Windows via MSVC, or any box without a GPU)
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j

-DGGML_CUDA_FA_ALL_QUANTS=ON is not in most tutorials. Without it, the moment you use --flash-attn on with any quantized KV cache type, you’re on the slow path – and nothing tells you.

First run: download a GGUF and start inference

Grab a model. As of mid-2026, over 60% of quantized models on Hugging Face ship in GGUF format – so you almost certainly don’t need to convert anything. For a first run, a small model keeps the feedback loop tight.

# Install the HF CLI once
pip install -U "huggingface_hub[cli]"

# Download a 3B-class model to ./models
mkdir -p models
huggingface-cli download bartowski/Llama-3.2-3B-Instruct-GGUF 
 Llama-3.2-3B-Instruct-Q4_K_M.gguf 
 --local-dir ./models

# Run interactive CLI (source build)
./build/bin/llama-cli 
 --model ./models/Llama-3.2-3B-Instruct-Q4_K_M.gguf 
 --n-gpu-layers 99 
 --ctx-size 8192 
 --flash-attn on 
 -p "Write a one-line description of a black hole."

Two notes. Older guides reference ./main – that binary was renamed to llama-cli in mid-2024, so anything pointing at ./main is stale. And --n-gpu-layers 99 offloads all layers to GPU; drop this number if you hit CUDA out-of-memory errors, or use the --fit flag (available in recent builds) and let llama.cpp calculate a safe layer count automatically.

Docker equivalent

docker run --rm -p 8080:8080 --gpus all 
 -v $(pwd)/models:/models 
 ghcr.io/ggml-org/llama.cpp:server-cuda 
 --model /models/Llama-3.2-3B-Instruct-Q4_K_M.gguf 
 --host 0.0.0.0 --port 8080 
 --n-gpu-layers 99 --ctx-size 8192 --flash-attn on

The KV cache trap nobody writes about

Quantizing the KV cache saves VRAM. It also has two silent failure modes that most tutorials ignore.

Trap 1: asymmetric K/V types on AMD. Turns out discussion #22411 documents something genuinely surprising: on HIP/ROCm, -ctk q4_0 -ctv q4_0 (symmetric) hits the fused Flash Attention kernel, but -ctk q4_0 -ctv f16 (asymmetric) drops to a non-fused fallback – slower, no warning, no log message.

Trap 2: quantized KV without FA_ALL_QUANTS on CUDA. Same class of bug on NVIDIA. Build the standard way and try to save VRAM with --cache-type-k q8_0 --cache-type-v q4_0? Back on CPU attention. The server won’t tell you.

Pro tip: After starting llama-server, send a 4-token prompt and watch the timings. If prompt eval time is more than ~50 ms/token on a modern GPU, your attention path collapsed to CPU. Rebuild with -DGGML_CUDA_FA_ALL_QUANTS=ON or switch to matched K/V types.

Silent failures are the hardest class of bug to diagnose in a local inference stack – not because the fix is complicated, but because the system gives you no signal that anything went wrong. A server that starts cleanly and produces correct output, just slowly, looks healthy by every normal measure. That’s what makes the FA fallback so expensive: it’s invisible until you think to measure it.

Verify the install actually works

Three quick checks. Each takes seconds.

1. Version and backends:./build/bin/llama-cli --version – prints the build tag (e.g., b10154) and compiled backends (CUDA/Metal/Vulkan/BLAS).
2. GPU visibility (Docker + CUDA):docker run --rm --gpus all ghcr.io/ggml-org/llama.cpp:server-cuda nvidia-smi – your GPU should appear inside the container.
3. End-to-end completion: With llama-server running, hit the OpenAI-compatible endpoint:

curl -s http://localhost:8080/v1/chat/completions 
 -H "Content-Type: application/json" 
 -d '{"messages":[{"role":"user","content":"ping"}],"max_tokens":16}'

Benchmark to sanity-check against: an RTX 4090 running a 13B Q4_K_M should hit 35-45 tok/s. CPU-only? Around 8 tok/s. At 5 tok/s on a 4090, GPU offload isn’t happening – confirm layer placement with nvidia-smi during an active request.

Common errors and real fixes

These are the ones that hit real deployments, not textbook cases.

• CUDA OOM even with VRAM free. The reported “free” number lies once driver reservation is counted. Fix: drop --ctx-size, enable --flash-attn on (which reduces KV cache memory per token), or use --fit so llama.cpp recalculates a safe layer count automatically.
• Windows MSVC: error C2039: 'system_clock' is not a member of 'std::chrono'. Not your compiler. Per issue #1963, the fix is updating the llama.cpp submodule inside llama-cpp-python – reinstalling Build Tools won’t help.
• cmake fails with “CUDA not found”. The CUDA toolkit isn’t on PATH. Add the CUDA bin directory to your PATH (typically /usr/local/cuda/bin on Linux) and re-run CMake from a clean build directory.
• Server runs but latency is poor. Layers didn’t land on the GPU. Bump --n-gpu-layers, confirm with nvidia-smi during a request.
• CPU inference is slow despite a good CPU. You over-threaded. The --threads flag should match your physical core count – setting it higher causes contention and hurts performance.

Upgrade and uninstall

No semantic versions to migrate between. Upgrading is a rebuild.

# Source: pull, reconfigure, rebuild
cd llama.cpp
git pull
cmake -B build -DGGML_CUDA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON
cmake --build build --config Release -j$(nproc)

# Docker: pull the new tag and recreate the container
docker pull ghcr.io/ggml-org/llama.cpp:server-cuda
docker stop llama-server && docker rm llama-server
# ...then docker run again

For production: don’t use :latest. Pin to a specific build tag like :b10154-cuda – the server HTTP API and CLI flags do change between builds. The ./main → llama-cli rename is the loudest example, but flag changes happen more quietly.

Uninstall. Source: rm -rf llama.cpp. Docker: docker rm -f llama-server && docker rmi ghcr.io/ggml-org/llama.cpp:server-cuda. Your ./models directory is separate – delete it too if you’re done.

FAQ

Do I need to convert models myself, or just download GGUF files?

Download. Nearly every open model has a community GGUF upload within days of release. Convert with convert_hf_to_gguf.py only when a model ships exclusively as safetensors and no one has uploaded a GGUF yet.

What quantization should I actually pick?

Q4_K_M first. A 2.4 GB F16 1B model compresses to 668 MB – 72% smaller – and quality holds up well at that size. Move to Q5_K_M or Q8_0 only after you can measure a quality difference on your own prompts with the VRAM to spare. Below Q4 (Q3, Q2, IQ2_XS) is mainly worth considering for 30B+ models where the goal is squeezing them into consumer VRAM at all.

Is llama.cpp better than Ollama or LM Studio?

They’re not really competitors. Ollama and LM Studio are wrappers around the same ggml tensor library that llama.cpp uses directly – so the inference engine underneath is the same. What you’re choosing is how much control you want. llama.cpp directly means access to every flag: -DGGML_CUDA_FA_ALL_QUANTS=ON, ROCm builds, Vulkan, SYCL, fine-grained KV cache control. The wrappers give you a GUI and simpler setup, at the cost of not being able to reach those levers. If you’re reading this guide, you probably already know which side of that line you’re on.

Next step: rebuild your existing llama.cpp with -DGGML_CUDA_FA_ALL_QUANTS=ON, restart llama-server with --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0, and check the prompt eval time. If it dropped by an order of magnitude, you were on the CPU fallback the whole time.