QWE AI Academy Here’s the take nobody wants to say out loud: the “you need a $2,000 GPU to run local LLMs” era ended quietly in spring 2026. Not because GPUs got cheaper – because Gemma 4 26B is a Mixture-of-Experts model, and MoE accidentally rescued every dusty server sitting in a closet since 2016. If you have an old Xeon and 128 GB of RAM, you now have a legitimate local LLM box.
The proof is a viral write-up on point.free where the author ran Gemma 4 26B on a single 2016 Xeon, no GPU, at reading speed – roughly 5 tokens per second, which is around the pace of human reading. This guide walks through how to actually replicate it.
The short version
You do NOT use Ollama for this. You do NOT use stock llama.cpp. You use a fork called ik_llama.cpp, a 4-bit GGUF quant of Gemma 4 26B, and roughly two dozen CPU-tuning flags. On a decade-old Xeon with DDR3 that gets you ~5 tok/s. On a modern DDR4 desktop, more. That’s the whole story – the rest of this article is the “why” and the “how,” plus the traps.
Why this even works (30 seconds of theory)
Gemma 4 26B isn’t really a 26B model when it runs. Google’s April 2026 announcement describes it as a Mixture of Experts with only 3.8 billion of its 25.2 billion parameters active during any single forward pass. Every token you generate touches roughly one-seventh of the weights.
That matters because LLM decoding on CPU is memory-bound, not compute-bound. Each token requires pulling model weights from RAM into cache repeatedly, and – turns out – the E5-2620 v4’s DDR3 bus delivers roughly one-fifth the bandwidth of current laptop memory. The CPU spends most cycles stalled, waiting for the next weight block. Move fewer bytes per token, get more tokens per second. MoE moves fewer bytes. That’s the whole trick.
For contrast: a CEUR-WS study on CPU-only llama.cpp deployment benchmarked a similar-era Xeon E5-2695 v2 at roughly 2 tokens/sec on a dense 24B model at Q4. The 26B MoE hitting ~5 tok/s on comparable hardware is the architecture doing what it was designed to do – not hardware magic.
Why not just ollama run gemma4?
Every mainstream Gemma 4 tutorial ends with that one-liner. Works on a 2024 MacBook. Does not work here.
Two independent write-ups reached the same wall: Ollama and default llama-cli binaries target GPU workloads and expose almost no runtime switches for CPU-only paths. The knobs don’t exist in those tools – and the knobs are the entire point. Both authors bypassed Ollama and standard llama-cli builds entirely to reach usable inference speeds on DDR3.
The fix is ik_llama.cpp, a fork by ikawrakow. Its README (as of July 2026) describes it as a llama.cpp fork with better CPU and hybrid GPU/CPU performance, new SOTA quantization types (IQK, Trellis), first-class Bitnet support, fused MoE operations, and tensor overrides. Gemma 4 architecture support landed in PR 1581; MTP decoding for Gemma 4 came in PR 1744.
Before you tune anything: build ik_llama.cpp with
llama-sweep-benchfirst. The built-in benchmark program gives repeatable output that will actually tell you which flags help on your specific system. Guessing is a great way to waste an afternoon.
The walkthrough
Requirements: x86 CPU with AVX2 (any Xeon E5 v3 or newer qualifies), at least 24 GB RAM free after your OS, roughly 18 GB disk, and Linux. Windows works but adds friction.
Step 1: Build ik_llama.cpp for CPU only
git clone https://github.com/ikawrakow/ik_llama.cpp
cd ik_llama.cpp
cmake -B build -DGGML_NATIVE=ON -DGGML_CUDA=OFF -DGGML_BLAS=OFF
cmake --build build --config Release -j$(nproc)The -DGGML_CUDA=OFF flag matters – Unsloth’s docs call this out specifically for CPU-only inference. GGML_NATIVE=ON tells the compiler to use every instruction your CPU actually supports. On a 2016 Xeon that’s AVX2 and FMA, not AVX-512. Don’t skip it.
Step 2: Download a 4-bit Gemma 4 26B quant
huggingface-cli download unsloth/gemma-4-26B-A4B-it-GGUF
--local-dir ./gemma4-26b
--include "*Q4_K_M.gguf"Q4_K_M is the sweet spot for CPU inference here. Higher bit-widths (Q6, Q8) move more bytes per token – exactly the thing killing performance on DDR3 bandwidth. Lower (Q2, Q3) starts eating output quality. Community reports put the Q4_K_M file at roughly 15-16 GB, though this may vary by the specific quant version you pull.
Step 3: Launch with the right flags
./build/bin/llama-server
-m ./gemma4-26b/gemma-4-26B-A4B-it-Q4_K_M.gguf
--ctx-size 8192
--threads 8
--n-gpu-layers 0
-fmoe -rtr -amb 512
-b 512 -ub 512-fmoe enables fused MoE operations – ik_llama’s headline feature for models like this. -rtr repacks tensors in RAM to a row-interleaved format that’s friendlier to CPU cache lines. --threads should match your physical core count, not logical: hyperthreading generally hurts memory-bound workloads. Then open http://127.0.0.1:8080 and you’re chatting.
Edge cases nobody warns you about
This is where the “5 minute setup” tutorials fall apart.
The -rtr flag is a foot-gun if you ever add a GPU
Read this carefully. The ik_llama.cpp README warns: for hybrid CPU/GPU inference on MoE models, do NOT use -rtr unless you understand the consequences. It repacks tensors in RAM to row-interleaved format – and k-quants like Q4_K have no CUDA row-interleaved implementation. Result: matrix multiplications that should run on the GPU stay on CPU instead, even with GPU offload active. Pure CPU-only? Use -rtr. Planning to add a GPU later? Remove the flag before you do, or you will make things slower.
ik_llama.cpp is not always faster than mainline
The fork’s whole pitch is speed. In practice it depends on your exact model, quant, and hardware. A GitHub issue filed in April 2026 (issue #1699) documented ik_llama.cpp running roughly 2x slower on prompt processing and 1.5x slower on token generation than mainline llama.cpp – for Qwen3 MoE at IQ4_XS on a Ryzen 7 5800X + GTX 1660 Super. That’s a specific combination, not a general verdict, but it means: benchmark both before committing. Don’t assume the fork wins.
Context length destroys your token rate
The Gemma 4 26B MoE supports up to 256K context per its HuggingFace model card. On a DDR3 Xeon, you cannot use anywhere near that. KV cache grows linearly with context, and every token generation reads the whole cache. Start at 4096. Go up only if you need to.
Is this actually useful, or a party trick?
Honest answer: for live interactive chat, 5 tok/s sits at the ragged edge of comfortable. You’ll notice the wait. For batch work – overnight document summarization, offline coding assistance, RAG pipelines where latency is irrelevant – it’s genuinely good infrastructure.
The bigger point isn’t the tokens per second figure. A class of hardware everyone had written off – cheap secondhand Xeons on AliExpress boards, corporate cast-offs, homelab pizza boxes – just became viable inference infrastructure. That’s the shift worth paying attention to. MoE didn’t just give Google a more efficient model; it accidentally reopened the hardware market for people who can’t spend $2,000 on a GPU.
FAQ
Do I need 128 GB of RAM like the viral post?
No. Community reports suggest 32 GB is enough at 4K context, with headroom for the KV cache and OS. The 128 GB machine in the original post has comfortable headroom, but it’s not the minimum.
Will this run faster if I add a cheap old GPU like a 1080 Ti?
Sort of – and it’s more nuanced than the YouTube comments suggest. On an MoE model, the right approach is offloading only the always-active parameters (attention layers, embeddings) to the GPU while leaving routed experts in CPU RAM. ik_llama.cpp’s tensor-override feature is built for exactly this split. The catch: remove -rtr before you do it, or you’ll hit the k-quant CUDA problem described above. Community estimates put token generation improvement at maybe 1.5-2x – not 10x, because the DDR3 memory wall on the CPU side doesn’t disappear just because you added a GPU. Prompt processing sees a bigger gain than generation speed.
Why not just use the Gemma 4 E4B model instead – it’s tiny?
You could, and on the same hardware it’ll fly. But here’s the actual tradeoff: the 26B model ranked #6 on the Arena AI text leaderboard as of April 2026, per Google’s Gemma 4 announcement – that’s frontier-adjacent reasoning quality. E4B is impressive for its size category, but it’s a different capability tier. If you want responses that compete with much larger closed models, 26B is the reason you’re doing any of this. If you want maximum speed and quality is secondary, E4B is the right call.
Next step: clone ik_llama.cpp and run llama-sweep-bench against your own machine before touching anything else. Your DDR3 timings and core count matter more than any generic tutorial’s numbers – including these. Post your tok/s in the r/LocalLLaMA thread; that’s where the real comparisons are happening right now.