Run an 80B Qwen in 4.3 GB RAM: The MoE Streaming Trick

The Show HN post about running an 80B Qwen in 4.3 GB of RAM is blowing up on Hacker News right now, and the #1 mistake people are making reading the headline is this: they think 4.3 GB is the whole model.

It isn’t. The full Qwen3-Next-80B-A3B weights, quantized to Q4_K_M, are still 48.4 GB on disk. What fits in 4.3 GB is the resident memory – the slice of weights actually paged in at any given moment. Understanding that difference is the whole point of this article.

The scenario: you saw the headline and want to try it

You have a Mac with 16 GB of RAM. Maybe a Mac Mini M4, maybe a MacBook Air. You keep reading that “35B on iPhone” and “80B on Mac in 4.3 GB” are real. Your instinct is to run ollama pull qwen3-next:80b and watch it work.

That will fail. Or freeze your machine. A community writeup documented exactly this: loading the 35B-A3B through default Ollama pulled 26 GB into memory on a 16 GB machine, triggered 4.3 million swapouts, and timed out after ten minutes without a single token produced. The tool isn’t wrong – its defaults just aren’t built for SSD streaming.

The trick isn’t to shrink the model. It’s to change how the operating system reads it.

Why an 80B model can run in 4.3 GB (the actual mechanism)

Two things have to be true simultaneously. First: Qwen3-Next-80B-A3B is a Mixture-of-Experts model – 512 experts total, only 10 activate per token plus 1 shared expert (per LM Studio’s model page). Total parameters: 80B. Actually computed per token: ~3B. Second: llama.cpp supports memory-mapped loading (--mmap), which maps the file into virtual address space and lets the OS page in only the blocks the model touches – evicting the rest when RAM pressure rises.

Think of it like a reference book too large to fit on your desk. You don’t need the whole thing open – just the pages you’re currently reading. MoE routing is that index: it tells the OS exactly which pages to pull for each token, so the rest stay on the shelf.

On each token generation: the router picks ~10 experts, those specific weight pages get pulled from SSD into RAM, the rest sit on disk. Next token, different experts, different pages. The resident set stays small because sparsity is baked into how MoE works – not a workaround, a structural property.

Apple’s unified memory makes this smoother on Macs. No CPU→GPU copy step – the Metal GPU reads the same pages the CPU just faulted in.

Setup: getting this running on a 16 GB Mac

The path with the fewest gotchas is llama.cpp built from source. Ollama can do this but its defaults fight you (as the 26 GB / 4.3M swap example shows).

# Build llama.cpp
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build -DGGML_METAL=ON
cmake --build build --config Release -j

# Download the Q4_K_M quant (~48 GB, one time)
huggingface-cli download Qwen/Qwen3-Next-80B-A3B-Instruct-GGUF 
 Qwen3-Next-80B-A3B-Instruct-Q4_K_M.gguf --local-dir ./models

# Run with mmap explicitly enabled
./build/bin/llama-cli 
 -m ./models/Qwen3-Next-80B-A3B-Instruct-Q4_K_M.gguf 
 --mmap 
 -c 8192 
 -ngl 99 
 -p "Explain MoE routing in one paragraph."

Two flags matter. --mmap is the mechanism – it tells llama.cpp to memory-map the GGUF instead of reading it into RAM. -c 8192 caps context at 8K, which keeps the KV cache small enough that the resident set really does stay near a few gigabytes. Drop that cap and the KV cache starts competing with the expert pages for the same RAM.

Before you start: On a 16 GB Mac, close Chrome and Slack. The moment other apps push macOS to swap, the OS starts fighting llama.cpp for pages and tok/s collapses. The memory isn’t the problem – the memory contention is.

The iPhone side: how 35B runs on a phone

“6 GB on iPhone 14 Pro” sounds promising. Turns out iOS only gives ~3 GB of that to a single app process, even with the expanded-memory entitlement – a detail buried in the llama-ios project notes. Fitting weights in RAM on a phone is a dead end past toy-sized models. Streaming is the only route that scales.

The current best-known iOS path is MLX Swift. Apple’s framework shares memory between the CPU and Metal GPU through unified memory. A SwiftUI reference app demonstrates Qwen 3.5 running in airplane mode at over 22 tok/s for text (as of mid-2025, per BetterStack community testing). For weight-streamed large MoEs, Daniel’s research prototype goes further: a custom Metal pipeline in Objective-C streams a 397B-parameter Qwen3.5 from the SSD with only ~5.5 GB resident – covered by TweakTown’s analysis.

“35B on iPhone” is not weights-in-RAM. It’s a custom Metal kernel plus SSD streaming plus MoE sparsity working together. If you want to try it today without writing Swift, apps like DeviceAI expose smaller Qwen variants on the App Store; the 35B streaming setup remains a research demo you build yourself.

Advanced: making the streaming actually fast

The naive setup is slow. Daniel’s starting baseline: 0.28 tok/s on the streamed 397B model. After 90+ rounds of optimization on an M3 Max 48 GB, that reached 5.74 tok/s – roughly 20× faster, but still nowhere near what dense-in-RAM inference feels like for interactive use.

Three levers:

That last row is the trap most people miss. Even with model weights streaming from SSD, the KV cache lives in RAM – always. Long conversations eat into the memory budget you’re relying on for the active expert set. Interactive coding at 8K-16K context? Works well. Long-document summarization at 128K? You’ll feel it in both tok/s and system responsiveness.

The honest limitations nobody in the hype thread is naming

This is real, and it’s cool. The trade-offs the headline hides:

• You still need the full file on disk. Q4_K_M is 48 GB. Q8_0 is 84.8 GB. Plan for storage before you plan for RAM.
• SSD wear is a real concern. One of the top comments on the HN thread flags this directly – a long inference session reads gigabytes of weight pages per minute. Consumer SSDs are rated for substantial read cycles, but sustained random reads at this scale generate heat and stress the controller.
• Cold start is painful. The first response after loading is slow because no experts are cached yet. A warm-up prompt helps.
• tok/s does not compete with dense-in-RAM. On the same hardware, a 30B dense model fully in RAM beats a streamed 80B on interactive latency almost every time. You’re trading speed for capacity.

So who is this for? Not everyone. It’s for people who need the capability of a bigger model occasionally, on hardware that couldn’t otherwise touch it – not for anyone building latency-sensitive apps or running the model on a loop.

FAQ

Is the 4.3 GB claim actually accurate, or is it marketing?

Accurate – for peak resident memory during inference with --mmap enabled on an MoE model. Not accurate as a description of storage, and not a promise about speed.

Can I run this on a Mac with only 8 GB of RAM?

For the 80B-A3B, 8 GB is too tight in practice. Here’s the specific reason: the KV cache, OS working set, and resident expert pages together will exceed 8 GB once context grows past a few thousand tokens. Community testing consistently puts 16 GB as the real minimum for the 80B streaming setup. If you have 8 GB, the 35B-A3B class with a very short context cap is a better starting point – and even then, close everything else first.

Why not just use a smaller dense model instead?

Often that’s the smarter call. Chat, short coding tasks, quick Q&A – a Qwen 3 9B or 14B dense model in RAM will respond faster on the same Mac. The reason to reach for the streamed 80B is when the reasoning quality gap actually matters: Qwen3-Next-80B (released September 11, 2025) benchmarks against much larger dense models, and for genuinely hard problems that gap is real. You’re paying in tok/s to get there.

Next step: download the Q4_K_M GGUF, build llama.cpp with Metal support, and run one prompt with --mmap -c 8192 before deciding whether the trade-off works for your machine. You’ll know inside thirty seconds.