QWE AI Academy Every time someone posts “I ran Llama 70B on my laptop,” the replies fill up with the same math: 70B parameters at FP16 = ~140GB of VRAM, and consumer GPUs top out at 24GB. The dream dies in the comments. That’s exactly the wall AirLLM 70B inference with single 4GB GPU tears down – and the project is having a viral moment again after a fresh release bumped it past 21,000 GitHub stars (as of mid-2025).
The catch is it’s not free lunch. AirLLM doesn’t compress the model – it streams it. Below, I’ll show you the setup that actually works, the honest speed you should expect (spoiler: sometimes minutes per token), and the specific gotchas nobody puts in the README summary.
Why AirLLM works – the one-sentence version
Transformer inference only needs one layer’s weights active at a time. Everything else is idle. AirLLM exploits that: it splits the model into per-layer shards on disk, loads a single layer, runs the forward pass through it, frees the VRAM, and pulls the next one.
The peak VRAM footprint becomes one layer + activations, not the whole model. According to community analysis, for a 70B model each layer is roughly 1.6GB – which is why a 4GB card can hold it with room to spare for the KV cache and working tensors.
Think of it like watching a movie by streaming instead of downloading the whole file – you only need the current frame in memory. The trade is bandwidth: every token generation reads the entire model from disk. Sequentially. Every time.
The honest performance ceiling (read this before installing)
Most tutorials copy the marketing line and stop. Here’s the number that matters: community experiments consistently show generation speeds well below 1 token per second for very large models. One Hacker News commenter testing v3.1.0 reported Kimi K3 on an RTX 6000 Ada (48GB VRAM) taking 292 seconds per token. Yes, per token. On a $7,000 GPU.
That’s because the bottleneck stopped being GPU memory the moment you switched to AirLLM. Now it’s your SSD’s sequential read speed. A slower disk turns “slow” into “go make coffee.”
Before you install anything: benchmark your disk. Run
hdparm -Tt /dev/nvme0n1(or your platform’s equivalent). Reading under 2 GB/s sequentially? Expect painful throughput. AirLLM on a spinning HDD is technically possible and practically miserable.
Setup: from pip to first token
Installation is one line. The interesting part is the parameters most people leave at defaults.
pip install airllm
pip install -U bitsandbytes # only if you want 4bit/8bit compression
Minimum working example, adapted from the official README:
from airllm import AutoModel
model = AutoModel.from_pretrained(
"meta-llama/Meta-Llama-3-70B-Instruct",
compression='4bit', # optional - see note below
delete_original=True, # critical - see next section
profiling_mode=True # shows where time is actually spent
)
input_text = ['Explain layer-wise inference in two sentences.']
input_tokens = model.tokenizer(input_text, return_tensors="pt",
return_attention_mask=False, truncation=True, max_length=128)
out = model.generate(input_tokens['input_ids'].cuda(),
max_new_tokens=64, use_cache=True, return_dict_in_generate=True)
print(model.tokenizer.decode(out.sequences[0]))
The compression flag is the first real decision. Block-wise 4-bit quantization (added in v2.0, per the official README) claims up to 3x speed improvement with negligible accuracy loss – so unless you specifically need strict FP16 behavior, turn it on. Beyond that: profiling_mode=True breaks down disk-load time vs. compute time, which tells you immediately whether a faster SSD would help you. And delete_original – that one deserves its own section.
Pitfalls the README buries
Four things I wish someone had told me before the first run.
- Two copies on disk by default. That’s the trap. AirLLM decomposes the model and saves it layer-wise – but it keeps the original alongside the sharded version unless you tell it otherwise. For a 70B model that’s around 120-140GB per copy. The fix is one flag:
delete_original=Truein yourfrom_pretrainedcall, which keeps only the transformed version and cuts storage roughly in half. - Prefetching: the README mentions it as a speedup, but it only fires for AirLLMLlama2. If you’re loading Qwen, DeepSeek, or anything else through the AutoModel path, that ~10% overlap-load-with-compute gain silently disappears. No warning, no error. You just don’t get it.
- Turns out gradient propagation needs the full model in memory simultaneously – which layer sharding can’t provide. AirLLM is inference-only, full stop. Planning to LoRA-tune through it? That path doesn’t exist. Look at QLoRA with bitsandbytes or Unsloth for fine-tuning on limited hardware.
- The first run isn’t inference – it’s preprocessing. AirLLM has to split the model into shards before it can do anything. On large models, that initial pass can be lengthy depending on your disk speed and model size, and there’s no clean progress indicator for the shard creation phase. Don’t mistake silence for hanging.
AirLLM vs. the alternatives (when to use what)
AirLLM isn’t the only way to squeeze big models onto small hardware. It fills a specific slot – and misses the mark for most use cases.
| Tool | Best for | Typical speed on 70B | Compresses model? |
|---|---|---|---|
| AirLLM | Occasional runs on tiny GPUs, testing full-precision behavior | <1 tok/s (bigger models: minutes/token) | Optional (4/8-bit block-wise) |
| llama.cpp (GGUF) | Interactive chat on consumer hardware | Interactive speeds (seconds per response) | Yes (Q4/Q5/Q8 GGUF) |
| Ollama | Beginners wanting a one-command install | Similar to llama.cpp (it wraps it) | Yes (GGUF) |
| vLLM / TGI | Production serving with real GPUs (24GB+) | Production-grade throughput | Optional |
Short version: if 4-bit quantization is acceptable, llama.cpp or Ollama will be dramatically faster on the same hardware. AirLLM wins in one scenario – you need unquantized behavior. Reproducing benchmark results, validating a model card claim, running an architecture that lacks a good GGUF conversion. In those cases, the slow disk-streaming is the price. The model list is broad (Llama 2/3/4, Qwen, DeepSeek V2/V3, Mistral, ChatGLM, Baichuan, InternLM, Phi, Gemma – as of this writing), so architecture coverage probably isn’t the limiting factor.
So who is this actually for?
Honestly? Researchers, curious tinkerers, and people running one-off evaluations. If you want a chatbot that responds in under a minute, this isn’t it. If you want to prove to yourself that a 405B model can produce a token on hardware that costs less than a used bike, AirLLM is genuinely the only tool that does that today without a cloud bill.
FAQ
Do I need an NVIDIA GPU, or does AirLLM work on Mac?
Both work. CPU-only inference landed in v2.10.1 (August 2024, per the official changelog), which covers Mac and any machine without a discrete GPU. Apple Silicon users get Metal acceleration on top of that. Expect performance somewhere between a slow GPU run and pure CPU inference – the specific gap depends on your chip generation and available unified memory.
Can I fine-tune a model through AirLLM?
No – and this trips people up. AirLLM is strictly for inference. Because it only holds one layer in memory at a time, there’s no full computation graph to backpropagate through. For fine-tuning a 70B model on limited hardware, look at QLoRA with bitsandbytes or Unsloth instead. Those actually keep gradients flowing.
Is the “no quantization” claim really true if I use 4-bit compression?
Fair callout. If you leave compression=None, weights stay at full FP16 precision – streamed from disk, nothing quantized. The 4-bit flag, added in v2.0, is strictly opt-in. So the original claim holds for the default path; the speed improvement is available if you want it, not forced on you.
Next step: clone lyogavin/airllm, pick the smallest model you can (start with a 7B for a sanity check, not 70B), and run it with profiling_mode=True. Look at where your seconds go. That single benchmark tells you whether AirLLM is a fit for your disk – and whether you should keep going or reach for llama.cpp instead.