Fine-Tune an 8B Model on a 4GB Laptop GPU: Hands-On Guide

Here’s a number that sounds made up: 3.32 GB peak VRAM. That’s what it takes to fine-tune Llama-3.1-8B on a laptop right now – the same 8B model every tutorial said needs 12 GB minimum. A Show HN project called Soup (as of mid-2025) does exactly this: fine-tune an 8B model on a 4 GB laptop GPU without cheating on the math. Once you understand the mechanism, the 12 GB “floor” that every guide repeats stops making sense.

Why every tutorial keeps quoting 12 GB

Full fine-tuning loads all 8B parameters in FP16, plus optimizer states, plus gradients – that’s roughly 60-80 GB of VRAM (per community benchmarks, as of 2025). Tutorials pivot to QLoRA + Unsloth and land on 12 GB as the accepted floor. The math behind that: quantization from 16-bit floats to 4-bit integers cuts memory by ~75%, accuracy barely moves on most tasks – an 8B model shrinks from ~32 GB to ~8 GB. Add LoRA adapter gradients, activations, and a KV cache and you’re at 10-12 GB. That’s the whole story most guides tell.

But a lot of people don’t own 12 GB laptops. RTX 3050 Laptop, GTX 1650 Ti, 3060 mobile at 6 GB – these are the machines actual students and hobbyists have. “Buy a 4090” isn’t a tutorial, it’s a paywall.

Why layer streaming changes the math

When you fine-tune with LoRA, the base model is frozen. Every one of those 8 billion parameters just sits there doing forward passes. Nothing updates. So why is it eating your VRAM?

Turns out it doesn’t have to. Per the Soup README: the frozen base stays out of VRAM entirely. Each decoder layer gets pulled from host RAM (regular system memory) on demand, the forward/backward pass runs through that layer, then it’s discarded. Only the tiny LoRA adapter and the currently-active layer live on the GPU at any moment.

This isn’t the first paper to touch the idea. Practical Offloading for Fine-Tuning LLM on Commodity GPU explored layer-wise CPU-GPU overlapping (arXiv, 2024), and ChunkFT fine-tunes an 8B model on a single 24 GB card using chunk-local operators. What Soup adds is a one-command CLI that packages the technique and pushes the floor to 4 GB – with receipts.

Is this the right trade for every situation? Probably not. If you have a 16 GB card, streaming layers you don’t need to move just burns time. The interesting question is whether the throughput cost stays acceptable as models grow past 8B – and that’s still an open number.

The receipts

From the Soup README (as of mid-2025), on an RTX 3050 Laptop 4 GB:

That “bit-exact” row matters more than the VRAM number. Most memory-saving tricks – aggressive quantization, activation checkpointing shortcuts, mixed-precision hacks – introduce small numerical drift. Soup claims its streamed runs match a fully-resident run exactly, with a verification protocol described in the linked preprint. That’s a strong claim. Worth checking yourself before you trust a 12-hour training run to it.

Running it

Install:

pip install "soup-cli[train]"

Scaffold and train:

soup init --template chat
soup train --config soup.yaml

The YAML is where the switch lives. Model name, dataset, LoRA rank, learning rate – all standard. One line changes everything:

training:
 stream_layers: true # base streams out of VRAM;
 # only the adapter lives on-GPU

Set that, point it at a Llama-3.1-8B checkpoint, and it’ll run on a 4 GB laptop where every other framework throws CUDA OOM on the first forward pass. One practical note: pin a specific commit hash rather than tracking main – this feature is still BETA and the interface may shift.

A note from the community

One comment from the HN thread cuts to why the hardware floor matters beyond benchmarks: hosted large models make headlines, but the vast majority of AI applications don’t need that scale or cost – bringing the tech down to small local models is likely where use cases converge and where ROI actually comes from. Fine-tuning a 3B or 8B on your own narrow task, on your own hardware, often beats paying GPT-4 token costs indefinitely. The bottleneck was hardware access. Layer streaming moves it.

The gotchas nobody’s mentioning yet

Three things to know before you run this on anything that matters:

1. PCIe bandwidth is the hidden variable. Streaming decoder layers from RAM to GPU every step puts the PCIe link on the critical path. The 119.6 tok/s figure comes from one specific laptop configuration. Budget laptops often wire the dGPU on PCIe Gen3 x4 – expect noticeably lower throughput than that number on slower links. Test on your hardware before planning around it.
2. It’s opt-in and BETA. The README is explicit. Pin a commit hash if you’re using this for anything beyond personal experiments.
3. 8B+ is not validated on the maintainer’s hardware. The README states: donations buy GPU time for hardware-gated work – multi-GPU, 8B+ validation, Apple Silicon – that a single 4 GB laptop cannot reach. Planning 13B or 70B on this pipeline? Treat it as unverified and run your own numbers first.

Practical check: Before trusting a full 8B run, do 20 steps with stream_layers: true and 20 steps with stream_layers: false on a 1B model that fits without streaming. If the loss curves diverge, you’ve caught a setup issue in five minutes instead of five hours.

What to actually do with this

Pick a task – not “fine-tune Llama”. A code review assistant that knows your codebase’s conventions. A domain summarizer for legal docs your team actually uses. Something where a fine-tuned 8B on your data beats GPT-4 on your specific inputs because it has seen those inputs.

Clone Soup, feed it 200-1000 examples in the YAML template, run soup train, check the loss curve in the morning. If it runs on 4 GB, it runs – and you own the model.

FAQ

Is layer streaming slower than fitting the model in VRAM normally?

Yes. PCIe transfer time per step is the cost. You get the ability to train at all on 4 GB; a 12 GB card running the same job without streaming will be faster. That’s the trade.

Does bit-exact really mean bit-exact – same loss values as a fully-resident run?

That’s what the README claims, and the linked preprint describes a correctness protocol to verify it. Run the 20-step diff yourself (see the practical check above) before committing to a full 8B fine-tune. Most memory-saving approaches – activation checkpointing shortcuts, mixed-precision hacks – introduce small numerical drift. If Soup genuinely avoids that, it’s the more interesting result than the VRAM number. But “claimed in README” and “verified on your hardware” are different things.

Can I use this on Apple Silicon or multiple GPUs?

Not yet – hardware-gated per the maintainer. Watch the issue tracker.