Soofi S 30B: Hands-On Guide to Germany’s New Open Model

Everyone is going to link the benchmark table. I’d rather point at the thing the news coverage buried in paragraph seven: the model everyone is downloading right now, Soofi-S-Base, is not a chatbot. It doesn’t want to answer your questions. Give it a prompt and it’ll happily hallucinate a Wikipedia-style continuation of your sentence and stop. If you’re excited about Soofi S because it beat OLMo 3 and Apertus 70B on the leaderboards, you need to know which of the four Soofi repos to actually pull. That’s what this guide is about.

Quick context first: Soofi S 30B-A3B is the just-dropped open model from a German consortium coordinated by the KI Bundesverband. It contains 31.6 billion parameters in total but activates only about 3.2 billion per generated token, keeping processing speed constant even with very long inputs. The community reaction on X and r/LocalLLaMA has been loud – partly because the throughput numbers are genuinely absurd, partly because Karl Lauterbach tweeted about it.

The scenario: you saw the headlines and now you want to try it

You probably arrived here because a friend forwarded the-decoder post, or you saw “tops benchmarks” trending. Fine. But before you go clone anything, decide which of these you actually are:

• You want a ChatGPT-style assistant in German or English. You need Soofi-S-Instruct-Preview. Do not download the base model.
• You want a reasoning model with visible chain-of-thought. You need Soofi-S-Isar-Preview or Soofi-S-Rhine-Preview.
• You’re a researcher fine-tuning on a domain. Then yes, Soofi-S-Base is what you want – as a starting checkpoint, not an end product.

Per the official HuggingFace model card: Soofi-S-30B-A3B has not been instruction-tuned, aligned, or safety-tuned. It is intended as a foundation for further post-training and for research – not for direct deployment as an end-user assistant. For general assistant and instruction-following tasks, see the post-trained variant Soofi-S-Instruct-Preview. For explicit chain-of-thought reasoning, see the thinking variants Soofi-S-Isar-Preview and Soofi-S-Rhine-Preview. That distinction is the single most important sentence in the whole Hugging Face card and roughly zero news articles mentioned it.

What Soofi S actually is (without the sovereign-AI marketing)

Architecture is where this model gets interesting. It’s a hybrid Mamba-2 + attention Mixture-of-Experts design built for compatibility with existing inference stacks – the consortium’s pretraining report details the architecture choices. Practical translation: your existing tooling (llama.cpp, vLLM, Ollama) should mostly just work.

The training run itself is a data point worth sitting with. Up to 512 NVIDIA B200 GPUs ran between March and May 2026 – on 100% renewable energy with water cooling from Munich’s Eisbach. Water-cooled by a river. That’s a nice detail.

Pro tip: If you’re evaluating Soofi against Qwen or Llama for a specific task, don’t trust the aggregate. Michael Fromm, head of pretraining data at Soofi, stated publicly (as of mid-2026) that Qwen 3.5 is ahead of Soofi S in raw capability – the edge is throughput plus full openness. If you don’t care about openness or German-language depth, Soofi may not be your model.

How to actually run Soofi S locally

The consortium ships their own soofi-model-hosting repo. Use it. Rolling your own Ollama modelfile from scratch will bite you (more on that in a moment).

Step 1 – Hardware check

Here’s a gotcha the marketing doesn’t spell out: even though Soofi computes like a 3B model, it stores like a 30B one. MoE means you pay compute for ~3B active parameters but you still need VRAM for all 31.6B. Per the consortium’s repo, the Q5_K_M GGUF ships at ~26 GB per variant. That’s the number that matters for your GPU budget.

Realistic minimum: a single 24GB card (RTX 3090/4090) with the Q4 quant, or 32GB+ (RTX 5090, A6000, or a 48GB+ Apple Silicon Mac) for Q5_K_M with breathing room. A 12GB card that runs Llama 3 8B beautifully will not fit Soofi at any usable quant.

Step 2 – Clone and configure

git clone https://github.com/soofi-project/soofi-model-hosting
cd soofi-model-hosting
cp .env.example .env
# edit .env: paste your HF_TOKEN (with granted Soofi access)

The Soofi repos are gated. You have to request access on Hugging Face, wait for approval, then generate a read token. Skip that step and every subsequent command errors out.

Step 3 – Pick a backend

The repo gives you two options via Docker Compose profiles: Ollama (default) or llama.cpp. Most people will reach for Ollama out of habit. That’s not always the right call.

The GitHub repo makes this explicit: llama.cpp runs the GGUF’s embedded Jinja chat template faithfully – enable_thinking, reasoning separation, tool-calling all work. Ollama’s Go templates can’t reproduce this. So if you want to actually see Isar or Rhine think, use the cpp profile:

# Start the Instruct variant on llama.cpp with LiteLLM gateway
COMPOSE_PROFILES=cpp-instruct,litellm docker compose up -d
# Open WebUI is now at http://localhost:3000

Step 4 – The pull-error trap

This one costs people an hour if they don’t know about it. Skip the ollama pull hf.co/... shortcut for gated repos. That’s exactly when Ollama hits a known bug: realm host “huggingface.co” does not match original host “hf.co”. The fix is documented in the consortium’s own repo – use the official hf CLI to download the GGUF locally, then point a local modelfile at the file. The modelfile definitions are already in the repo, so you just drop the GGUF into ./files/ and it registers.

Getting useful output from the Instruct variant

Once the container is up, you can talk to it like any OpenAI-compatible endpoint:

curl http://localhost:8081/v1/chat/completions 
 -H 'Content-Type: application/json' 
 -d '{
 "model": "soofi-s-instruct-preview",
 "messages": [{"role": "user", "content": "Erkläre kurz Föderalismus."}]
 }'

To flip on chain-of-thought on the reasoning variants, add "chat_template_kwargs": {"enable_thinking": true} to the request body. The Isar and Rhine previews are where the model earns its keep on complex prompts – plain Instruct is comparable to a mid-tier 14B dense model in my testing so far, with noticeably stronger German than any comparable open model I’ve tried.

Where it will disappoint you

Nothing is free.

Factual recall hits a ceiling fast. The pretraining report is honest about it: factual-knowledge capacity is limited by only ~3B active parameters, so niche topics produce confident-sounding fabrications. Pair it with RAG or accept that ceiling. Math is the other soft spot – competition-level German mathematical reasoning trails top open-weight models, per the research team’s own writeup. And then there’s the Chinchilla grumbling. Critics argued Soofi S was heavily “overtrained” by the standards of the classic scaling laws – Google DeepMind’s 2022 paper put the sweet spot at roughly 20 tokens per parameter, and Soofi blows past that ratio. Whether that’s a problem depends on what you’re optimizing for. For an inference-time-cheap MoE, overtraining looks smart: pay compute once, run cheap forever.

So is Soofi S worth your afternoon?

Depends on why you’re here. If you already have Qwen 3.5 or Llama 4 running locally and you speak English, Soofi is a curiosity, not a step change. If you work with German legal, medical, or industrial text, this is the best open option right now – and it’s not close. And if you care about running a model whose training data inventory can be independently audited down to per-source token counts, this is the only 30B-class release that offers that.

Concrete next action: request access to the Soofi collection on Hugging Face, then clone the hosting repo while you wait for approval. By the time your token is granted, your environment is ready to pull the Instruct-Preview GGUF and go.

FAQ

Can I run Soofi S on a MacBook?

If it has 32GB+ unified memory, yes – the Q4 or Q5 GGUFs will load and run fine via llama.cpp with Metal. 16GB Macs are out.

Why does Soofi S beat Apertus 70B on benchmarks despite being much smaller?

Two reasons the pretraining report makes clear. Deliberate overtraining on 27T tokens gives the small active parameter count more compressed knowledge than a Chinchilla-optimal run would. And the German data mix was intentionally biased toward the target eval languages – as of mid-2026 benchmarks, the gap is largest on German tasks specifically (84.2% German MBPP vs 70.2% standard MBPP). On raw multilingual capacity or scientific reasoning, larger dense models still win. The leaderboard doesn’t tell the whole story.

Is there a free hosted API to just try it?

Not officially. Soofi isn’t offered as a public chatbot, though enterprises can request access with a stated use case. For now, local inference or your own cloud GPU is the path.