Deploy MeditronFO Apertus-8B [Latest Open Medical LLM]

You’ll leave with a local OpenAI-compatible endpoint serving MeditronFO’s current model – Apertus-8B-MeditronFO – on your hardware. No cloud keys after the download.

I burned an evening on 2023 Llama-2 Meditron checkpoints before noticing Fully Open Meditron had already taken over for new work. This is the path that actually boots as of early 2026.

What you’re installing (and why the 8B)

Apertus-8B-MeditronFO is the practical member of the MeditronFO line from EPFL LiGHT. Base: swiss-ai/Apertus-8B-Instruct-2509. One-epoch finetune on the clinician-audited Fully Open Meditron corpus (~534k examples) via Axolotl/DeepSpeed. BF16 weights land around 16.1 GB. Context length is 65 536. License: Apache-2.0.

On the MOOVE side, clinician-aligned judges preferred these models about 70% of the time over their bases; they also pitch Medium (8B) as laptop-runnable and Large (70B) as a 2-GPU job. Full write-up sits on arXiv:2605.16215. The 70B exists if you have the iron. Most of us stop at 8B.

Medical LLMs still feel like handing a sharp tool to someone who reads fast – useful, not a license. That gap is why the rest of this page is install mechanics, not clinical advice.

System requirements

Numbers below track the current Hugging Face card and community GGUF builds (as of early 2026). Official docs do not publish a hard system-RAM or CPU-only tokens/sec floor – only GPU VRAM and marketing that Medium “runs on a laptop” once you quantize.

CPU-only runs. It’s slow. Nobody published a clean throughput number on the card or the MOOVE Meditron page.

Official download source

Weights live on Hugging Face. Skip random mirrors.

• Primary: EPFLiGHT/Apertus-8B-MeditronFO – model card
• Family collection: MeditronFO
• GGUF (community): mradermacher/Apertus-8B-MeditronFO-GGUF – Q4_K_M ~5.06 GB, Q5_K_M ~5.81 GB, Q8_0 ~8.57 GB, F16 16.1 GB
• Training/eval code: EPFLiGHT/FullyOpenMeditron

Optional: huggingface-cli login if rate limits or private cache mounts get in the way.

Install and first run (SGLang – recommended)

The HF card points at SGLang. You get /v1/chat/completions without bolting on a separate gateway.

# 1. Environment
python -m venv meditron-env && source meditron-env/bin/activate
pip install --upgrade pip
pip install sglang torch transformers accelerate

# 2. Launch server (first run pulls ~16 GB)
python3 -m sglang.launch_server 
 --model-path "EPFLiGHT/Apertus-8B-MeditronFO" 
 --host 0.0.0.0 
 --port 30000

Docker path (useful on shared boxes):

docker run --gpus all 
 --shm-size 32g 
 -p 30000:30000 
 -v ~/.cache/huggingface:/root/.cache/huggingface 
 --env "HF_TOKEN=${HF_TOKEN}" 
 --ipc=host 
 lmsysorg/sglang:latest 
 python3 -m sglang.launch_server 
 --model-path "EPFLiGHT/Apertus-8B-MeditronFO" 
 --host 0.0.0.0 
 --port 30000

Chat template is already in the card. Temperature 0-0.3 for tight clinical-style answers; raise it only if you want wider differentials on purpose.

Pro tip: In raw transformers, always call tokenizer.apply_chat_template(..., add_generation_prompt=True). Skip it and you get loops or garbage fast – same class of failure legacy Meditron users hit when vocab (32005+) and embed size (32000) disagreed, except now the fix is the template, not a DIY resize.

Alternative: pure transformers (script / notebook)

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "EPFLiGHT/Apertus-8B-MeditronFO"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
 model_id,
 torch_dtype=torch.bfloat16,
 device_map="auto",
)

messages = [{"role": "user", "content": "A 62-year-old woman with three days of dyspnea on exertion and productive cough. Differential diagnosis?"}]
inputs = tokenizer.apply_chat_template(
 messages, add_generation_prompt=True, tokenize=True,
 return_dict=True, return_tensors="pt"
).to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Low-VRAM path: GGUF + llama.cpp

8-12 GB card? Laptop? Pull Q4 or Q5 from mradermacher and serve with llama.cpp (build from the upstream llama.cpp project for your OS – don’t trust random install mirrors).

llama serve -hf mradermacher/Apertus-8B-MeditronFO-GGUF:Q4_K_M
# OpenAI-compatible server prints a port - point clients there

Q4_K_M is ~5.06 GB on disk. That’s the practical pick for most consumer GPUs. Context past ~8K still taxes KV cache; if you OOM after weights fit, shorten context before you chase another quant.

Verify the install works

Once the process says it’s listening:

curl -X POST "http://localhost:30000/v1/chat/completions" 
 -H "Content-Type: application/json" 
 --data '{
 "model": "EPFLiGHT/Apertus-8B-MeditronFO",
 "messages": [{"role": "user", "content": "List three red-flag symptoms for pulmonary embolism. Be concise."}],
 "max_tokens": 256,
 "temperature": 0.1
 }'

JSON back with choices[0].message.content in a few seconds = good. Empty body or CUDA traceback = jump to the errors list.

Common install errors and fixes

1. CUDA OOM – BF16 wants ~16.1 GB just for weights; 12 GB cards die unless you move to Q4_K_M GGUF (~5.06 GB) or multi-GPU. Long context adds KV pressure on top.
2. Garbage tokens / device-side assert – Almost always a skipped apply_chat_template on the new card. Fix the call site.
3. Docker shm / NCCL hang – Keep --shm-size 32g and --ipc=host. No HF cache volume means a full 16 GB re-pull every start.
4. Model not found / glacial first boot – Exact repo id + writable ~/.cache/huggingface. First pull is the long one.

Upgrade from older Meditron / uninstall

Still holding epfl-llm/meditron-7b or 70b (or old TheBloke quants)? Different base (Llama-2 era), different failure modes, different license story. Treat as a separate experiment and reclaim disk:

rm -rf ~/.cache/huggingface/hub/models--epfl-llm--meditron-*

Current install teardown: stop SGLang/llama, drop the venv, optionally purge the EPFLiGHT/Apertus-8B-MeditronFO hub snapshot. Docker: docker rm the container; keep the volume only if you want the cache.

No migration script. Old prompts often still “work” – re-check accuracy yourself.

FAQ

Is MeditronFO safe for real patient decisions?

No. Model card and paper: research / assistive evaluation only. Verify everything against primary sources and licensed clinicians.

Can I run the 70B on a single 24 GB card?

Not in BF16. You’d need heavy quant (quality hit) or multi-GPU/offload. Realistic single-GPU target: 8B + Q4/Q5. 70B wants the multi-GPU layout MOOVE describes.

SGLang vs vLLM vs plain transformers – which should I pick?

SGLang is what the card documents and stands up an OpenAI API with less setup. Plain transformers fits one-off notebooks. vLLM is fine if your stack already centers on it – you’ll wire the model id yourself. Apple silicon or thin laptops? GGUF via llama.cpp usually hurts less than wrestling CUDA builds. Whatever you pick, keep the research-only constraint in the client UI so nobody treats completions as orders.

Next: run the SGLang one-liner (or Q4 GGUF serve), hit curl with a differential you actually know, and note whether the answer smells like guideline-style reasoning. Then decide if you need 70B iron or a RAG layer on top.