Deploy TGI v3.3.7: Docker Install Guide

Install Text Generation Inference (TGI) v3.3.7 with Docker: system requirements, copy-paste commands, health checks, and the errors that actually break first boot.

Morgan Hayes2026-09-217 min readIntermediate

The #1 mistake with text generation inference (TGI) isn’t picking the wrong model. It’s launching the Docker image without the NVIDIA Container Toolkit, without enough shared memory, and without an HF token for gated weights – then chasing a vague ShardCannotStart for half a day.

I did that. Twice. Once on a fresh A10 box, once after blindly copying an old :latest one-liner while the docs still pinned 3.3.5. Here’s the path that actually boots TGI v3.3.7 and stays up.

TGI is Hugging Face’s Rust/Python server for high-throughput LLM serving. As of ~11 Dec 2025 it’s in maintenance mode – minor fixes and docs only. Existing fleets on supported models are fine; new engine work on HF Inference Endpoints is steered elsewhere. On your own GPUs, Docker is still the install path that wastes the least time.

System requirements before you pull anything

Skip this and the container exits in seconds.

Resource	Minimum	Practical
OS	Linux x86_64 (Docker host)	Ubuntu 22.04+
GPU	1× NVIDIA with a recent driver	T4 / A10G / A100 / H100 (CUDA 12.2+ for full attention kernels)
VRAM (FP16 ballpark)	~14 GB for 7B	Headroom above model weights for KV cache; 70B needs multi-GPU or quant
Host	Docker + enough disk for Hub weights	SSD volume for the cache mount; RAM free enough that the host doesn’t thrash during download
Software	Docker + NVIDIA Container Toolkit	Same + HF token for gated models

HF’s NVIDIA notes call out H100/A100/A10G/T4 for the full-kernel path. Other NVIDIA cards still get continuous batching; flash/paged attention may simply not load. AMD is a separate -rocm image and device flag set – don’t mix that recipe with this one.

Confirm Docker can see the GPU before you touch TGI:

docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

No table? Fix drivers/toolkit first.

Official download source (pin the version)

Registry image: ghcr.io/huggingface/text-generation-inference.

github.com/huggingface/text-generation-inference holds releases. Latest tag as of 18 Dec 2025: v3.3.7. Repo banner: archived read-only 21 Mar 2026.

Docs snippets still show :3.3.5 in places. Pull what you mean:

docker pull ghcr.io/huggingface/text-generation-inference:3.3.7

Why 3.3.7 specifically for multi-GPU hosts? Auto device count used to collapse to 1 when NVIDIA_VISIBLE_DEVICES was all or void (and in some CDI toolkit setups). Release notes for 3.3.7 fix that. Single-GPU boxes care less; multi-GPU Docker hosts hit it constantly.

Install Text Generation Inference with Docker (step-by-step)

Cache weights on a host volume. Re-downloading 14 GB every restart is how afternoons disappear.

mkdir -p $PWD/tgi-data
Start with a public instruct model so token issues don’t stack on toolkit issues – e.g. mistralai/Mistral-7B-Instruct-v0.3 (accept the Hub license if the model card requires it).
Run:

model=mistralai/Mistral-7B-Instruct-v0.3
volume=$PWD/tgi-data

docker run --name tgi-337 --gpus all --shm-size 1g 
 -p 8080:80 
 -v $volume:/data 
 ghcr.io/huggingface/text-generation-inference:3.3.7 
 --model-id $model

First boot pulls weights into /data. Watch the logs past shard start. ShardCannotStart here usually means toolkit/GPU visibility – not the model ID.

Gated weights need a read token in the container env. Official gated-access walkthrough is here: gated model access.

docker run --name tgi-337 --gpus all --shm-size 1g 
 -e HF_TOKEN=hf_your_read_token 
 -p 8080:80 -v $PWD/tgi-data:/data 
 ghcr.io/huggingface/text-generation-inference:3.3.7 
 --model-id meta-llama/Meta-Llama-3-8B-Instruct

Four-way tensor parallel example (70B-class; only after nvidia-smi shows the cards you think you have):

docker run --name tgi-337 --gpus all --shm-size 1g 
 -e HF_TOKEN=$HF_TOKEN -p 8080:80 -v $PWD/tgi-data:/data 
 ghcr.io/huggingface/text-generation-inference:3.3.7 
 --model-id meta-llama/Meta-Llama-3-70B-Instruct 
 --sharded true --num-shard 4

Source builds and Nix exist. For prod serving, prefer the GHCR image – local compiles buy you pain, not a cleaner runtime.

First-time configuration that doesn’t fight you

v3 zero-config (Inference Endpoints TGI docs): omit max input/total/batch token flags and the process sizes them from free VRAM at startup. Leave them unset until traffic shape is real.

Knobs after a stable boot:

--max-input-tokens / --max-total-tokens – per-request memory budget
--quantize bitsandbytes-nf4 (or gptq/awq on already-quantized weights) – when the card is tight
--num-shard N – must match the GPUs you intend to use

Pro tip:--shm-size 1g minimum. Stock Docker /dev/shm is ~64 MB. NCCL and multi-process init blow up there even when VRAM looks fine – the failure often surfaces as shared-memory “no space left” or another ShardCannotStart. Some multi-GPU write-ups push much larger SHM; start at 1g and raise if NCCL still complains. On trusted hosts, --ipc=host is the blunt alternative.

First run: foreground only. Detach with -d after /health returns 200.

Is pinning a maintenance-mode server the hill you want to die on for a greenfield cluster? Maybe not. For a box that already speaks TGI, 3.3.7 is still a clean install target – just go in with eyes open on the roadmap.

Verify the install works

curl -s -o /dev/null -w "%{http_code}n" http://127.0.0.1:8080/health

Expect 200 when ready.

curl http://127.0.0.1:8080/generate 
 -X POST -H 'Content-Type: application/json' 
 -d '{"inputs":"Name two uses for continuous batching.","parameters":{"max_new_tokens":24}}'

curl http://127.0.0.1:8080/v1/chat/completions 
 -X POST -H 'Content-Type: application/json' 
 -d '{"model":"tgi","messages":[{"role":"user","content":"Reply with OK and nothing else."}],"max_tokens":8}'

OpenAPI UI: http://127.0.0.1:8080/docs. Generate works but chat 404s? Ancient image – re-pull 3.3.7.

Common install errors and fixes

ShardCannotStart / CUDA not available. Toolkit missing, bad --gpus flag, or a CPU-only attempt. GPU hosts: repair nvidia-container-toolkit, re-run the CUDA nvidia-smi container. Pure CPU is not a supported platform – drop --gpus all and pass --disable-custom-kernels only for experiments; community threads still show flash-attn import failures on CPU even with that flag.

Gated 401.-e HF_TOKEN=... plus accept the model license on the Hub as that same user. Token alone does not skip an unaccepted gate.

CUDA OOM at load. Weights + runtime overhead exceed the card. Try --quantize bitsandbytes-nf4, raise --num-shard, or pick a smaller ID. Rough FP16 map from community resource tables: 7B ~14 GB, 13B ~26 GB, 70B ~140 GB – before KV cache.

One GPU busy, seven idle. Pre-3.3.7 auto-shard bug with NVIDIA_VISIBLE_DEVICES=all. You’re on the wrong tag, or you never set --num-shard.

K8s restarts on a “healthy” busy pod./health can go bad when the request queue is full (see discussions around separate live/ready behavior). Lengthen readiness grace, or split live vs ready probes if the orchestrator allows – don’t use a strict liveness hit on /health under load.

Upgrade and uninstall

docker stop tgi-337 && docker rm tgi-337
docker pull ghcr.io/huggingface/text-generation-inference:3.3.7
# same docker run as before; keep the -v data mount

No special 3.3.x config migration. Maintenance mode means fix tags, not feature bumps.

docker stop tgi-337; docker rm tgi-337
docker rmi ghcr.io/huggingface/text-generation-inference:3.3.7
# optional: rm -rf $PWD/tgi-data # drops cached weights

Greenfield in 2026? Benchmark vLLM or SGLang on your hardware before freezing a new fleet on TGI – HF Inference Endpoints docs already point new engines that way. An existing TGI fleet pinned to 3.3.7 for known architectures remains a reasonable ops call.

Point any OpenAI-compatible client at http://host:8080/v1/ once /health is green.

FAQ

What is the latest TGI version I should run?

v3.3.7 (18 Dec 2025). Pin the tag. Floating :latest after the Mar 2026 archive is a lottery.

Do I need HF_TOKEN for every model?

No. Public non-gated IDs download without it. Gated or private Hub repos need -e HF_TOKEN on the docker run. Create a read token under HF Settings → Access Tokens, accept the model license in the browser first, then start the container – reverse that order and you’ll debug a 401 that isn’t a Docker bug.

Is TGI still worth deploying if it’s in maintenance mode?

If your clients already speak TGI routes and your models are on the supported list, yes – 3.3.7 still serves production traffic. The misconception is “maintenance = dead.” It means no new model families and no big perf roadmap. Brand-new stack? Compare throughput and ops surface with vLLM/SGLang on your GPUs before you commit. On HF Endpoints, migration is a new endpoint + cutover + retire – not an in-place engine flip.