Deploy TGI 3.3.7: Hugging Face Inference Guide

What you’ll have running

By the end of this page you will have Text Generation Inference (TGI) v3.3.7 on port 8080 with an OpenAI-compatible /v1/chat/completions endpoint. One Docker command, a weights volume, a health check, a test generate call.

TGI is Hugging Face’s production LLM server (Rust router + Python workers). As of late 2025 it sits in maintenance mode; the GitHub repo went read-only archived on Mar 21 2026. v3.3.7 still runs and took minor fixes – including device-count behavior – but HF steers new work toward vLLM, SGLang, llama.cpp, or MLX. Shipping on TGI in 2026 is a conscious short-horizon choice, not a default platform bet.

Odd feeling, pinning a stack the vendor already soft-retired. Teams do it anyway when configs, metrics dashboards, or internal clients already speak TGI. If you are greenfield, that tension is the decision – not another feature matrix.

System requirements for TGI 3.3.7

Linux + NVIDIA is the path that matches the docs. H100, A100, A10G, T4 and similar need CUDA 12.2+ host drivers and the NVIDIA Container Toolkit. Ampere and newer get full Flash Attention / Paged Attention; older cards fall back.

Figures above track community sizing notes (as of 2025 deploy guides) plus HF’s NVIDIA install doc – treat them as starting budgets, not guarantees. CPU-only with --disable-custom-kernels boots; it crawls. Multi-GPU is tensor parallelism through --num-shard.

Official download source

Pull only the official GHCR image. Skip random mirrors for anything you care about.

docker pull ghcr.io/huggingface/text-generation-inference:3.3.7

Primary references: TGI quickstart, index (maintenance banner), and GitHub releases. Docs still mention 3.3.5 in places – pin 3.3.7 (or a hash) in scripts so floating latest cannot move under you. Hardware variant tags exist (Gaudi, Intel, TRT-LLM); stay on the NVIDIA image unless you already run that stack.

Install TGI – Docker step by step

Confirm the toolkit actually sees GPUs:

docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

Fail here → fix drivers/toolkit before any TGI pull.

Cache directory so weights survive restarts:

mkdir -p $PWD/tgi-data

Launch with the exact 3.3.7 tag (non-gated example):

model=teknium/OpenHermes-2.5-Mistral-7B
volume=$PWD/tgi-data

docker run --gpus all --shm-size 1g -p 8080:80 
 -v $volume:/data 
 --name tgi 
 ghcr.io/huggingface/text-generation-inference:3.3.7 
 --model-id $model

First boot downloads into /data. Wait until the router logs ready. Gated models need a Hub token and a prior license click:

-e HF_TOKEN=hf_your_token_here

HUGGING_FACE_HUB_TOKEN works too. Prefer a small open instruct model for the smoke test; swap the ID only after /health is clean.

Pro tip: official quickstart shows --shm-size 1g. Multi-GPU and fat context runs often want 8g-64g (NVIDIA-style examples use the high end). Undersized shared memory shows up as CUDA IPC deaths under load – not at idle.

Why do defaults land at 1g if production guides jump to 64g? Different authors optimizing for laptop demos vs eight-GPU nodes. Match shm to how many workers will open IPC channels, then stop guessing.

Source builds need Rust, protoc, Python 3.9+, and BUILD_EXTENSIONS=True make install. Fine for kernel contributors. For serving, extra failure surface. AMD/Gaudi/Inferentia/TPU images exist – use them only if that is already your home hardware.

First-time configuration that actually matters

Omit the token caps and v3+ zero-config will size max input / total / batch from the hardware (HF Inference Endpoints engine notes describe this). On a 24 GB card with a 7B instruct model, explicit caps still save you from surprise OOM on the first long prompt:

• --max-input-length 4096
• --max-total-tokens 8192
• --max-batch-prefill-tokens 4096 when you want prefill tighter
• --num-shard N for N identical GPUs
• --quantize bitsandbytes-nf4 or gptq when VRAM is tight

docker run --gpus all --shm-size 1g -p 8080:80 
 -v $PWD/tgi-data:/data 
 -e HF_TOKEN=$HF_TOKEN 
 ghcr.io/huggingface/text-generation-inference:3.3.7 
 --model-id meta-llama/Llama-3.1-8B-Instruct 
 --max-input-length 4096 
 --max-total-tokens 8192

Release 3.3.7 fixed num_devices / num_shard auto-detection when NVIDIA_VISIBLE_DEVICES is all or void (PR #3346 in the v3.3.7 notes). Orchestrators inject those strings constantly – older tags mis-counted GPUs.

Verify the install works

Health first:

curl -s http://localhost:8080/health

Expect a plain OK/healthy style body once weights are loaded.

Native generate (avoid the stock demo prompt every blog recycles):

curl http://127.0.0.1:8080/generate 
 -X POST 
 -H 'Content-Type: application/json' 
 -d '{"inputs":"Reply with the word pong and nothing else.","parameters":{"max_new_tokens":8}}'

OpenAI-style path – what most apps actually call:

curl localhost:8080/v1/chat/completions 
 -X POST 
 -H 'Content-Type: application/json' 
 -d '{
 "model": "tgi",
 "messages": [{"role":"user","content":"Say hello in one sentence."}],
 "max_tokens": 32
}'

Python via huggingface_hub:

from huggingface_hub import InferenceClient
client = InferenceClient(base_url="http://localhost:8080/v1/")
print(client.chat.completions.create(
 model="tgi",
 messages=[{"role":"user","content":"Count to 5"}],
 max_tokens=64
))

Common install errors and fixes

nvidia-container-cli: requirement error: unsatisfied condition: cuda>=12.1 – host driver older than the image expects. Bump drivers until nvidia-smi reports CUDA 12.2+, refresh the toolkit, re-run the cuda base smoke test. Tracked in community threads such as issue #1391.

Container starts, then dies on multi-GPU: NCCL noise, “Cannot assign requested address”, shard signal 7. Raise --shm-size, keep GPU types identical, set --num-shard to the visible device count. Some cloud GPU exposures need ldconfig/host lib care – pattern shows up on GKE-style setups more than bare metal.

Gated download failures → read-scoped token in -e HF_TOKEN=..., license accepted on the Hub UI, token actually entitled to that repo.

OOM on warmup or first long prompt → cut --max-input-length / --max-total-tokens, turn on quantization, or shrink batch/prefill. Zero-config helps; it will not invent VRAM on a 12 GB card running 13B FP16.

The catch is /health under backpressure. A full request queue can mark the process unhealthy even though it is doing useful work. Kubernetes-style probes that restart on failed health checks thrash a busy pod. Separate livez/readyz is a recurring ask in GitHub discussions and is not split the way many operators want – design probes knowing busy ≠ dead.

Upgrade and uninstall

Stop, remove, pull, reuse the volume:

docker stop tgi && docker rm tgi
docker pull ghcr.io/huggingface/text-generation-inference:3.3.7
# re-run your docker run with the new tag

Maintenance-mode jumps are mostly patches (device-count fix in 3.3.7 is the poster child). Retest /health and one short generate after every bump.

Cleanup:

docker stop tgi; docker rm tgi
docker rmi ghcr.io/huggingface/text-generation-inference:3.3.7
# optional: rm -rf $PWD/tgi-data # drops cached weights

Docker-only install leaves no system packages behind.

FAQ

Is TGI still worth deploying in 2026?

Yes when you already depend on it. Greenfield? Start with vLLM or SGLang – HF’s own recommendation after the maintenance banner.

How do I serve a private or gated model?

Hub token with read scope. Accept the model license page first. Pass -e HF_TOKEN=hf_... (or HUGGING_FACE_HUB_TOKEN) on docker run and keep the same /data volume so restarts skip the download. Llama-3.1-8B-Instruct is the usual footgun: missing either the token or the license click fails at pull, not at generate.

Docker vs install from source – which wins?

Docker. Source means Rust toolchain, exact protoc, CUDA dev headers, and a clean make install with extensions. Contributors hacking kernels need that. Deployments collect fragile host state for no gain if the GHCR image already covers the hardware. Pick source only when the container literally cannot express a required build flag.

Next: nvidia-smi Docker smoke test → 3.3.7 run with a small open model → /health. Clean response? Then swap the real model ID and token.