Install Yi Open Source Locally: Yi-1.5 Deploy Guide

By the end of this guide you’ll have 01.AI Yi open source running locally – specifically Yi-1.5-9B-Chat answering over Transformers or an OpenAI-compatible endpoint. No A800 required for the path below.

Yi is 01.AI’s bilingual LLM family trained from scratch. Practical deploy targets right now: the Yi-1.5 series (open-sourced 13 May 2024, Apache 2.0; continued pretrain on 500B tokens + fine-tune on 3M samples; 34B/9B/6B) and, for code, Yi-Coder (5 Sep 2024, 1.5B/9B, up to 128K context, Apache 2.0). The original 01-ai/Yi repo still holds the Docker image, hardware tables, and web demo tooling.

System requirements before you touch the weights

Pick the model from VRAM first. Figures below match the official Yi deployment tables on the main README (as of that doc – batch rows beat the headline “minimums” when you’re sizing a card).

Software: Linux or WSL2, NVIDIA drivers, Python 3.10+ for the Yi-1.5 pip path, CUDA matched to your torch wheel. Docker needs Engine plus nvidia-container-toolkit.

Disk is the silent killer. Official READMEs list hubs and VRAM – not a single on-disk size matrix for every Yi-1.5 precision. Check the Hugging Face file list for your shard set and leave a wide free margin; multi-shard 9B/34B pulls fail messily when the volume fills mid-download.

Think of model size like luggage for a weekend vs a month abroad. 9B fits most serious single-GPU boxes; 34B full precision is a different trip.

Official download sources (use these only)

Skip random mirrors. Weights and tokenizers ship from Hugging Face 01-ai (this guide uses 01-ai/Yi-1.5-9B-Chat), ModelScope 01ai, and wisemodel 01.AI. Code lives under github.com/01-ai/Yi and github.com/01-ai/Yi-1.5. Runtime image: ghcr.io/01-ai/yi:latest (alternate registry: registry.lingyiwanwu.com/ci/01-ai/yi:latest).

Architecture and training background sit in the Yi tech report (arXiv:2403.04652, 8 Mar 2024). You do not need the paper to install.

Install path A – pip + Transformers (Yi-1.5-9B-Chat)

Fresh venv. Torch pins from one repo will wreck another project’s stack if you share environments.

# 1) Environment
python3.10 -m venv ~/venvs/yi15
source ~/venvs/yi15/bin/activate
pip install -U pip

# 2) Tooling repo (inference helpers + requirements)
git clone https://github.com/01-ai/Yi-1.5.git
cd Yi-1.5
pip install -r requirements.txt
# Yi-1.5: transformers>=4.36.2, torch>=2.0.1,<=2.3.0, sentencepiece, accelerate, gradio>=4.13.0

# 3) Full weights - not a shallow git clone of LFS pointer stubs
pip install -U "huggingface_hub[cli]"
huggingface-cli download 01-ai/Yi-1.5-9B-Chat --local-dir ~/models/Yi-1.5-9B-Chat

Original Yi’s requirements.txt still pins torch==2.0.1 and pulls deepspeed. Yi-1.5 loosens that to torch>=2.0.1,<=2.3.0. Mixing the two files on a newer driver stack is a classic CUDA breakage – stick to Yi-1.5’s range for new deploys.

1. nvidia-smi sees the GPU
2. ~/models/Yi-1.5-9B-Chat holds full *.safetensors / bin shards (kilobyte “pointer” files mean the pull lied)
3. Tokenizer load will use use_fast=False in the verify script below

First-time config and verify it works

One chat turn. No UI. No server.

# save as verify_yi15.py
from transformers import AutoModelForCausalLM, AutoTokenizer

model_path = "/home/YOU/models/Yi-1.5-9B-Chat" # change me

tokenizer = AutoTokenizer.from_pretrained(model_path, use_fast=False)
model = AutoModelForCausalLM.from_pretrained(
 model_path,
 device_map="auto",
 torch_dtype="auto",
).eval()

messages = [{"role": "user", "content": "Reply with exactly: yi-ok"}]
input_ids = tokenizer.apply_chat_template(
 conversation=messages,
 tokenize=True,
 add_generation_prompt=True,
 return_tensors="pt",
)
output_ids = model.generate(input_ids.to(model.device), max_new_tokens=32)
print(tokenizer.decode(output_ids[0][input_ids.shape[1]:], skip_special_tokens=True))

Run python verify_yi15.py. You want a short coherent reply without CUDA OOM. Optional UI follows the main Yi tree pattern: python demo/web_demo.py -c ~/models/Yi-1.5-9B-Chat (clone 01-ai/Yi if you need that demo/ folder).

Pro tip: Load “succeeds,” first generate dies? Another process is sitting on the card. Headline VRAM assumes a quiet GPU and short context – long prompts chew the leftover budget fast.

Alternative installs (when pip isn’t the right hammer)

Docker on a host that already passes GPU into containers:

docker run -it --gpus all 
 -v ~/models/Yi-1.5-9B-Chat:/models 
 ghcr.io/01-ai/yi:latest
# inside: point model_path at /models, reuse the same Python snippet

Ollama if you just want a chat REPL: install Ollama, leave ollama serve up, then ollama run yi:1.5 (Yi-1.5 README path; tags listed at ollama.com/library/yi/tags).

vLLM when apps expect an OpenAI-shaped API:

python -m vllm.entrypoints.openai.api_server 
 --model 01-ai/Yi-1.5-9B-Chat 
 --served-model-name Yi-1.5-9B-Chat
# curl http://localhost:8000/v1/chat/completions ...

Yi-Coder only:git clone https://github.com/01-ai/Yi-Coder.git && cd Yi-Coder && pip install -r requirements.txt needs Python ≥3.9. Load 01-ai/Yi-Coder-9B-Chat the same Transformers way – 128K-class context, Apache 2.0.

Common install errors and fixes

These are the failures that burn evenings. Each one maps to a real report or an official snippet quirk.

• Load dies after a clean-looking download – Weights are still Git LFS pointer stubs. 01-ai/Yi#473: install git-lfs or re-run huggingface-cli download until every shard is full size.
• ValueError on tokenizer init – Official Yi / Yi-1.5 chat snippets set AutoTokenizer(..., use_fast=False). Fast paths blow up on Yi tokenizers (community MLX reports hit the same conversion error). Keep the flag off.
• OOM on a “12GB-capable” card – Tables list Yi-6B-Chat batch=1 at 12GB while the overall minimum is 15GB. Discussion #250 is full of real OOM on 12GB hardware once activations and context grow. Cut max_new_tokens, switch to 4-bit AWQ, or drop to Yi-Coder-1.5B / 6B-4bit.
• Torch / CUDA mismatch after “following the docs” – You mixed original Yi’s torch==2.0.1 pin with a machine aimed at newer torch. Stay inside Yi-1.5’s >=2.0.1,<=2.3.0 window or burn the venv and rebuild.
• Docker sees no GPU – Host missing nvidia-container-toolkit. Prove the host first: docker run --gpus all nvidia/cuda:12.1.0-base-ubuntu22.04 nvidia-smi.

Upgrade, migrate, uninstall

No one-click Yi version bump. You swap weight directories and keep the matching code repo.

# Move to Yi-1.5 chat weights
huggingface-cli download 01-ai/Yi-1.5-9B-Chat --local-dir ~/models/Yi-1.5-9B-Chat
# point verify_yi15.py / vLLM --model at the new path
# keep Apache attribution if you ship derivatives (Yi-1.5 LICENSE)

# Cleanup
deactivate
rm -rf ~/venvs/yi15 ~/models/Yi-1.5-9B-Chat
docker rm -f $(docker ps -aq --filter ancestor=ghcr.io/01-ai/yi:latest) 2>/dev/null
docker rmi ghcr.io/01-ai/yi:latest 2>/dev/null
ollama rm yi:1.5

Apache 2.0 on Yi-1.5 is plain in the project LICENSE. Whether that changes the commercial calculus versus earlier community-license weights is still a counsel call – engineering can only hand them the README text.

FAQ

Which Yi open-source model should I deploy first?

Yi-1.5-9B-Chat on a 24GB GPU. Yi-Coder-9B-Chat if the workload is mostly code. 34B only when you already have multi-GPU or heavy quantization lined up.

Is the Docker image required?

No. Pip plus a local weight directory covers single-node inference. Docker helps on shared GPU boxes where you want a locked runtime without root-level Python fights. Ollama wins when you only need a chat REPL and do not care about raw Transformers knobs – a teammate on an RTX 4090 can finish the pip path in one sitting; the same person on a locked-down company host often bind-mounts weights into ghcr.io/01-ai/yi:latest instead.

Can I use older Yi-34B-Chat weights with Yi-1.5 code?

Basic from_pretrained chat generation often works – same Transformers patterns – but do not assume identical chat templates, eval numbers, or license terms. Yi-1.5 is its own continued-pretrain + SFT line (500B tokens / 3M fine-tune samples per the project README). If generation looks “almost right” then falls over on formatting, pin code and weights from the same series instead of loading whichever folder is largest on disk.

Next: create the venv, run the three install commands, execute verify_yi15.py once. Clean reply? Wire vLLM or the web demo second – not first.