Deploy TinyLlama 1.1B: Open Source Tiny LLM Guide

The 640 MB open-source brain that still fits on a laptop stick

Skip the parameter-count sermon. What got me to ship TinyLlama-1.1B-Chat-v1.0 last week was simpler: Ollama’s tinyllama blob is 638 MB (library listing), Q4_K_M GGUF lands around 0.66 GB, and the stack still speaks plain Llama-2 tokenizer + arch. Every normal Llama tool loads it. No custom loader. No cluster.

I needed a private offline chat endpoint for a small internal tool. Cloud was off-limits. Heavier 7B quants on a 16 GB box left no headroom. TinyLlama was the first checkpoint that loaded, answered, and left RAM for the rest of the stack.

System requirements that actually matter

Practical floors for chat-v1.0 as of late 2025 – community runtime notes plus Ollama’s published size – not slideware.

GPU is optional. Pure CPU still returns short replies fast enough to pair with a local UI. Disk is boring. The real surprise: the quantized path wants less RAM than a busy Chrome session.

Official sources (use these, not random mirrors)

• Model card & safetensors: TinyLlama/TinyLlama-1.1B-Chat-v1.0
• Training code (archived Jul 30 2025): jzhang38/TinyLlama
• Paper: arXiv:2401.02385 (tech report; Llama-2-compatible 1.1B line, long 3T-token training target)
• Ollama library entry: ollama.com/library/tinyllama – 638 MB, 2K context tag
• Common GGUFs: TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF on Hugging Face (~0.66 GB Q4_K_M class)

GitHub is read-only now. Live path = frozen HF weights + Ollama tag + community GGUFs. Architecture snapshot from the card/README/paper: 22 layers, 32 heads, 4 GQA groups, hidden 2048, intermediate 5632, seq 2048, Apache-2.0. Chat-v1.0 is the Zephyr-style SFT+DPO stack (UltraChat + UltraFeedback) on the 3T-token intermediate – not a mystery fine-tune.

Install TinyLlama the fast way (Ollama)

If you want a socket today, not a research notebook, start here.

1. Install Ollama (Linux/macOS):

curl -fsSL https://ollama.com/install.sh | sh

Windows PowerShell: irm https://ollama.com/install.ps1 | iex (or winget install Ollama.Ollama).

2. Pull and run:

ollama pull tinyllama
ollama run tinyllama

First run fetches the 638 MB blob. Land in the chat REPL. /bye exits.

3. API on the default port (http://localhost:11434):

ollama serve
# other terminal
curl http://localhost:11434/api/chat -d '{
 "model": "tinyllama",
 "messages": [{"role": "user", "content": "Summarize the Apache 2.0 license in one sentence."}],
 "stream": false
}'

After the first good reply, run ollama ps. You’ll see GPU vs CPU and how long the weights stay resident.

First-time configuration that isn’t optional

People load the weights, send a plain string, and call the model dumb. Wrong template. Ollama’s default Modelfile already wires chat formatting for tinyllama. Raw transformers or a hand-rolled Modelfile? You must run the tokenizer chat template (Zephyr-style roles). HF’s card is explicit: apply_chat_template before generate.

Minimal Python path (transformers ≥ 4.34, alternative to Ollama):

pip install "transformers>=4.34" accelerate torch
import torch
from transformers import pipeline
pipe = pipeline(
 "text-generation",
 model="TinyLlama/TinyLlama-1.1B-Chat-v1.0",
 torch_dtype=torch.bfloat16,
 device_map="auto",
)
messages = [
 {"role": "system", "content": "You are a concise technical assistant."},
 {"role": "user", "content": "Two bullets: why ship a 1.1B chat model on-device?"},
]
prompt = pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
out = pipe(prompt, max_new_tokens=128, do_sample=True, temperature=0.7, top_p=0.9)
print(out[0]["generated_text"])

llama.cpp / LM Studio: point at tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf from TheBloke’s card. Same 2K ceiling.

Honest pause: a 1.1B chat model will not replace your 70B reasoning stack. It wins when the box is small, the network is gone, or you need a cheap draft pass before a bigger model. If that job description doesn’t match yours, stop here and pick a larger quant.

Verify the install in under a minute

• ollama list – tinyllama near 638 MB
• ollama run tinyllama "Reply with only the string OK-TINY"
• curl -s http://localhost:11434/api/tags | head
• ollama -v

REPL answers + tags JSON = you’re good for basic chat.

Common install errors and the fixes that worked

“tinyllama:latest does not support tools” – Straight failure mode for Home Assistant Assist and anything that posts a tools schema. This chat finetune has no tool/function calling. Point the agent at a tools-capable model, or strip tools from the prompt path.

Transformers OSError / “Could not load model … with any of the following classes” – Almost always transformers<4.34 or a bad HF cache. pip install -U transformers accelerate, delete the local model folder, re-pull. trust_remote_code isn’t required for this plain LlamaForCausalLM checkpoint.

OOM on first load – FP16 on a tight machine. Stay on Ollama’s default quant or import a smaller GGUF via Modelfile. Close the browser; it often steals more RAM than TinyLlama.

Slow first token / stuck on CPU – NVIDIA: drivers + check ollama ps. Apple: Metal is automatic. AMD: Ollama’s ROCm notes. On CPU-only boxes, stop fighting other heavy processes for cores.

Port 11434 busy? Kill the old Ollama process or rebind the host.

Upgrade, migrate, uninstall

ollama pull tinyllama
ollama rm tinyllama # only when you want it gone

Linux binary refresh: re-run curl -fsSL https://ollama.com/install.sh | sh. macOS/Windows apps self-update.

Because jzhang38/TinyLlama archived read-only on Jul 30 2025, don’t wait on an official v1.1 from the original authors. Intermediate code/checkpoints remain, but new chat weights will be community forks – test their templates like they’re new models.

# Linux cleanup example - ~/.ollama wipes EVERY local model
sudo systemctl stop ollama 2>/dev/null
sudo rm -f /usr/local/bin/ollama
rm -rf ~/.ollama

Windows: Apps & features → uninstall Ollama, then delete %USERPROFILE%.ollama if you really want a blank slate.

FAQ

Is TinyLlama still worth deploying in 2025/2026?

Yes – if the machine is small, the demo must be offline, or you need a Llama-compatible draft model under ~700 MB. Newer 1-3B checkpoints may win benchmarks. TinyLlama still ships clean Apache-2.0 chat weights with wide mirror availability.

Ollama vs transformers vs llama.cpp – which should I pick?

Ollama when you want a one-command chat server and simple HTTP. transformers when you’re already in a Python notebook and need HF extras (LoRA hooks, etc.). llama.cpp/GGUF when you care about CPU knobs or embedding the model inside a C++/Rust binary. I started on Ollama, then moved a GGUF only when I needed a single static binary.

Can I fine-tune it on a single consumer GPU?

The 1.1B chat checkpoint is a normal small-Llama LoRA/QLoRA target with PEFT-style stacks; people routinely do adapter work on consumer cards. Full-parameter finetunes cost more VRAM. Original SFT scripts still sit in the archived repo. Build datasets with the 2048 hard context in mind – long samples get cut, and quality tanks quietly if you ignore that ceiling.

Next step: ollama pull tinyllama && ollama run tinyllama. Ask something from your actual domain. Note latency. If it clears that bar, wire /api/chat before you chase a fatter model.