Skip to content
AI Tools

AI Server Setup Guide: Private OpenAI Endpoint Fast

Spin up an AI Server for a private OpenAI-compatible endpoint with local models. Docker steps, gotchas, and how it beats bare Ollama or pure cloud.

5 min readBeginner

By the end of this you’ll have a running AI Server: one private OpenAI-compatible endpoint on your machine that routes chat, embeddings and more to local models (via Ollama) or optional cloud providers, complete with API keys and a simple admin UI. Point Cursor, Open WebUI, your own scripts or any OpenAI SDK at it and the data stays yours.

No H100 rack. Docker, a few commands, a model that fits your RAM/VRAM. Straight path that works as of 2026.

What an AI Server Actually Does for You

Here “AI Server” means a self-hosted gateway – not a bare GPU chassis and not another “what is VRAM” lecture. ServiceStack’s free OSS AI Server sits in front of the providers you pick. Apps speak one base URL and one key set. Behind that it can fan out to Ollama on localhost, OpenAI, Anthropic, Groq, OpenRouter, Mistral, Google, or a ComfyUI/FFmpeg-style agent when you need media jobs.

Central keys. Request history. Analytics. Swap a backend without touching client code. Bare Ollama on 11434 forces every app to hard-code the port and leaves you without a shared key story.

Think of it like a reverse proxy that also speaks model names: clients stay dumb and stable while you change what actually runs the weights.

Step-by-Step: Get the AI Server Running

Prerequisites (as of ServiceStack’s current quickstart): Docker Engine + Compose, Git. Linux or macOS natively; Windows via WSL2 + Docker Desktop. GPU optional.

  1. Clone and install: 
    git clone https://github.com/ServiceStack/ai-server
cd ai-server
cat install.sh | bash
  2. The script sniffs common API keys in your environment, asks which providers to turn on, writes a .env, and brings the stack up with Compose.
  3. Admin Portal: http://localhost:5006/admin. Per the quickstart, default portal password / AUTH_SECRET is p@55wOrd – change it in .env and restart before you do anything else.
  4. Add an Ollama provider (often http://host.docker.internal:11434 or whatever address the containers can reach). Mint an API key for clients.
  5. Hit the gateway with curl or any OpenAI SDK using your AI Server base URL + that key. Built-in chat lives under the admin area.

No Ollama yet? curl -fsSL https://ollama.com/install.sh | sh on Linux/macOS, then ollama pull llama3.2 (or another small tag). Ollama’s docs put an OpenAI-compatible API on 11434 by default.

Keep Ollama on 127.0.0.1. Let only the AI Server talk to it. Community hosting write-ups keep repeating the same failure mode: raw Ollama has no request auth, so OLLAMA_HOST=0.0.0.0 plus an open firewall is free inference for strangers.

Common Pitfalls That Waste an Afternoon

p@55wOrd is public. If :5006 is reachable, that’s admin until AUTH_SECRET rotates. Do it in the same sitting as install.

Model larger than free RAM/VRAM? Process vanishes – “Killed”, or a llama-server termination line. dmesg shows the OOM killer. Rough weight math people actually use: about 0.5 GB per billion params at Q4, ~1 GB at 8-bit, ~2 GB at FP16, then add KV cache for context and concurrency. A 7-8B Q4 build often lands in the 8-12 GB card class with headroom; 70B Q4 wants ~40 GB-class iron. Size first. Pull second.

CPU-only boxes often auto-pick a low context (commonly 4K from VRAM detection). Long prompts truncate quietly unless you set num_ctx or OLLAMA_CONTEXT_LENGTH (or override through the gateway/UI). You only notice when the model “forgets” the bottom of the paste.

Port clash or junk volumes from a half-failed run: wipe App_Data and re-run install, or docker compose down -v when you mean it.

Is one extra Docker service “worth it” when Ollama alone already answers tokens? Only if you care about keys, mixed providers, and not re-teaching every client the same base URL later.

How This AI Server Stacks Against Alternatives

Approach Best for Trade-off
ServiceStack AI Server + Ollama Single endpoint, keys, multi-provider, analytics Docker overhead; still must size models
Bare Ollama Fastest solo setup, zero extra services No auth, no central keys, every client hard-codes 11434
Cloud only (OpenAI etc.) Zero local hardware, huge models Per-token cost; data leaves your network
Software Tailor AI Server Windows-first app + full local engines Free tier is localhost/loopback only; network use needs Pro Personal or Pro Commercial (as of their licensing docs)
Full hardware AI server (multi-GPU rack) Training or high-concurrency inference CapEx, power, cooling – overkill for personal/team chat

Gateway + local runner is the better default for private chat and tool wiring. Cloud when you need a giant reasoner once. Racks when you’re training or serving a crowd – not when you’re wiring Cursor to a 8B Q4.

FAQ

Do I need a GPU for an AI Server?

No. Small quantized models (roughly 3-8B) run on CPU. Slow. Fine for smoke tests.

Can I mix local and cloud models?

Yes. Register Ollama plus an OpenAI, Anthropic, or Groq key in the Admin Portal. Clients ask by model name; the gateway routes. Example: local 8B for bulk edits, cloud model only when the local card is saturated or you want a stronger single-shot plan.

Is the free ServiceStack version production-ready for a small team?

For internal gateway use – keys, logging, multi-provider routing – yes, with boring ops discipline. Cap host resources. Rotate keys. Terminate TLS on a reverse proxy. Do not publish the admin port. Heavy ComfyUI-style media work belongs on a GPU box running the optional agent. Paid ServiceStack support licenses exist if you want a vendor on call; the AI Server project itself starts free and open.

Next: run the clone + cat install.sh | bash pair, change AUTH_SECRET, pull one small model, send one chat completion with curl at the new endpoint. When that returns JSON, point your real client at it.

Related Tutorials