Deploy AI Crawler Crawl4AI v0.9.4 Install Guide

The #1 mistake with an AI crawler install isn’t picking the wrong image. It’s running pip install crawl4ai (or a bare Docker pull) and assuming you’re done. Crawl4AI drives real Chromium through Playwright. Skip browser setup on pip, or skip the API token and shared-memory flag on Docker, and the first crawl dies with “Executable doesn’t exist,” a dead browser, or a “healthy” container that resets every published connection.

Crawl4AI ships v0.9.4 as the current stable tag on the project’s GitHub releases (dated 23 Sep 2026 in that tag – verify on the releases page before you pin). Self-hosted library, Docker API server, or hosted cloud. Goal here: crawl today, not after three GitHub issues.

Pick a deploy path before you touch the keyboard

Three runtimes. Three cost/control tradeoffs. Match the path to who owns the browser process.

Cloud starter credit still shows as a first $10 pack free until 31 Dec 2026, then a smaller paid start (as of late 2026 project docs – confirm on crawl4ai.com before you budget). Self-host stays free; you pay CPU/RAM only.

System requirements for Crawl4AI v0.9.4

Python path: 3.10+ (pyproject classifiers list 3.10-3.13). Docker path: Docker 20.10+; Compose if you use the repo file. Under-provision and the image may look “installed” then exit at boot.

• Library floor: ~2 GB RAM, ~500 MB-2 GB disk after Chromium lands; 8 GB+ RAM for heavy JS or extras.
• Docker floor: plan ~4 GB RAM for the container on steady crawls. Startup logic checks free memory and exits when free RAM inside the container is under 2 GB (free -m / MEM < 2048 pattern in the image).
• Shared memory: Chromium wants ~1 GB /dev/shm. Docker’s default 64 MB is not enough.
• Network: outbound HTTPS for targets and the first browser download.
• Optional extras ([torch], [transformer], [all]): multi-GB models – skip unless you need them.

OS: Linux, macOS, Windows. Minimal Linux servers often miss Playwright system libs; crawl4ai-setup or playwright install --with-deps covers most of that.

Official download sources

Stick to these channels only:

• PyPI:pip install crawl4ai==0.9.4 (or -U crawl4ai for latest stable)
• Docker Hub:unclecode/crawl4ai:0.9.4 and unclecode/crawl4ai:latest
• Source:github.com/unclecode/crawl4ai
• Docs:docs.crawl4ai.com installation

Security work landed in this tag for self-hosted Docker: SSRF paths on robots.txt and link-preview, plus an untrusted-config gate that could leak env vars. Library callers pick up faster default pruning through PruningContentFilterLXML (~10x on that step) with no breaking API change – see the changelog.

Install the AI crawler (Docker recommended for deploy)

API, dashboard, non-Python agents: Docker is the cleaner boundary. Generate a token first. Without a real value, current images keep the server on the container loopback; published ports look open, then reset.

# 1) Token (keep this secret)
export CRAWL4AI_API_TOKEN="$(openssl rand -hex 32)"
echo "$CRAWL4AI_API_TOKEN" # save it

# 2) Pull exact version
docker pull unclecode/crawl4ai:0.9.4

# 3) Run - pass the VALUE, not a bare -e name
docker run -d 
 --name crawl4ai 
 -p 11235:11235 
 --shm-size=1g 
 -e CRAWL4AI_API_TOKEN="$CRAWL4AI_API_TOKEN" 
 unclecode/crawl4ai:0.9.4

Wait ~10-40 seconds for Playwright/Redis/API. For longer-lived hosts, prefer the repo docker-compose.yml: it sets shm_size: "1gb" and a /health check.

Pro tip: Never use bare -e CRAWL4AI_API_TOKEN without =$CRAWL4AI_API_TOKEN. If the shell variable is unset, Docker silently passes an empty value. MIGRATION.md flags this – empty env means “no credential,” so the process stays loopback-only while docker ps still says healthy.

Pip alternative (library in your venv)

python3 -m venv .venv && source .venv/bin/activate # Windows: .venvScriptsactivate
pip install -U pip
pip install crawl4ai==0.9.4
crawl4ai-setup # downloads Chromium - do not skip
crawl4ai-doctor # should report a clean environment

Setup failed on a skinny Linux image? Manual browser fix:

python -m playwright install --with-deps chromium

Binaries land under ~/.cache/ms-playwright/.

First-time configuration (minimum viable)

Docker minimum: token + default image. Optional LLM keys belong in .llm.env (never commit it) with --env-file .llm.env. Reverse-proxy: terminate TLS up front; send Authorization: Bearer <token> on every route except GET /health.

Library side needs no config file. Context manager is enough – and yes, we crawl a stdlib docs page on purpose, not the usual demo homepage:

import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
 async with AsyncWebCrawler() as crawler:
 result = await crawler.arun(url="https://docs.python.org/3/library/json.html")
 print(result.markdown[:500] if result.markdown else result.error_message)

asyncio.run(main())

Verify the install works

Docker health (no auth):

curl -sf http://localhost:11235/health

Markdown scrape (auth required):

curl -s http://localhost:11235/md 
 -H "Authorization: Bearer $CRAWL4AI_API_TOKEN" 
 -H "Content-Type: application/json" 
 -d '{"url": "https://docs.python.org/3/library/json.html"}'

Expect Markdown body fields, not connection reset. UI: http://localhost:11235/dashboard and /playground (auth-gated on hardened builds).

Pip path: crawl4ai-doctor green, then the script above. Some installs also ship a crwl CLI – use it only if the entry point exists after setup.

Shared memory isn’t extra headroom from the host. That 1 GB shm is carved from the same cgroup memory limit as Python and Chrome. Cap the container at 1.2 GB, fill shm, and you’ve left roughly a couple hundred megabytes for everything else – the exact “random Chrome death” pattern people mis-blame on Crawl4AI.

Common install errors and fixes

• BrowserType.launch: Executable doesn't exist – Wheel without browser. Run crawl4ai-setup or python -m playwright install chromium.
• Container healthy, curl connection reset on :11235 – Token missing or empty. Export a real value; pass -e CRAWL4AI_API_TOKEN="$CRAWL4AI_API_TOKEN". Re-read MIGRATION.md if you upgraded from pre-0.9 server defaults.
• Chrome crash / target closed under Docker – Add --shm-size=1g (or compose shm_size). If shm is already set, raise container memory so Chromium still has working set after the 1 GB carve-out.
• Startup exit citing low RAM – Give the VM/container ≥2 GB free at boot; heavy crawls want more than the floor.
• Ubuntu missing libs for Playwright – install --with-deps; docs list extra packages when even that fails.
• 401 / missing Bearer on /md – Expected after secure-by-default server changes. /health stays open; data routes need the token.

Upgrade and uninstall

Library:pip install -U crawl4ai==0.9.4 then crawl4ai-doctor. Library-only callers usually need no code edits for 0.9.4. Docker from older API majors (especially pre-0.9): set a fresh token and re-issue clients – old tokens die after the auth rewrite.

docker pull unclecode/crawl4ai:0.9.4
docker stop crawl4ai && docker rm crawl4ai
# re-run the docker run block with the same token

Uninstall library:pip uninstall crawl4ai. Optional full cleanup: ~/.cache/ms-playwright/ and ~/.crawl4ai/. Docker:docker stop crawl4ai && docker rm crawl4ai && docker rmi unclecode/crawl4ai:0.9.4.

FAQ

Do I need Docker to use Crawl4AI as an AI crawler?

No. Pip + crawl4ai-setup covers Python scripts and local RAG. Docker when you want a shared HTTP API, dashboard, or isolation from host Python.

Why does my Docker AI crawler ignore –shm-size and still crash?

Usually the flag never reached the container – Compose key misspelled, orchestrator override – or Memory is lower than shm + browser working set. Check:

docker inspect crawl4ai --format '{{.HostConfig.ShmSize}} {{.HostConfig.Memory}}'

Shm should read ~1073741824. Memory must sit higher than that with room for Chromium. Remember the carve-out: shm is not “on top of” the limit.

Is v0.9.4 safe to expose on a public port?

Only behind TLS with a strong CRAWL4AI_API_TOKEN. Auth is on by default; request bodies count as untrusted. Put a reverse proxy in front, don’t publish Redis, and stay on 0.9.4+ if you accept untrusted URLs – the 0.9.x line closed robots.txt and link-preview SSRF paths plus the untrusted-config env leak. The in-process library never opens a network listener, so its threat model is a different animal.

Next step: run the Docker block (or the pip + setup + doctor path), hit /health, then scrape one URL you actually need for your RAG or agent pipeline. Wire that Markdown into your embedder while the failure modes above are still fresh.