Deploy Garak v0.15.0: NVIDIA LLM Security Scanner

End state you’re building toward: a terminal that runs python3 -m garak --target_type openai --target_name gpt-4o-mini --probes dan.Dan_11_0, spits out a progress bar for each probe, and drops a report.jsonl plus an HTML dashboard onto disk. That’s the deploy. Everything below gets you there in about ten minutes on a clean machine.

NVIDIA’s own AI Red Team runs Garak to check whether a model can be jailbroken, tricked into leaking data, or coerced into toxic output. The tool itself has an interesting origin: created in Spring 2023 by Prof. Leon Derczynski of ITU Copenhagen during a sabbatical at the University of Washington, first dropped under GPL on 13 June 2023, later relicensed to Apache 2.0, and moved to NVIDIA’s public GitHub in November 2024 (Wikipedia). The current release is v0.15.0 (May 2026) – it shipped a multi-turn GOAT probe, an Agent-breaker probe for testing tool-use attack surfaces, a system-prompt-extraction probe, a ModernBERT refusal detector, and NeMo Guardrails server support. The Garak paper on arXiv (Derczynski et al., 2024) covers the four-component architecture – Generators, Probes, Detectors, Buffs – if you want the full technical picture.

System requirements before you install

Garak is a Python CLI. Nothing exotic – but the Python version window is narrow.

The Python constraint comes straight from the official install docs. 3.13 isn’t blocked intentionally – deps just haven’t caught up yet.

Install (recommended path: isolated conda env)

Skip the global pip install. Garak has a wide dependency tree and it will fight with whatever else lives in your system Python.

conda create --name garak "python>=3.10,<=3.12" -y
conda activate garak
python -m pip install -U garak

That pulls the latest PyPI release. If you want the bleeding-edge main branch – where bug fixes land weeks before PyPI – swap the last line for:

python -m pip install -U git+https://github.com/NVIDIA/garak.git@main

For a source clone with editable install (per the NVIDIA README):

gh repo clone NVIDIA/garak
cd garak
python -m pip install -e .

Docker isn’t officially published by NVIDIA. Community images exist but lag main by weeks – pip is the sanctioned path.

First-time configuration

No config file to fill out on install. You configure per-run: environment variables for API keys, CLI flags for model and probes. For an OpenAI target:

export OPENAI_API_KEY="sk-..."
python3 -m garak --list_probes | head -20

That second command is your smoke test – it should print probe families like dan, encoding, promptinject, latentinjection, xss. If you see them, the install works.

About the flags: the README now uses --target_type and --target_name – for example, python3 -m garak --target_type openai --target_name gpt-5-nano --probes encoding. Turns out almost every tutorial older than a few months still shows --model_type and --model_name. The old flags resolve via aliases, so copy-pasting from those guides technically works – but you’ll hit deprecation warnings, and certain probes break under the old names. New scripts: use --target_*.

Verify it works: your first scan

Point Garak at a cheap model and run one probe. Not the full battery – you’ll regret that on billing day.

python3 -m garak 
 --target_type openai 
 --target_name gpt-4o-mini 
 --probes dan.Dan_11_0 
 --generations 3

What you get back: a progress bar per probe, then a PASS/FAIL row per detector. Anything that triggered undesirable behavior shows a failure rate. Artifacts land in Garak’s platform data directory – a .report.jsonl, a hit log, and (since v0.14.0) an auto-generated HTML report with scores grouped by probe category.

Cost lever: The --generations flag controls how many attempts Garak sends per prompt. Default is 10 – and a full-battery scan sends thousands of prompts. That multiplies fast across 50+ probe modules. Set --generations 1 for exploratory runs against paid APIs; commit to higher counts only when you’re doing a real assessment.

Here’s a question worth sitting with before you scale up: what are you actually measuring? Garak reports a failure rate – not a binary pass/fail for the model. A 3% failure rate on a dan probe might mean the model resists jailbreaks reliably, or it might mean your 3-generation sample was just lucky. The HTML reports got a full redesign in v0.14.0 (February 2026) and now include bootstrap confidence intervals for attack-success rates (added in v0.14.1, April 2026), which makes that distinction much clearer. Worth understanding before you bring results to a stakeholder.

Common errors from real GitHub issues

All three of these are live tracker items. If you see the exact message below, don’t debug your setup – it’s a known tool issue.

• NameError in TAP/PAIR probes – GitHub issue #1451: garak/resources/tap/generator_utils.py line 40 references an undefined variable target_name when the function parameter is actually model_name. Both tap.TAP and tap.PAIR crash before the probe even starts. Fix: pull main (patch merged), or edit that line locally to model_name.
• Replicate + parallelism → pickle crash – the Replicate generator throws TypeError: cannot pickle module object when you add --parallel_requests or --parallel_attempts > 1 (issue #1160). Other generators are unaffected. Workaround: drop the parallel flags for Replicate targets, or route through --target_type litellm which uses a different code path.
• -G flag returns ‘Expecting value: line 1 column 1 (char 0)’ – issue #1303 shows this JSON-parse error on valid YAML files. The bug is isolated to the -G flag; env-var config and CLI flags still work fine. Workaround: skip -G until the fix ships.

Errors outside these three almost always trace back to Python 3.13 (downgrade to 3.12) or a stale pip cache (pip install --no-cache-dir -U garak).

Upgrade and uninstall

Upgrading: pip install -U garak. The one thing to grep your scan scripts for before upgrading is the --model_* → --target_* flag rename – everything else in the 0.14.x → 0.15.x window has been stable.

Uninstall:

pip uninstall garak
rm -rf ~/.local/share/garak # reports, cached models, logs (Linux/macOS)
conda env remove -n garak # if you used the recommended install

Garak’s data directory grows with every run – every JSONL report you’ve produced lives there. Clean it periodically or archive it if you need the audit trail. After a few full-battery scans, it’s worth checking the size.

Where to actually point it

Garak works against a wide range of targets. OpenAI, Hugging Face Hub, AWS Bedrock, Cohere, Replicate, Groq, Ollama, NVIDIA NIM (via build.nvidia.com), llama.cpp/gguf, LiteLLM, generic REST – all supported, per the PyPI page. For NIM endpoints, check the official NIM documentation for the correct API key environment variable and auth setup.

The real question after you install: what do you scan first? Running the default – all probes, all detectors – against a production endpoint is a common first mistake. That’s how you burn through an API budget in an afternoon. Start with one probe family (promptinject or dan), get familiar with the report format, then scale up deliberately.

Frequently asked questions

Do I need a GPU to run Garak?

No. Against API-hosted models (OpenAI, Bedrock, NIM, Anthropic via LiteLLM), Garak just sends HTTP requests. GPU only matters if the target is a local Hugging Face model running through the pipeline generator.

How does Garak differ from NeMo Guardrails?

Different jobs. Garak scans a model before deployment to find failure modes; NeMo Guardrails enforces policy on requests at runtime. NVIDIA uses both as part of its Trustworthy AI process – they’re designed to work together, not compete. The practical pipeline: Garak in CI to catch regressions when a model version changes, NeMo Guardrails in production to intercept bad prompts in real time. One is a testing tool, the other is a traffic cop.

Can I write my own probe?

Yes, and it’s less involved than it sounds. Inherit from Garak’s base TextProbe class, override the minimum interface, test locally with the built-in blank generator. The official reference docs have the full plugin contract. One misconception to clear up first: you don’t need to touch the detector or generator layers to write a probe – most custom probes are just a list of prompt strings plus a name. The dan/ and promptinject/ modules in the repo are small enough to read in an hour and serve as better templates than any documentation.

Next action: pick one model you’re about to ship (or one already in production) and run python3 -m garak --target_type <yours> --target_name <yours> --probes promptinject --generations 3. Read the HTML report. The highest-failing probe is your first real security ticket for the quarter.