Deploy Evidently AI v0.7.12: ML Model Monitoring Setup Guide

Here’s something most ML monitoring tutorials don’t mention: the Evidently codebase you’ll find in 90% of blog posts from 2024 won’t run on the current version. The v0.7 release introduced breaking changes – the new Evidently API became the default, migration is required, and most copy-pasted code fails silently with an ImportError before you even get to the interesting part.

This guide deploys Evidently AI v0.7.12 – the current release as of August 2025 – for ML model monitoring. Skip the marketing pitch. Straight to commands that actually work on the new API.

What you’re installing

An open-source ML and LLM observability framework, 6.9k GitHub stars, Apache-2.0, 20m+ downloads (as of August 2025). Turns out it covers everything from tabular data drift to Gen AI text evals – 100+ metrics total. You can use it as a one-off Python library that spits out standalone HTML reports, or spin up the full self-hosted web UI and watch metrics over time.

Two deployment paths: the OSS package you self-host, or Evidently Cloud, which adds dataset management, alerting, and no-code evals on top of a free tier. This guide covers OSS.

System requirements

The Python version isn’t arbitrary. V0.7.12 updated Dockerfiles to a 3.11-slim-bookworm base – that’s the tested target. Pyspark users: watch out. V0.7.7 hard-capped pyspark to version 3.x. Installing Evidently into an environment with pyspark 4.x will either downgrade your Spark or throw a resolver conflict. Separate venv, or pin pyspark to 3.x.

Install Evidently AI for ML model monitoring

The fast path: uv, the modern Python package manager. One command, no virtualenv setup, demo projects included:

uv run --with evidently evidently ui --demo-projects all

Most tutorials still default to the pip+virtualenv pattern and never mention this. If you don’t have uv, pip works:

# Isolated environment
python -m venv evidently-env
source evidently-env/bin/activate # Linux/macOS
# evidently-envScriptsactivate # Windows

# Old pip resolvers choke on Evidently's dependency tree - upgrade first
python -m pip install --upgrade pip

pip install evidently
# Or pin the exact version covered here:
pip install evidently==0.7.12

After installing, run evidently ui --demo-projects all and open localhost:8000. Done. Need only offline reports – no UI at all? Stop at pip install evidently and generate HTML from a notebook. 100 messages/day free on Cloud sounds generous, but one debugging session can burn through that. Offline HTML is often the right call for early prototyping.

First-time configuration: a real monitoring report

The new API changed import paths. If you’ve used pre-v0.7 Evidently, from evidently.report import Report and the old ColumnMapping class no longer work the same way – the migration guide documents the specific swaps. Here’s a minimum viable script using the current API:

import pandas as pd
from sklearn import datasets
from evidently import Report
from evidently.presets import DataDriftPreset

iris = datasets.load_iris(as_frame=True).frame

report = Report([
 DataDriftPreset(method="psi")
], include_tests="True")

result = report.run(iris.iloc[:60], iris.iloc[60:])
result.save_html("drift_report.html")

PSI (Population Stability Index) is a common choice for tabular drift – it measures distributional shift relative to the reference, rather than a fixed threshold. Run this once, open drift_report.html, and you have a baseline artifact you can wire into a cron job.

Reference window matters more than the metric: Don’t compare your reference dataset against itself with random sampling. Pick a fixed window – training data, or last month’s production data – and rerun against fresh batches. Drift is only meaningful when reference is stable.

What’s the right reference window for your use case, though? That depends on whether your model is time-sensitive (a fraud model in January is not the same as in December) or stable (a text classifier on evergreen content). There’s no universal answer. Running multiple reference windows in parallel and watching which one first signals drift is worth experimenting with before you commit to a monitoring cadence.

Verify the install

Three checks:

1. evidently --version – prints installed version
2. python -c "import evidently; print(evidently.__version__)" – confirms the package loads
3. http://localhost:8000 after evidently ui – demo projects visible means the service is up

Common install errors and fixes

ImportError on ColumnMapping or old evidently.report paths – you’re running code written for v0.4-v0.6. The catch is that v0.7 replaced ColumnMapping with DataDefinition and restructured the import tree. Either port the code using the migration guide, or pin pip install "evidently<0.7" temporarily.

ERROR: Could not find a version that satisfies the requirement evidently – Python too old, or pip is. Check python --version is 3.9+, upgrade pip, retry.

Pyspark version conflict – environment already has pyspark 4.x. Install Evidently in a separate venv, or pin pyspark to 3.x in your project.

UI loads but shows no projects – launched evidently ui without --demo-projects all and haven’t created a project yet. Expected. Relaunch with the flag or create one via the Python API.

Docker build can’t read from S3 or GCS – the default service image doesn’t include cloud storage extras. V0.7.12 added support for installing optional extras in the service Docker image specifically for S3 and GCS. Rebuild with those extras enabled.

Upgrade and uninstall

Upgrading from pre-0.7? Expect to rewrite imports – there’s no shim. The migration isn’t massive, but it’s real work if you have a lot of report definitions.

1. Freeze the current env: pip freeze > requirements-old.txt
2. New venv, install latest: pip install -U evidently
3. Port report definitions – Reports now take a list of Presets/Metrics in the constructor; report.run(current, reference) returns a result object you call .save_html() on
4. Keep the old env alive in parallel until the new pipeline is verified

To uninstall cleanly:

pip uninstall evidently
rm -rf workspace/ # local UI snapshot storage
rm -rf ~/.evidently/ # config cache, if present

The UI service stores snapshots as JSON files in your working directory by default. Delete those if you don’t want orphaned monitoring data.

FAQ

Is Evidently AI free?

The OSS library is fully free, Apache-2.0. Evidently Cloud has a free tier with paid plans above it.

Can I use Evidently with Grafana instead of the built-in UI?

Yes – and if your org already runs Grafana for service monitoring, plugging drift metrics into the same dashboard avoids tool sprawl. You export Report results as JSON or push numerical metrics to PostgreSQL or Prometheus; Grafana reads from there. The trade-off: you lose the rich interactive report views Evidently renders natively – those don’t translate into Grafana panels. Whether that matters depends on whether you need stakeholders browsing reports or just engineers watching alert thresholds.

Does it work for LLM monitoring or just classical ML?

Both. Text descriptors – sentiment, length, toxicity, semantic similarity – sit alongside tabular drift and classification metrics in the same framework.

Next step: pick one model in production, run the script above with last week’s data as current and your training set as reference, check what drifted. That’s the smallest possible useful deployment. Everything else is scaling that loop.