Install Self-Operating Computer v1.5.8: Open Source Computer Use Guide

By the end of this guide you’ll have the Self-Operating Computer Framework running on your machine, watching your screen, and clicking buttons on your behalf. You’ll know which install path to pick, which $5 trap waits for you on the OpenAI side, and how to recover when Pillow refuses to build. This is the most established open source computer use project out there, and it deserves a real walkthrough – not a copy of its README.

Quick framing before commands: Self-Operating Computer is the HyperwriteAI/OthersideAI framework that lets a multimodal model view your screen and decide on mouse and keyboard actions to reach an objective, released in November 2023 as one of the first full computer-use projects. We’re installing v1.5.8, tagged February 28, 2025 – the current latest as of writing.

What you’re actually deploying (and when not to)

You’re installing a Python CLI called operate that screenshots your primary display, sends the image to a vision model, and replays the model’s coordinate decisions through PyAutoGUI. Three modes ship in the box: vanilla vision, OCR-assisted, and Set-of-Mark. Based on the project’s own tests, OCR performs better than SoM and vanilla, so OCR is the default – invoked simply by running operate or operate -m gpt-4-with-ocr.

Honest take: if you want a polished GUI app and don’t need to fork the agent loop, Open Interface is worth looking at for a friendlier experience. Pick Self-Operating Computer when you want to read the agent code, swap models, or experiment with SoM prompting. It’s the reference implementation, not the consumer product.

System requirements

The Linux line matters. The official README states compatibility with Linux with X server – there’s no documented Wayland support. On Ubuntu 22.04+ default sessions you’ll likely need to log out, pick “Ubuntu on Xorg,” and log back in.

Get the source (and pick the right install method)

Two paths exist. The PyPI package self-operating-computer lets you pip install in one line, but it lags behind the GitHub main branch. The clone-and-install path gives you v1.5.8 today and lets you swap the SoM weights.

Most tutorials default to pip install self-operating-computer and move on. That’s fine if you only need the OCR mode. If you want SoM or plan to read the agent loop, clone the repo.

# Clone v1.5.8
git clone https://github.com/OthersideAI/self-operating-computer.git
cd self-operating-computer

# Python venv (use 3.11 if you have Pillow trouble on 3.12)
python3 -m venv venv
source venv/bin/activate # Windows: venvScriptsactivate

# Install deps and the CLI
pip install --upgrade pip
pip install -r requirements.txt
pip install .

The pip install --upgrade pip line isn’t cosmetic – it’s the workaround for the most reported install failure.

First-run configuration

Rename the env template, paste your key, and grant OS permissions. That’s the entire minimum config.

1. mv .example.env .env (Windows: ren .example.env .env)
2. Open .env and set OPENAI_API_KEY='sk-...'
3. Run operate once. On macOS, Terminal will prompt for Screen Recording and Accessibility under System Settings → Privacy & Security. Approve both, then restart Terminal.

Here’s the gotcha nobody flags clearly: the gpt-4o model requires your OpenAI account to have spent at least $5 in API credits before access unlocks – pre-paying for credits will unblock it if you haven’t already (as of early 2025; check the OpenAI help center if the behavior has changed). If you created a fresh key on a brand-new account, your first operate call will return a 404 “model_not_found” and look like a bug. It isn’t. It’s billing.

Pro tip: Before your first run, pre-pay $5 on the OpenAI billing page and wait about ten minutes. Then test with curl https://api.openai.com/v1/models | grep gpt-4o – if you see it listed for your account, you’re cleared.

Verify it works

Three checks, fastest first.

# 1. CLI is on PATH
operate --help

# 2. Dry-run with a trivial prompt
operate --prompt "open a new browser tab and search for 'hello'"

# 3. Try SoM mode (loads the bundled YOLOv8 weights)
operate -m gpt-4-with-som

If step 2 actually moves your cursor and types into a browser – congrats, you’re running open source computer use locally. If it freezes after the first screenshot, the model call is failing (usually the $5 gate above).

The SoM mode is more interesting than the docs make it sound

The framework supports Set-of-Mark (SoM) Prompting via the gpt-4-with-som command. The technique, from Yang et al., arXiv:2310.11441, overlays numeric marks on segmented screen regions – this improves visual grounding for large multimodal models, letting GPT-4V outperform fully fine-tuned grounding models in zero-shot benchmarks.

The bundled YOLOv8 weights (model/weights/best.pt) are trained for generic button detection. Swap them out for weights trained on screenshots of your specific UI – a CRM, a niche internal tool – and grounding accuracy jumps noticeably. Per the official README, users are explicitly encouraged to drop in their own best.pt. No tutorial I’ve seen calls this out.

Common install errors and fixes

• Could not build wheels for Pillow on Windows + Python 3.12. Per a documented PyAutoGUI issue, the error reads: “Could not build wheels for Pillow, which is required to install pyproject.toml-based projects.” Fix: downgrade to Python 3.11 or run pip install --upgrade pip setuptools wheel before pip install -r requirements.txt.
• operate: command not found. You skipped pip install . (which registers the CLI entry point) or your venv isn’t active. Re-activate and reinstall.
• 404 model_not_found on first call. The $5 OpenAI credit gate covered above.
• Mouse moves but clicks land off-target on macOS. Display scaling. Set your display to the default “Looks like” resolution; Retina plus scaled resolutions can confuse PyAutoGUI’s coordinate math.
• Linux: blank screenshot or PyAutoGUI throws DISPLAY error. You’re on Wayland. Switch to an X session.

Local model, upgrade, and uninstall

Want to avoid sending screenshots to OpenAI? LLaVA via Ollama is the local option. Ollama supports macOS and Linux (Windows in preview as of this writing) and the model itself runs about 5 GB of storage. One honest caveat from the README: error rates with LLaVA are very high. Treat it as a research experiment rather than a reliable daily driver.

Upgrading from an older clone: git pull && pip install -r requirements.txt && pip install . from inside your activated venv. The .env file is gitignored, so your key survives. If you installed via PyPI instead, pip install --upgrade self-operating-computer.

Uninstall is just as cheap: pip uninstall self-operating-computer, then rm -rf venv self-operating-computer. On macOS, revoke Terminal’s Screen Recording and Accessibility permissions in System Settings if you don’t plan to reuse them – leaving them on for a Terminal app is a real attack surface worth thinking about.

FAQ

Is the OpenAI cost predictable, or will it surprise me?

It will surprise you. Each agent step fires a separate vision API call, and complex objectives stack up fast. Set a hard usage limit in your OpenAI dashboard before your first serious test – don’t rely on estimating upfront.

How does this compare to Anthropic’s Claude Computer Use?

Different layer of the stack. Claude’s computer-use is a model capability you call via Anthropic’s API with a tool-use schema – you still write the loop that takes screenshots and executes actions. Self-Operating Computer is that loop, pre-built, and model-agnostic. If you want a working agent today, this project gets you there faster. If you want the most capable single model behind the steering wheel and don’t mind building the use, go direct to Claude.

Can I run it headless on a server to automate a remote desktop?

Not really. PyAutoGUI needs a real display. Xvfb works as a community workaround on Linux, but click accuracy degrades and voice mode breaks entirely.

Next step: pre-pay the $5 OpenAI credit, clone v1.5.8, and run operate --prompt "take a screenshot of the active window" as your smoke test. If that works, swap to a real objective.