Mesh LLM on iroh: Beginner’s Guide to Distributed AI

“I’ve got a decent gaming PC, a Mac mini, and an old workstation with a 3090 – can I just… use all of them together to run one big model?” That’s the question popping up in every AI forum right now, and the answer is finally a real one. Mesh LLM, a distributed AI project built on iroh, just dropped and hit 179 points and 41 comments on Hacker News – with the community immediately asking where the benchmarks are. This is the beginner walkthrough for that question, with the gotchas the launch posts skipped.

The scenario this actually solves

You have GPUs scattered across machines. One box has 24 GB of VRAM, another has 16, a laptop has 8. Any single one of them can run a small model. None can run a 70B at usable quantization. Buying a bigger card costs more than the machines combined.

Mesh LLM’s pitch: you don’t have to pick one. Pool the memory across boxes – over your LAN or the internet – and get back a single endpoint that behaves like OpenAI’s API.

What it actually is

A Rust binary on each machine. MIT licensed, built with iroh and llama.cpp (as of mid-2025, per the GitHub README). Iroh is the peer-to-peer networking layer – it punches through NAT so your machines find each other without router config. Mesh LLM sits on top, deciding per request: run locally, forward to a peer that already has the model loaded, or split a giant model across nodes as a pipeline. One OpenAI-compatible endpoint on port 9337. Add nodes later; the mesh sorts out where the work lands.

The right mental model: This isn’t “my own ChatGPT server.” It’s a way to keep a local OPENAI_BASE_URL that stays sane as your hardware collection grows. That reframe makes the tradeoffs worth it – or not, depending on what you’re building.

Setup

On any macOS or Linux node:

curl -fsSL https://meshllm.cloud/install.sh | bash

The installer probes hardware and picks the right acceleration backend – CUDA, ROCm, Metal, Vulkan, or CPU. No manual flags. Then start a private mesh on your first box:

mesh-llm --auto --model GLM-4.7-Flash-Q4_K_M --mesh-name "home-lab"

Turns out the --mesh-name flag also implies --publish, and it strictly filters discovery to that name only – your group won’t accidentally merge with a stranger’s mesh. Run the same command on your other machines. That’s the whole setup. Test from any node:

curl http://localhost:9337/v1/chat/completions 
 -H "Content-Type: application/json" 
 -d '{"model":"GLM-4.7-Flash-Q4_K_M","messages":[{"role":"user","content":"hello"}]}'

A web console also runs on port 3131 alongside the API if you prefer a UI (as of mid-2025).

The 80ms rule – read this before adding remote nodes

Almost no launch coverage mentions this. It’s the single most important thing to understand before you plug in a friend’s machine across the country.

The README lays it out: if a model fits on one node (VRAM ≥ size × 1.1), it runs solo – split only when the VRAM math forces it. When splitting is needed, peers get sorted by RTT, lowest first. The catch: 80ms is a hard ceiling. Nodes above that threshold participate as API clients, not as split partners. They’ll show up in your mesh, accept requests, forward completions – but they won’t be part of a layer pipeline.

Practically: your gaming PC in the next room is a great split partner. A friend’s box 200ms away is a coordination node, not a compute pool. Ping first. If ping shows 90ms, you know before you start.

Splitting a model that doesn’t fit anywhere

Ten tokens per second. That’s what two Mac Studios on 1 Gbps ethernet managed running GLM 5.2 Q2 MTP together – reported by the author in the HN thread. Not fast. Not zero either, on a model neither machine could hold alone.

The mechanism (called Skippy internally): a model gets partitioned by layer ranges into pipeline stages. Layers 0-15 on node one, 16-31 on node two, activations flowing forward. The technique itself is a variant of pipeline parallelism across heterogeneous networks, formalized in SkipPipe (arXiv:2502.19913). What Mesh LLM adds is packaging – you don’t need to be a distributed systems engineer to try it.

The catalog ships with 40+ models as of mid-2025, from sub-billion-parameter options up to a 235B mixture-of-experts model. Whether consumer-grade pipeline parallelism ever gets close to single-GPU throughput on real workloads is still an open question – the HN thread flagged the absence of formal benchmarks, and nobody’s published controlled comparisons yet.

The “mesh” virtual model

Two or more different models loaded across your nodes? There’s a shortcut. Send "model": "mesh" and the proxy fans out to every model in parallel, then arbitrates the responses in code. A mixture-of-agents pattern, free and automatic. The requirement: at least two distinct models in the mesh. A single-model setup silently skips this route – no helpful error, just ordinary single-model behavior. That behavior is only documented in MOA_GATEWAY.md, not the main README.

curl http://localhost:9337/v1/chat/completions 
 -H "Content-Type: application/json" 
 -d '{"model":"mesh","messages":[{"role":"user","content":"What is the capital of Japan?"}]}'

Where it breaks

Straight from the maintainers and the community’s first weeks with it:

Production-ready? No. Easiest distributed inference option to actually try right now? Probably – for someone with spare hardware and low tolerance for YAML.

FAQ

Do I need to open ports on my router?

No. Iroh handles NAT traversal via hole-punching and falls back to a public relay when direct connections fail.

Can I use this with existing agent tools like Claude Code or Continue?

Yes. Point OPENAI_BASE_URL at http://localhost:9337/v1 and any OpenAI-compatible client works – curl scripts, Continue, goose, Claude Code’s custom endpoint setting. Tested pattern in the community: swap the base URL, keep every other config line identical. No rewriting around a proprietary API. That’s a big part of why this got traction fast.

Is the default public mesh safe to join?

Traffic is model-serving, not file access, and that’s how the project is designed to be tried. But if you’re on a work machine or care about which hardware serves your prompts – skip --auto, use your own --mesh-name from the first command. Private meshes filter strictly by name and don’t broadcast publicly. Starting private adds maybe 30 seconds to setup.

Your next step: install on your two closest machines, ping between them to confirm you’re under 80ms, start both with the same --mesh-name. If that works, load a model that fits on neither box alone and see what Skippy does with your hardware.