QWE AI Academy Most tutorials about building a local LLM personal assistant get the priority wrong. They obsess over which model to download, argue about Ollama vs LM Studio for paragraphs, then ship you a chatbot with amnesia and call it an assistant.
An assistant that forgets what you told it yesterday isn’t an assistant. It’s a search box that talks back. The interesting work – the part nobody covers – is the layer around the model. So that’s where this guide spends most of its time.
The scenario: you, a 16GB laptop, and a real problem to solve
You want something like ChatGPT, but running on your own machine. No OpenAI reading your journal entries, your client notes, or your half-written business plan. You have a reasonably modern laptop – a MacBook with 16GB RAM, or a Windows machine with a mid-range GPU – and about an hour.
Here’s what’s realistic in that hour: a working chat interface to a small open-weight model that runs entirely offline. Plan for at least 8 GB of system RAM for 7B-parameter models, with 16 GB recommended once you move up to 13B (as of 2025, per Sitepoint’s comparison). Models on disk range from roughly 2 GB to over 40 GB.
What’s not realistic in that hour: a Jarvis-style assistant that remembers your meetings, executes actions across your apps, and reasons over your documents. That’s a project, not a download.
Ollama or LM Studio – and why it barely matters
Same inference engine. Both run on llama.cpp under the hood, so raw speed is nearly identical (Zen van Riel’s comparison confirmed this). The choice is workflow, not performance.
| Aspect | Ollama | LM Studio |
|---|---|---|
| Interface | Terminal commands | Desktop GUI (~500 MB app) |
| Model discovery | Curated registry (ollama.com/library) | Hugging Face search inside the app |
| Custom behavior | Modelfiles (system prompt, temp, context) | Visual sliders |
| API | Always-on REST at localhost:11434 | Optional OpenAI-compatible server |
| Best for | Scripting, building tools on top | Trying models, prompt tuning |
Building automations or a custom UI? Ollama. Its API is always running, OpenAI-compatible, and most OpenAI-compatible libraries drop in without changes – point them at http://localhost:11434/v1 and swap the model name. Just experimenting visually? LM Studio is the smoother start. Honest answer: install both. They coexist fine.
The 15-minute setup
- Install Ollama from the official site. On macOS and Linux it sets itself up as a background service. On Apple Silicon, Metal GPU acceleration works out of the box as of early 2025 – unified memory means the whole model loads into fast shared RAM (Claude5 guide).
- In a terminal:
ollama pull llama3.2(ormistral, orqwen2.5). Pick one. Don’t agonize. - Run
ollama run llama3.2. You’re chatting. - Want a ChatGPT-style web UI? Install Open WebUI – it plugs into Ollama’s API and adds conversation history, document upload, and multi-model chat.
That’s the base layer. Now the actual work starts.
The part nobody covers: giving your assistant a memory
An LLM is stateless. Close the window – conversation gone. Open it tomorrow – model has no idea who you are. That’s the gap between “local LLM” and “personal assistant,” and it’s wider than most guides admit.
Turns out the Towards AI architecture guide frames this exactly right: a real assistant needs a context store holding both short-term conversational state and long-term working state – timers, the last task you paused, the article you’re halfway through. Before the LLM reasons about a new request, it reads from this store. After it answers, it writes updates back.
Start with this: a local SQLite file. Two tables – one for facts the assistant should always know about you (name, projects, preferences), one for conversation summaries. Before every prompt, your wrapper script prepends the relevant rows as a system message. After every session, ask the model to summarize what happened and write it back.
One thing to watch: Don’t stuff your entire history into the context window. Summarize hard. A 200-word summary of last week beats a 20,000-token raw transcript – shorter, denser context gets more model attention than long, sparse context.
The plumbing nobody wants to write but everyone needs. This is what separates a local LLM that feels like an assistant from one that just feels like a stranger you keep re-introducing yourself to.
The limits nobody mentions in tutorials
Before you commit a weekend to this, five real gotchas:
Silent failures on memory overrun. Load a model that exceeds your available RAM and you won’t get a clean error – you’ll get a freeze or crash. Sitepoint’s comparison flags this specifically: “Attempting to load a model that exceeds available memory can cause silent failures or system instability.” Stay one size tier below your maximum.
Tool-calling is unreliable. Want your assistant to actually do things – read files, send messages, control apps? Small local models frequently misformat tool calls. There were reported issues with tool-calling on Ollama-hosted models as of December 2025 (per the Towards AI guide). This may have improved – verify with your specific model before building anything serious around it.
“It runs” is not “it’s usable.” Three minutes. That’s how long one JDriven engineer waited for a larger model to answer a real coding question (JDriven blog, Nov 2025). Dropping to mistral:7B cut it to ~15 seconds – workable, but still slow. For context: on an M2 with 16GB, a 7B model generates roughly 15-25 tokens/second (Codiste benchmark). Comfortable for chat. Not comfortable for coding loops.
Not 100% offline by default. Both Ollama and LM Studio check their registries for model updates on startup. Per DevToolReviews (2026), the metadata check doesn’t include your prompts or outputs – but if full network isolation matters, disable update checks or block the binaries at your firewall.
Local coding assistants trail cloud ones. Frontier models are an order of magnitude larger than what fits on a laptop. For sensitive code, local is the right tradeoff. For raw output speed, the gap is real.
What to build next
RAG over your own documents. Drop notes, PDFs, and project files into a folder. Index them with a local embedding model – ollama pull nomic-embed-text takes one command – plus a tiny vector store like Chroma or sqlite-vec. Your assistant can now answer questions about your stuff, not just whatever made its training cut-off.
A Modelfile per role. Ollama’s Modelfile system bakes a system prompt, temperature, and context window into a named variant. writing-coach with a sharp editor’s prompt. journal with a warmer one. Switch by name – no retyping prompts each session.
A scheduled job hitting the API. Cron + curl + Ollama’s localhost API is enough for a daily-summary assistant that reads your notes and emails you a recap. Crude. Powerful.
FAQ
Do I need a GPU for a local LLM personal assistant?
No. A modern Mac with Apple Silicon runs 7B models fine via Metal. GPU-less Windows or Linux machines handle 3B-7B too, just slower.
How is this different from just paying for ChatGPT Plus?
If you want the best chatbot and that’s it – honestly, ChatGPT or Claude still wins. Frontier models are bigger and smarter than anything that fits on a laptop. Local makes sense in three specific situations: you handle data that can’t touch the cloud (legal, medical, personal journals, proprietary code); you want zero recurring cost and can accept a quality drop; or you need a always-available API to build custom automation on without rate limits. None of those apply to you? $20/month for a hosted model is a rational choice.
Which model should I download first?
Run ollama pull llama3.2:3b. Small, fast, enough to test the whole pipeline. Upgrade once you know what’s missing.
Next action: Open a terminal right now, install Ollama, run ollama pull llama3.2:3b. Don’t read another comparison first. The 10 minutes of friction you’ll hit installing it teaches you more than an hour of reading.