Install, download and use local open source LLMs on your machine.
A complete, no-nonsense guide to running open-source LLMs locally on Apple Silicon, with a clean terminal workflow and a browser-based chat interface. Tailored for a MacBook Pro M5 with 16 GB of unified memory, but the principles apply to any Apple Silicon Mac.
This setup combines four pieces:
~/.zshrc to start and stop everything easilyBy the end of this guide, you'll have a fully local AI assistant accessible from your browser with a single terminal command, with no data leaving your machine.
On 16 GB of unified RAM, the available memory budget for a model is roughly 8–10 GB once macOS, your browser, and your daily apps are loaded. This puts us in the 7B–9B parameter range with Q4 quantization. Larger models (30B+, including MoE architectures like qwen3-coder-next) will either refuse to run or swap heavily to disk, making them unusable in practice.
The two models chosen are both from the Qwen 2.5 family by Alibaba, Apache 2.0 licensed:
qwen2.5-coder:7b — A code-specialized variant fine-tuned on programming tasks. Strong on Python, JavaScript, PHP, SQL, shell scripting, and similar languages. Use this for all code generation, refactoring, debugging, and technical commenting in English. Disk footprint: ~4.7 GB.
qwen2.5:7b — The general-purpose variant. Better at natural language tasks, especially in French and other non-English languages. Use this for writing, summarizing, brainstorming, translation, and any prose work. Disk footprint: ~4.7 GB.
Both models support up to 128K tokens of context natively, though Ollama caps it at 4096 by default. We'll override this to 32K, a good balance between usable context length and RAM usage.
A note on division of labor: don't be tempted to use the coder model for prose, even in English. Code-specialized models lose ground on natural language tasks, particularly in non-English languages. Keep the two roles strictly separate.
Assuming Homebrew is already installed on your system:
brew install ollamaStart Ollama as a background service so it launches automatically and stays running:
brew services start ollamaVerify it's responding:
curl http://localhost:11434You should see Ollama is running. The server listens on port 11434, which is what every tool in this stack will connect to.
Pull both models. This takes about 10–20 minutes depending on your connection, with each model around 5 GB.
ollama pull qwen2.5-coder:7b
ollama pull qwen2.5:7bCheck what's installed:
ollama listYou should see both models listed with their sizes.
By default, Ollama limits the context window to 4096 tokens, which is too small for serious work (reading a long script, analyzing a multi-file project, processing a sizeable text). We'll create custom variants of each model with a 32K context window, which is the sweet spot for 16 GB of RAM — enough for substantial context, without saturating memory.
Create a Modelfile for the coder variant:
cat > ~/Modelfile-coder << 'EOF'
FROM qwen2.5-coder:7b
PARAMETER num_ctx 32768
EOF
ollama create qwen-coder-32k -f ~/Modelfile-coderAnd one for the general variant:
cat > ~/Modelfile-general << 'EOF'
FROM qwen2.5:7b
PARAMETER num_ctx 32768
EOF
ollama create qwen-general-32k -f ~/Modelfile-generalThese custom variants point to the same underlying weights, so Ollama doesn't duplicate the model files on disk. Only the context configuration differs.
Tuning the context window: if ollama ps later shows the model + context taking too much RAM (say, more than 10 GB), reduce num_ctx to 16384. If you have headroom, you can push it to 65536 or higher, up to the model's native limit.
To start a chat session in the terminal:
ollama run qwen-coder-32kThis opens an interactive prompt where you can type questions or paste code. Useful keyboard shortcuts:
/bye to exit/? to see all commandsFor verbose mode that shows performance statistics (tokens per second, total duration, etc.) after each response:
ollama run qwen-coder-32k --verboseYou can also pipe input directly:
cat my_script.py | ollama run qwen-coder-32k "Explain this code"Or save the output:
ollama run qwen-coder-32k "Refactor this code [...]" > response.mdTo see what's currently loaded in memory:
ollama psThis shows the model name, size, processor (CPU vs GPU), context length, and how long until the model is unloaded from RAM due to inactivity.
Expected throughput on a base M5 with 16 GB RAM running a 7B Q4 model:
At 30 tokens per second, you're generating roughly 20 words per second — faster than reading speed, so the experience is fluid.
The ollama ps output shows 100% GPU, which means the model runs entirely on the M5's integrated GPU via Metal. If you ever see 50% GPU / 50% CPU or similar, it means the model is partially offloaded to CPU, which is much slower. Reduce num_ctx or close other apps if this happens.
Open WebUI is a polished, ChatGPT-like browser interface that connects to Ollama automatically. We install it using uvx, which provides isolated Python environments without manual venv management.
First, install uv:
brew install uvThen launch Open WebUI for the first time:
uvx --python 3.11 open-webui@latest serveThe first launch downloads dependencies (around 500 MB) and takes 30–60 seconds. Subsequent launches are much faster (5–15 seconds).
Once you see logs indicating the server is up, open http://localhost:8080 in your browser. On first visit, you'll be asked to create a local admin account — this is stored only on your machine, no data is sent anywhere.
In the Open WebUI interface, your two models (qwen-coder-32k and qwen-general-32k) appear in the model selector at the top. Pick the right one for the task at hand.
To stop Open WebUI from the terminal where it's running, press Ctrl+C.
~/.zshrcTo avoid typing long commands every time, add helper functions and aliases to your shell config. Open the file:
open -e ~/.zshrcAdd the following block at the end:
# ─── Open WebUI ──────────────────────────────────────────────
webui() {
# Make sure Ollama is running
if ! curl -s http://localhost:11434 > /dev/null 2>&1; then
echo "⚠️ Ollama is not running — starting it..."
brew services start ollama
sleep 2
fi
# Check if Open WebUI is already running on port 8080
if curl -s http://localhost:8080 > /dev/null 2>&1; then
echo "✓ Open WebUI already running — opening browser"
open http://localhost:8080
else
echo "🚀 Starting Open WebUI..."
nohup uvx --python 3.11 open-webui@latest serve > ~/.openwebui.log 2>&1 &
echo "⏳ Waiting for startup (10-20s usually, up to 2 min on first run)..."
for i in {1..120}; do
if curl -s http://localhost:8080 > /dev/null 2>&1; then
echo "✓ Open WebUI ready"
open http://localhost:8080
return 0
fi
sleep 1
done
echo "✗ Startup timed out — check the log with: webui-logs"
return 1
fi
}
webui-stop() {
pkill -f "open-webui" && echo "✓ Open WebUI stopped" || echo "Open WebUI wasn't running"
}
webui-logs() {
tail -f ~/.openwebui.log
}
# ─── Ollama shortcuts ────────────────────────────────────────
alias ollama-start='brew services start ollama'
alias ollama-stop='brew services stop ollama'
alias coder='ollama run qwen-coder-32k --verbose'
alias general='ollama run qwen-general-32k --verbose'
alias llm-off='ollama stop qwen-coder-32k 2>/dev/null; ollama stop qwen-general-32k 2>/dev/null; echo "✓ Models unloaded from RAM"'Save and close the file, then reload your shell config:
source ~/.zshrcWhat you get from this block:
webui — Starts Ollama if needed, starts Open WebUI in the background if it's not already running, then opens the browser. If everything is already up, just opens the browser tab.webui-stop — Stops Open WebUI cleanly. Useful before launching memory-intensive applications.webui-logs — Tails the Open WebUI logs for debugging. Exit with Ctrl+C.coder — Shortcut to start a terminal chat with the code model.general — Same for the general model.llm-off — Immediately unloads both models from RAM (frees ~7 GB) without stopping the Ollama service itself.ollama-start / ollama-stop — Start/stop the Ollama background service.After a fresh boot of your Mac, Ollama starts automatically (because we registered it with brew services). Open WebUI does not — by design, to avoid consuming resources when you don't need it.
To get the full stack running, just type:
webuiThat's it. The function checks that Ollama is up (starts it if not), launches Open WebUI in the background, waits for it to be ready, and opens the browser. After this, you can close the terminal window — Open WebUI keeps running in the background.
For chat work, use the Open WebUI browser tab. Switch between qwen-coder-32k and qwen-general-32k from the dropdown depending on the task.
For quick one-off questions in the terminal, the aliases coder and general are faster than opening a browser.
For piping files or scripting LLM calls into other workflows, use ollama run directly with stdin/stdout redirection.
Three levels of shutdown depending on what you need to free up:
Free RAM but keep everything ready to use (recommended when launching heavy apps like Rhino or running an EnergyPlus simulation):
llm-offThis unloads both models from RAM (frees ~7 GB each if both are loaded), without stopping Ollama or Open WebUI. Your next prompt will reload the model automatically, with a 2–3 second delay.
Stop Open WebUI but keep Ollama available:
webui-stopOpen WebUI stops, but you can still use ollama run in the terminal.
Stop everything:
webui-stop
brew services stop ollamaTo start it all again later, just run webui — Ollama gets restarted automatically.
By default, after a fresh Mac restart:
brew services), with no model loaded — minimal RAM impact (~50–100 MB)webui to launch itThis is intentional. Running Open WebUI 24/7 isn't a problem in itself (it idles at ~300 MB of RAM), but it's better to only start it when needed.
Apple Silicon uses unified memory — RAM is shared between CPU, GPU, and Neural Engine. On a 16 GB Mac, the practical breakdown when running a 7B model is:
This is workable, but tight. Monitor memory pressure with:
memory_pressureThe output gives a percentage of free memory. Above 80% free = no concern, 60–80% = system compressing pages (some background CPU cost), below 60% = swapping to SSD, and you'll feel slowdowns.
For a graphical view, open Activity Monitor → Memory tab → look at the "Memory Pressure" graph at the bottom. Green = fine, yellow = working, red = swapping.
If you need to free RAM urgently (about to launch Rhino, run a simulation, or start a memory-intensive task), run llm-off to unload the models instantly.
Open WebUI doesn't start, times out
Check the logs:
webui-logsCommon causes: Ollama isn't running, port 8080 is already in use, or uvx couldn't download dependencies (network issue). The first launch can take longer than expected — the timeout in the function is set to 120 seconds to accommodate this.
Open WebUI loses my admin account between sessions
This can happen if uvx re-creates its cached environment between runs. To force persistent data storage, edit the webui() function and change the launch line to:
DATA_DIR=~/.open-webui-data nohup uvx --python 3.11 open-webui@latest serve > ~/.openwebui.log 2>&1 &Create the data directory once:
mkdir -p ~/.open-webui-dataModel runs very slowly or shows 50% GPU / 50% CPU in ollama ps
The model + context is too big for available GPU memory. Lower the context size by editing the Modelfile and recreating the model:
cat > ~/Modelfile-coder << 'EOF'
FROM qwen2.5-coder:7b
PARAMETER num_ctx 16384
EOF
ollama create qwen-coder-32k -f ~/Modelfile-coderOr close other heavy applications.
Model gives nonsensical or low-quality answers
For a 7B local model, quality is inherently limited. Make sure you're using the right model for the task (coder for code, general for prose). For tasks that require strong reasoning, domain expertise (specialized APIs, niche fields), or careful nuance (academic writing in non-English languages), the model may be insufficient — consider falling back to a cloud frontier model for those specific cases.
Out of disk space
To see how much Ollama is using:
du -sh ~/.ollama/modelsTo remove a model you no longer use:
ollama rm model-nameTo clear the uvx cache (will trigger a redownload on next webui run):
uv cache cleanA local 7B model is a capable assistant for many tasks, but it's not a frontier cloud model. Here's where it excels and where it struggles:
Strong performance:
Weak or unreliable:
The right workflow is hybrid: use the local model for drafts, fast iteration, offline work, and privacy-sensitive content, and fall back to a cloud frontier model when quality or domain-specific accuracy matter more than speed, cost, or confidentiality.
Keep models updated: every few weeks, run:
ollama pull qwen2.5-coder:7b
ollama pull qwen2.5:7bAfter pulling, recreate your custom variants:
ollama create qwen-coder-32k -f ~/Modelfile-coder
ollama create qwen-general-32k -f ~/Modelfile-generalKeep Open WebUI updated: uvx automatically pulls the latest version of Open WebUI each time you run webui, because of the @latest tag. No manual update needed.
Keep Ollama updated:
brew upgrade ollama
brew services restart ollamaDisk cleanup: occasionally check du -sh ~/.ollama/models and du -sh ~/.cache/uv to make sure cached files aren't growing out of control.
You now have a fully local LLM stack running on Apple Silicon, with one command (webui) to start everything and clean shutdown options at three different levels. All data stays on your machine, there are no subscription costs, no rate limits, and it works offline.
For daily use, the workflow is straightforward: type webui in the morning, work in the browser tab throughout the day, run llm-off before launching memory-intensive applications, and webui-stop at the end of the day if you want to reclaim resources.
The two-model setup (coder + general) covers most use cases without bloating disk or memory, and the 32K context window is large enough for the vast majority of practical tasks. When you outgrow this — typically when you start wanting to run 30B-class models — that's the signal to consider a Mac with 32 GB or more of unified memory.
Last updated: 27-05-2026 at 08:26