There’s a moment every builder has with local LLMs. You type a prompt, the answer streams back instantly, and you realise no API call happened. No token bill. No data leaving your machine. Just your laptop, thinking.
I’ve been running local models on my MacBook Pro (M5 Max, 36 GB unified memory) for months now, and they’ve quietly become part of my daily workflow. Not for everything — I still use Claude for the heavy lifting on Dszape, my hotel management SaaS. But for a specific set of jobs, local models are genuinely better than the cloud.
This is the guide I wish someone had handed me at the start. No hype, no benchmark worship. Just what fits in what RAM, what quantisation actually means, and what’s worth running locally versus what isn’t.
One caveat before we start: I’m not a career engineer. I spent 10 years in startup operations before building software solo with AI. So this is written in plain operator language, because that’s the only language I have.
Why Apple Silicon Is Quietly the Best Local LLM Machine
Most people think you need an NVIDIA GPU rig to run models locally. For serious training, sure. But for inference — actually using models — Apple Silicon has a structural advantage that surprised me when I first understood it.
It’s called unified memory. On a normal PC, your GPU has its own separate memory (VRAM), usually 8 to 24 GB, and models must fit inside it. On a Mac, the CPU and GPU share one pool of memory. My 36 GB machine can hand most of that to a model.
That means a mid-range MacBook can comfortably run models that would require an expensive dedicated GPU on the PC side. No drivers, no CUDA setup, no fan noise like a jet engine. You install one tool and you’re running.
The trade-off is speed. A top-end NVIDIA card will generate tokens faster. But for the workloads I care about — drafting, voice generation, audio processing — the Mac is more than fast enough, and it’s the machine I already own.
What Model Sizes Fit in What RAM
This is the question everyone asks first, so let’s answer it properly.
Models are measured in parameters — 7 billion (7B), 13B, 30B, 70B and so on. More parameters generally means smarter, but also more memory. With standard 4-bit quantisation (more on that below), here’s the honest picture:
- 8 GB Mac: small models only. 3B to 7B class. Fine for autocomplete-style tasks and simple drafting. You’ll feel the ceiling quickly.
- 16 GB Mac: the 7B to 13B class runs comfortably. This is where local models start being genuinely useful for summarising, drafting, and classification.
- 32 to 36 GB Mac (mine): the sweet spot. 13B models fly, and the 30B class — which is where quality gets impressive — runs comfortably with room left for your actual work.
- 64 GB and up: 70B-class models become practical. This is approaching last year’s cloud-model quality, on a laptop.
A rough rule of thumb that has served me well: a 4-bit quantised model needs roughly 0.6 GB of RAM per billion parameters, plus a few GB of overhead for context. A 30B model lands around 18 to 20 GB. On my 36 GB machine that leaves enough headroom to keep my browser, editor, and FL Studio open at the same time.
One thing nobody tells you: leave RAM for your life. If a model eats 30 of your 36 GB, everything else on the machine crawls. I treat 20 to 22 GB as my practical ceiling for a resident model.
Quantisation, Explained Like You’re an Operator
Quantisation is the reason any of this is possible, and it’s simpler than it sounds.
A model’s parameters are just numbers. Stored at full precision, each number takes 16 or 32 bits. Quantisation stores them at lower precision — 8 bits, 4 bits, sometimes less. Think of it like compressing a photo: you lose a little fidelity, but the file becomes a quarter of the size.
The surprising part is how little quality you lose. A 4-bit version of a model is usually 95%+ as capable as the full-precision original, at roughly a quarter of the memory. That’s why 4-bit (you’ll see it written as Q4) is the default everyone uses.
The practical guidance:
- Q4: the default. Use this unless you have a reason not to.
- Q5 / Q6 / Q8: slightly better quality, noticeably more memory. Worth it only if you have RAM to spare.
- Q2 / Q3: aggressive compression. The model gets visibly dumber. I’d rather run a smaller model at Q4 than a bigger one at Q2.
That last point is the one mistake I see most often: people squeeze a 70B model into memory at brutal quantisation and get worse results than a clean 30B would have given them.
Getting Started: Ollama in Five Minutes
I use Ollama and I’ve never needed anything else. It’s the Vercel of local models — it makes the annoying parts disappear.
The entire setup:
brew install ollama
ollama run llama3.1That’s it. First run downloads the model; after that it’s instant and offline. ollama list shows what you’ve got, ollama pull <model> fetches new ones, and Ollama exposes a local API on port 11434 so any script or app on your machine can call your models exactly like it would call a cloud API.
That local API is the underrated feature. It means every tool you build can be pointed at localhost instead of a paid endpoint. When I’m iterating on a prompt fifty times in an hour, that’s the difference between a free experiment and a bill.
What I Actually Run Locally
Theory is cheap, so here’s my real workload. These are the jobs where local won on merit, not ideology.
A persistent voice for my AI assistant
I run a local TTS model (Voxtral-4B) that gives my always-on AI assistant a consistent voice. Text-to-speech is exactly the kind of workload local models are made for: it runs constantly, the quality bar is “good enough and consistent” rather than “frontier”, and paying per generated second of audio would be absurd for something that talks to me all day.
A fully offline vocal mastering studio
I produce Konkani music on the side, and I built a local AI vocal mastering pipeline — autotune, a processing chain, loudness normalisation — that runs entirely on this machine. Free, offline, unlimited iterations. Mastering is iterative by nature; you tweak and re-render dozens of times. Doing that against a paid API would either bankrupt the hobby or make me ration my experiments. Locally, iteration costs nothing.
Drafting and private text work
First drafts, rewrites, summarising my own notes, working through ideas that aren’t ready for anyone else’s servers. A local 13B to 30B model handles this fine. It’s not as sharp as a frontier model, but for a first pass it doesn’t need to be — and there’s something freeing about drafting against a model where nothing leaves the room.
The experiment queue: a model that writes like me
The next thing on my list is fine-tuning a small local model on my own writing — a “voice model” that drafts in my style instead of generic AI-speak. Local is the only sane place to run that experiment, because it’s private data and it will take many failed attempts to get right. I’ve written up the thinking behind that in my guide to fine-tuning vs RAG vs prompting.
When Local Is the Wrong Tool
This is the section most local-LLM content skips, so let me be blunt about where local models lose.
Coding agents on a real codebase. Dszape is 719 source files, 245+ API routes, and 180+ SQL migrations. When an AI is writing production code against that — billing logic, database migrations, multi-tenant auth — I want the smartest model that exists, full stop. I use Claude for all of it. A local 30B model writing my folio billing engine is a production incident with extra steps, and I’ve had enough real ones to know the cost. I wrote about those in my solo SaaS journey.
Anything where wrong answers are expensive. Legal, financial, customer-facing. The gap between a good local model and a frontier model shows up exactly when the stakes rise.
Long-context work. Feeding in a whole codebase or a 200-page document. Local models have shorter usable context windows and slow down badly as context grows.
When you haven’t done the maths. If you only run a few thousand tokens a day, cloud API costs are pennies and local buys you nothing but tinkering time. Local wins on volume and privacy, not on principle.
My actual split is simple: frontier cloud models for anything touching production code or high-stakes output; local models for private, repeated, or creative workloads where iteration volume is high and “very good” is good enough. I’ve broken that decision framework down fully in Local LLM vs Cloud API: What Startups Should Actually Use.
The Practical Setup I’d Recommend Today
If you’re starting from zero on a Mac:
- Install Ollama via Homebrew.
- Pull one good general model in the biggest size your RAM allows at Q4 (use the rule of thumb above).
- Give it a real job from your actual week — summarise your meeting notes, draft your emails — not a benchmark puzzle.
- Only add more models when a real task demands one. Model hoarding is the local-LLM equivalent of buying gym equipment.
And if you’re buying a Mac partly for this: RAM is the whole game. The chip generation matters far less than the memory. A machine with 36 GB or more keeps the entire interesting middle class of models (13B to 30B) on the table.
FAQ
How much RAM do I need to run local LLMs on a Mac?
16 GB is the honest minimum for useful work (7B to 13B models). 32 to 36 GB is the sweet spot, comfortably running 30B-class models with headroom for your other apps. 8 GB limits you to small models that feel like toys quickly.
Are local LLMs as good as ChatGPT or Claude?
No, and it’s important to be honest about that. A well-chosen 30B local model is roughly comparable to cloud models from a year or two ago — genuinely useful, but a tier below the current frontier. For drafting, summarising, TTS, and creative iteration the gap rarely matters. For production code and high-stakes reasoning it absolutely does.
Is Ollama free?
Yes. Ollama is open source and the models it runs are open-weight models that cost nothing to download and use. Your only costs are the disk space, the electricity, and the Mac you presumably already own.
Will running LLMs damage or overheat my MacBook?
No. Inference is a heavy workload, so you’ll hear fans during long generations, but it’s no more harmful than video rendering or gaming. Apple Silicon is remarkably efficient at it — my M5 Max barely warms up on 13B models.
Can I fine-tune models locally on a MacBook?
Fine-tuning small models (up to roughly 7B, using efficient methods like LoRA) is feasible on a high-RAM Mac, though slow compared to cloud GPUs. That’s exactly my plan for the voice-model experiment. For anything larger, rent cloud GPUs for the training run and bring the finished model home to run locally.
I write about building SaaS solo with AI, from the trenches — the wins and the production incidents alike. If this was useful, there’s more where it came from: start with how I build with AI day to day or my full AI tool stack, and you can read more about me here.
