Primer: How To Run AI Models Locally
By now, you’ve probably tried ChatGPT or Microsoft Copilot, or Grok or Claude.ai. These are all server-hosted AI services. But you can also run AI models locally, too. If you’ve got the right hardware, it gives you privacy and an unmetered (read: no usage cost) way to experiment with the various models.
First, we need a place to browse the various models. Have you heard of “Hugging Face“? It’s named after the whimsical “hugging face” emoji:
What Hugging Face is
Hugging Face is a hosting site for AI models, plus the tooling around them. Think GitHub, but for model weights, datasets, and demo apps.
Three parts matter:
- The Hub: at this writing, there are about 2 million model repositories. Each repo holds weight files, a config, and a model card (the README).
- Libraries:
transformers,diffusers,datasets. Python code that loads and runs those files. - Spaces: small hosted web demos.
Hugging Face does not make most of the models. It stores what Meta, Alibaba (Qwen), Google (Gemma), Mistral, DeepSeek, and thousands of individuals publish.
One caution on the word “open”. Most of these are open weight, not open source. You get the weights and a license. You do not get the training data. Licenses vary, so read them before commercial use.
Can you run them locally?
Yes, for many of them. The limit is memory, not permission.
A rough rule: a model needs about 0.6 GB of RAM or VRAM per billion parameters at 4-bit quantization. Quantization compresses the weights and costs a little quality.
| Your memory | What runs well |
|---|---|
| 8 GB | Qwen3 8B, Gemma 3 4B |
| 16 GB | Qwen3 14B, Gemma 3 12B, gpt-oss 20B |
| 24 to 32 GB | Qwen3 30B-A3B, Qwen3-Coder 30B |
| 48 GB+ | Llama 3.3 70B, gpt-oss 120B |
Apple Silicon Macs are good at this. The unified memory means a 64 GB MacBook Pro can hold models that need a very expensive NVIDIA card otherwise.
The largest frontier models (DeepSeek at 600B+ parameters) do not fit on a laptop. They need a server rack.
Is there a standard runtime?
There’s no single standard. But the field has settled into three clear lanes, and each lane has one dominant tool.
llama.cpp is the local standard. It is a C/C++ engine that runs on almost anything, from a Raspberry Pi to a Mac. It reads a file format called GGUF, which packs the weights and the metadata into one file. Ollama, LM Studio, and GPT4All are all friendlier wrappers around llama.cpp. If you download a .gguf file, it will run on one of these platforms.
vLLM is the server standard. It runs on Linux with NVIDIA or AMD GPUs, handles many users at once, and gives you an OpenAI-compatible API. Most commercial providers of open models run vLLM underneath. It reads the raw safetensors files straight from Hugging Face.
MLX is Apple’s framework and is now the fastest path on M-series chips, often 2 to 3 times quicker than the older Metal backend.
So the formats have standardized more than the engines have. Safetensors is the universal storage format on the Hub. GGUF is the universal format for local, quantized use. Almost everything reads one or the other, and converting between them is routine.
One Place To Start: LM Studio
Have a Mac Mini or Macbook Pro? Install LM Studio. It gives you a search box, a download button, and a chat window. Pick Qwen3 14B or Gemma 3 12B and see how it feels. If you like the CLI better, use Ollama instead: ollama run qwen3:14b.
Ollama Quick Start
Step 1: Install Ollama
Ollama is the engine that runs the model on your machine. It is free and open source.
- Go to ollama.com/download.
- Click the button for your system.
- On a Mac, open the downloaded file and drag Ollama into your Applications folder. On Windows, run
OllamaSetup.exe. You do not need an admin password. - Launch it once.
Ollama now runs quietly in the background. Look for a small llama icon in your menu bar (Mac) or system tray (Windows). It stays there.
Step 2: Open a terminal
The terminal is a window where you type commands instead of clicking buttons.
- Mac: Press Command and Space together. Type
Terminal. Press Return. - Windows: Press the Windows key. Type
PowerShell. Press Return.
A plain window opens with a blinking cursor. This is your terminal.
Step 3: Confirm the install worked
Type this and press Return:
ollama --versionYou should see a version number. If you see “command not found”, close the terminal, open a new one, and try again.
Step 4: Download and start a model
Type this and press Return:
ollama run qwen3:8bCode language: CSS (css)Two things happen. First, Ollama downloads the model. This is about 5 GB, so it takes a few minutes on a normal connection. You will see a progress bar.
Then the download finishes and you see a prompt like this:
>>>The model is now loaded into your computer’s memory and waiting.
Step 5: Ask it something
Type a question and press Return. For example:
Write three subject lines for a newsletter about weekend hiking trails.The answer prints a few words at a time. That is the model thinking, one token at a time.
Nothing you type leaves your computer. Turn off your wifi and it still works.
Step 6: Stop
Type /bye and press Return. You are back at the normal terminal prompt.
To use it again later, type ollama run qwen3:8b again. The model is already downloaded, so it starts in seconds.
Which model should you start with?
Model names follow a pattern: name:size. The b means billion parameters. More parameters usually means a smarter model and a slower one.
| Your memory | Try this | Download size |
|---|---|---|
| 8 GB | llama3.2:3b | about 2 GB |
| 16 GB | qwen3:8b | about 5 GB |
| 16 GB | gemma3:12b | about 8 GB |
| 32 GB | qwen3:14b | about 9 GB |
| 48 GB or more | qwen3:30b | about 18 GB |
Start small. If it feels fast, try the next size up. Browse the full list at ollama.com/library.
Commands worth knowing
| Command | What it does |
|---|---|
ollama list | Shows every model you have downloaded |
ollama ps | Shows what is running right now |
ollama rm qwen3:8b | Deletes a model and frees the disk space |
ollama pull gemma3:12b | Downloads a model without starting it |
ollama stop qwen3:8b | Unloads a model from memory |
What to expect
Be realistic. A local model on a laptop is not the same product as a hosted frontier model.
- Speed: Expect 10 to 40 words per second on a modern Mac. Slower on an older PC without a graphics card.
- Quality: A good 8B model in 2026 is roughly as capable as a leading hosted model from two years ago. That is genuinely useful for drafting, summarizing, and rewriting.
- No internet: The model cannot search the web or read a link. It only knows what it learned during training.
- Heat: Your fans will spin up. Your battery will drain faster.
If something goes wrong
It answers one word per second. The model is too big for your machine. Delete it with ollama rm and try a smaller one.
“Error: model requires more system memory”. Same problem. Pick a smaller model.
Your disk is full. Run ollama list to see what you have downloaded, then ollama rm the ones you do not use.
The terminal says “command not found”. Ollama did not install correctly, or you need to open a fresh terminal window. Try the restart first.
Sources:
