What it takes to run a model on your machine

Running an LLM on your own machine has one bottleneck that decides everything: VRAM. Get that one number right and the rest of the hardware question almost answers itself.

This is post 3 of 7 in the Running series. Last post tour-guided the available models. This one is the hardware they actually run on, and what each piece of silicon does for inference.

What inference actually does

Inference is one thing repeated millions of times: a forward pass through the model. The runtime takes your prompt, converts it to tokens, multiplies those tokens by every relevant weight matrix, and produces probabilities for the next token. Then it does that again for the next token. And the next.

Two questions determine whether your machine can do that:

1. Can the weights even fit somewhere addressable?
2. How fast can the chip multiply matrices?

The first is a memory problem. The second is a compute problem. They are not the same chip.

CPU vs GPU vs NPU

A modern laptop has three things that can do math, and they are wildly different at the math an LLM needs.

• CPU. General-purpose, sequential, eight to sixteen cores. Great at branching logic, terrible at the kind of bulk linear algebra inference is. CPU-only inference works (llama.cpp will do it) but you'll get 2–8 tokens per second on a 7B model. Fine for testing, useless for real work.
• GPU. Thousands of small cores, all doing the same operation across different data. This is exactly what matrix multiplication looks like. An RTX 4090 will do 100+ tok/s on a 7B model. The catch: the model has to fit in the GPU's own memory (VRAM), not the system RAM the CPU sees.
• NPU. A neural processing unit, baked into the SoC on phones, M-series Macs, and recent Snapdragon / Intel laptops. Optimized for power-efficient inference, not raw throughput. Great for small on-device models. Not the chip you target for a 70B local LLM.

For LLMs of any meaningful size, you are running on the GPU. Everything else is a fallback.

VRAM is the hard ceiling

Here is the rule that catches everyone:

If the model file does not fit in VRAM, the GPU cannot run it.

VRAM is the memory soldered to the GPU. On a discrete card it is separate from system RAM and not interchangeable. An RTX 4090 has 24GB. An RTX 5090 has 32GB. An H100 has 80GB.

A model's VRAM requirement at inference is roughly:

• Weights at FP16: parameters × 2 bytes.
• Weights at 4-bit: parameters × 0.5 bytes (plus a small overhead).
• KV cache: scales with context length and batch size.

Worked example. Llama 3 70B at 4-bit quantization:

• Weights: 70B × 0.5 bytes = 35 GB.
• KV cache at 4K context: ~2 GB.
• Overhead: ~1 GB.
• Total: ~38 GB.

A 24GB 4090 cannot run that. A 32GB 5090 cannot run that. You either need an H100, two 4090s with tensor parallelism, or a Mac with 64GB+ unified memory (more on that next post).

A 7B model at 4-bit needs about 4–6 GB. That fits everywhere.

The number that matters: parameters × bytes-per-param

If you remember one formula from this post, remember this:

Pick the row matching the model. Pick the column matching your VRAM ceiling. If the cell value is bigger than your VRAM, that combination won't run.

This table also shows why quantization is the main lever you have.

Quantization, in plain words

Model weights are originally stored at 16-bit (FP16) or 32-bit (FP32) floating point. Quantization compresses each weight into fewer bits — 8-bit, 4-bit, even 3-bit or 2-bit. Same shape, lower precision per number.

The trade-off:

• Q8 (8-bit). Almost indistinguishable from FP16. If you can fit Q8, take it.
• Q5 / Q4 (4–5 bit). The sweet spot for most local use. Quality loss is small and usually invisible in chat. This is what Ollama and LM Studio default to.
• Q3 / Q2. You'll feel it. The model gets fuzzier, especially on reasoning and code. Use only when you must.

The format names you'll see in filenames (Q4_K_M, Q5_K_S, etc.) come from llama.cpp's GGUF format. The K-quants are smarter than the legacy ones — Q4_K_M is the standard pick for "I want 4-bit and I want it good".

A 70B Q4_K_M model is around 40GB on disk and produces output that is genuinely 95% of what the FP16 version would. That is a remarkable engineering achievement, and it's the only reason local LLMs are interesting to most people.

RAM still matters (a bit)

System RAM is not the GPU's VRAM, but it does two jobs that affect inference:

• Loading. The model file streams from disk through system RAM into VRAM on startup. You want enough RAM to not bottleneck that.
• CPU offload. If the model doesn't fully fit in VRAM, llama.cpp can keep some layers on the CPU. This is slow (you're back to 2–10 tok/s on the offloaded layers) but it's the difference between running and not.

For a discrete-GPU PC, 32GB system RAM is the practical floor; 64GB is comfortable. For Apple Silicon, the unified memory rules are different. That's the next post.

Disk and bandwidth

A 70B Q4 model is 40GB. If you're collecting a few of them, you'll burn through SSD space fast. Plan for 200–500 GB if you're going to play around with the open-weights ecosystem.

NVMe matters for first-load speed. The model maps from disk to memory once; after that, disk doesn't appear in the hot path.

A practical hardware ladder

Working budgets, mid-2026:

• Laptop with 8–16 GB unified memory or VRAM. 7B Q4 models. Drafts, autocomplete, light chat. Fine.
• 24 GB GPU (4090 / 5080). Up to 32B Q4 dense, or 70B Q2 with quality loss. Real work happens here.
• 32 GB GPU (5090) or 32–64 GB Mac. 32B Q8 or 70B Q4. The first tier where you're not constantly compromising.
• 64–128 GB Mac. 70B Q8 or 200B-class MoE at low quants. The serious solo-developer rig.
• Two 4090s, a single H100, or a Mac Studio M3 Ultra 192GB. Frontier territory. 405B-class dense or large MoE.

Beyond that you are buying servers, not workstations.

What this changes about model choice

Once you know your VRAM, you stop arguing about which model is "best" and start asking which model is best at the size you can run.

A 32B Qwen on a 4090 will out-reason a 7B Llama every time. A 70B Llama on a 64GB Mac will out-reason a 32B Qwen on a 4090. The chip decides what's available, the model decides what to do with it.

What's next

This post pretended discrete GPUs and unified-memory Macs follow the same rules. They mostly do, but Apple Silicon has one architectural choice that bends the VRAM ceiling enough to deserve its own post. That's next.