Quantization

Related terms

• GGUF
• Weights
• Model
• Inference

Posts that use this term

• Troubleshooting local LLMs (and how to keep up after this series)

  The full catalog of local-LLM failures: OOM, slow tok/s, garbage output, instruction drift, bad RAG hits, tool-call hallucination. Plus where to follow the field once you're on your own.

• Fine-tuning a model locally

  When fine-tuning is actually the right call (it usually isn't) and how to pull off a LoRA run on a 16GB Mac, with a worked Llama 3.2 3B example.

• Your first local LLM, start to finish

  Install Ollama, pull Llama 3.2 3B, chat with it, hit its API, and fix the five things that break on a first install. You finish with a working local LLM.

• Every machine can run a local LLM (here's what fits)

  A per-tier guide to running local LLMs in 2026, from 8GB integrated graphics to a 192GB Mac Studio. Specific models, specific speeds, specific configs.

• Picking a local model by task

  The 2026 open leaders, sorted by what you actually want to do: coding, chat, the small-model crowd, structured output, vision, embeddings, and audio.

• Streaming, throughput, and the KV cache

  Why TTFT and tok/s are different numbers, why streaming feels faster than it is, and the KV cache that makes the 1000th token cost about the same as the first.

• Quantization, distillation, pruning: how a 140GB model fits on your laptop

  Three ways to shrink an LLM, and why one of them does almost all the work. What Q4_K_M actually means and what each shortcut costs you.

• The local-LLM vocabulary

  Parameters, B, dense vs MoE, base vs instruct, tokens, context windows, chat templates, GGUF, and quant suffixes. Read it once and any HuggingFace model card stops being scary.

• The pitch for local LLMs in 2026

  The case for running an LLM on the machine you already own. Privacy, no per-call cost, faster first token, no rate limits, and it works on a flight.

• The runtimes: llama.cpp, Ollama, LM Studio

  llama.cpp is the engine. Ollama and LM Studio wrap it. What each one does, when to reach for which, and why the OpenAI-compatible APIs are mostly but not entirely interchangeable.

• Why Apple Silicon punches above its weight on local LLMs

  Unified memory lets the GPU see all of RAM. Here's why that beats a discrete-GPU PC past 32B parameters, what fits in 16/32/64/128/192GB, and where Apple Silicon still loses.

• What it takes to run a model on your own machine

  Why VRAM is the one number that decides whether a local LLM runs, what quantization really does to a model file, and the hardware ladder from an 8GB laptop to a 192GB workstation.

• The major LLMs in 2026

  A field guide to the closed frontier models and the open weights you can actually run. What the "B" numbers mean, and which size fits your machine.

• Where AI actually runs: cloud, local, edge

  When you use AI, a model file is sitting on a real machine. There are only three places it can be, and which one decides almost everything else.

• Install LM Studio

  Install LM Studio on macOS, Linux, and Windows, then flip on the local OpenAI-compatible server so any client library can talk to a model on your own machine.

• Install llama.cpp

  Build llama.cpp from source with Metal or CUDA, then run a GGUF model with llama-cli. The closest thing to bare-metal local inference.

• Install Ollama

  Get Ollama running on macOS, Linux, or Windows, pull your first model, and confirm it works with ollama list. The shortest path to a local LLM.