GGUF

Related terms

• Quantization
• llama.cpp
• Ollama
• Weights

Posts that use this term

• Troubleshooting local LLMs (and how to keep up after this series)

  The full catalog of local-LLM failures: OOM, slow tok/s, garbage output, instruction drift, bad RAG hits, tool-call hallucination. Plus where to follow the field once you're on your own.

• Fine-tuning a model locally

  When fine-tuning is actually the right call (it usually isn't) and how to pull off a LoRA run on a 16GB Mac, with a worked Llama 3.2 3B example.

• Your first local LLM, start to finish

  Install Ollama, pull Llama 3.2 3B, chat with it, hit its API, and fix the five things that break on a first install. You finish with a working local LLM.

• System requirements by OS for local LLMs

  What macOS, Linux, and Windows each need before you run a local LLM in 2026. Mac is the smoothest, Linux gives you the most knobs, and native Windows finally just works.

• Picking a local model by task

  The 2026 open leaders, sorted by what you actually want to do: coding, chat, the small-model crowd, structured output, vision, embeddings, and audio.

• Quantization, distillation, pruning: how a 140GB model fits on your laptop

  Three ways to shrink an LLM, and why one of them does almost all the work. What Q4_K_M actually means and what each shortcut costs you.

• The local-LLM vocabulary

  Parameters, B, dense vs MoE, base vs instruct, tokens, context windows, chat templates, GGUF, and quant suffixes. Read it once and any HuggingFace model card stops being scary.

• The runtimes: llama.cpp, Ollama, LM Studio

  llama.cpp is the engine. Ollama and LM Studio wrap it. What each one does, when to reach for which, and why the OpenAI-compatible APIs are mostly but not entirely interchangeable.

• Why Apple Silicon punches above its weight on local LLMs

  Unified memory lets the GPU see all of RAM. Here's why that beats a discrete-GPU PC past 32B parameters, what fits in 16/32/64/128/192GB, and where Apple Silicon still loses.

• What it takes to run a model on your own machine

  Why VRAM is the one number that decides whether a local LLM runs, what quantization really does to a model file, and the hardware ladder from an 8GB laptop to a 192GB workstation.

• Where AI actually runs: cloud, local, edge

  When you use AI, a model file is sitting on a real machine. There are only three places it can be, and which one decides almost everything else.

• Install LM Studio

  Install LM Studio on macOS, Linux, and Windows, then flip on the local OpenAI-compatible server so any client library can talk to a model on your own machine.

• Install llama.cpp

  Build llama.cpp from source with Metal or CUDA, then run a GGUF model with llama-cli. The closest thing to bare-metal local inference.