Quantization, distillation, pruning: making models fit
Three ways to shrink an LLM. Quantization (Q2-Q8 with K-quants in GGUF), distillation (teacher to student), pruning. Why Q4_K_M is the community default and what each lever costs.
Blog
#gguf
Posts on AI engineering, LLM systems, and software development.
The local-LLM vocabulary
Parameters, B, dense vs MoE, base vs instruct, tokens, context window, chat template, GGUF, quantization suffixes. After this post you can read any HuggingFace model card.