The local-LLM vocabulary

A HuggingFace model card is intimidating until you know the words. Once you do, every card reads the same way: a few labels, a few numbers, and the rest is marketing. This post covers the labels and numbers.

This is post 2 of 13 in the Local LLMs series. By the end, "Llama-3.1-8B-Instruct-Q4_K_M.gguf" will be a sentence you can read.

Parameters and "B"

A parameter is one number stored inside the model. A "weight," in the same family. A modern LLM has billions of them, hence "8B" (8 billion) or "70B" (70 billion). The number tells you two practical things:

• How much disk and memory it needs. At 16-bit precision, parameters × 2 bytes. So 8B at FP16 is roughly 16GB on disk. At 4-bit quantization, parameters × 0.5 bytes , about 4GB.
• How smart it is, very roughly. A 70B model has about 9x more parameters than an 8B, so more capacity to encode patterns. The ratio is not linear with intelligence (a well-trained 8B beats a poorly-trained 70B), but within the same model family, bigger is sharper.

When you see "Llama 3.2 3B," the 3B is the parameter count. That's the headline number. Everything else is detail.

Dense vs Mixture-of-Experts

The newer split. Models come in two architectures:

• Dense. Every parameter fires on every token. Llama 3.x 8B is dense , all 8B parameters are computed for every token you generate. Simple, predictable, fits any runtime.
• Mixture-of-Experts (MoE). The model has many "experts," each a sub-network. A router picks a small subset for each token. Total parameters is huge, but only a fraction (the "active" parameters) computes per token.

You'll see two numbers on MoE models. DeepSeek V4: 1.6T total / 49B active. Llama 4 Maverick: 400B / 17B. Qwen 3.5 235B-A22B: 235B total, 22B active. The first number is the file size; the second is the cost per token.

For local use, dense is usually what you want. MoE files are huge to download and need lots of RAM/VRAM to load, even though the inference itself is cheap. A 14B dense model fits easily on a 16GB GPU; a 200B MoE doesn't, regardless of how few active parameters it has.

Base vs instruct

Every released model exists in two versions:

• Base. Trained only on next-token prediction. It completes text. Ask it "What is 2+2?" and it might respond with another question, or with the next sentence in a textbook. Base models are for research and fine-tuning, not chat.
• Instruct (or "Chat", "IT"). The base model further trained to follow instructions and have conversations. This is what you want 99% of the time.

Filenames usually include the suffix: Llama-3.1-8B-Instruct, Qwen2.5-3B-Instruct. If a model name has no suffix, check the card. Using a base model when you wanted instruct is the most common "why is this thing not following my prompt" mistake new users hit.

Tokens and the tokenizer

A token is the unit a model reads and writes. Not a character, not a word , a chunk in between. For English:

• 1 token ≈ 4 characters
• 1 token ≈ 0.75 of a word
• 1,000 tokens ≈ 750 words ≈ a page
• 1M tokens ≈ a 750-page book

The mapping is the model's tokenizer. Different models use different tokenizers (BPE, SentencePiece, tiktoken). Same input text, different token counts. Hindi or Devanagari scripts often tokenize 2–3x worse than English on most tokenizers, so the same Hindi paragraph costs more tokens than the same English paragraph.

The number that matters when you compare bills, context lengths, or output speeds is always tokens, never characters or words.

Context window

The maximum number of tokens the model can see at once. The model's "working memory" for a single conversation.

Common sizes in 2026:

• 4K (4,096 tokens). Old. Fits a few messages.
• 8K. Llama 3 base. Workable for short chat.
• 32K. Comfortable for medium documents.
• 128K. The 2026 default for serious open models.
• 1M. Llama 4 Scout, Gemini 3.1 Pro. Whole codebases.
• 10M. Llama 4 Maverick experimental. A small library.

Bigger context costs memory (more on this when we hit the KV cache in post 4). On local hardware, you usually pick a context that fits your VRAM, not the model's maximum.

Chat template

Each instruct model is trained to expect a specific format around messages. Llama uses one wrapper (<|begin_of_text|><|start_header_id|>user<|end_header_id|>...), Mistral uses another ([INST]...[/INST]), Qwen uses a third (<|im_start|>user...<|im_end|>).

You don't usually write these by hand , Ollama, LM Studio, and llama.cpp's server endpoint handle the template for you. But "wrong chat template" is the second most common "why is this not working" , usually because someone is using a base model with an instruct template, or vice versa.

Quantization (preview)

The shorthand at the end of every GGUF filename: Q4_K_M, Q5_K_S, Q8_0. The "Q" plus a number is "quantization level": the number of bits per parameter.

• Q8: 8 bits per parameter. Almost no quality loss vs full precision.
• Q5/Q4: 5 or 4 bits. The sweet spot for local. Tiny quality drop.
• Q3/Q2: 3 or 2 bits. Noticeably degraded. Use only when you must.

The next post is entirely about quantization (and its cousins, distillation and pruning). For now, treat the suffix as "how aggressively the file was compressed," and pick Q4_K_M as a safe default.

GGUF

The file format that holds the model. One file with weights, tokenizer, metadata, chat template, everything. Self-contained. You can email a model to a friend (assuming gigabit upload).

GGUF is a llama.cpp invention. Almost every local-LLM tool reads it. When you ollama pull a model, the file behind the scenes is a GGUF.

Reading a model name

Now the full sentence: Llama-3.1-8B-Instruct-Q4_K_M.gguf

• Llama , model family (Meta).
• 3.1 , version.
• 8B , 8 billion parameters, dense.
• Instruct , instruction-tuned, chat-ready.
• Q4_K_M , 4-bit quantized, K-quants, M (medium) variant. A good default.
• .gguf , the file format.

Translation: "Meta's Llama 3.1, the 8-billion-parameter dense version, fine-tuned for chat, compressed to 4 bits using the medium-quality K-quant scheme, in the standard local-LLM file format."

That sentence is the entire metadata. Once you can read it, you can shop on HuggingFace.

A few more terms you'll see

• Context length / n_ctx. The runtime parameter that controls how big a context window you actually allocate. Often smaller than the model's maximum to save memory.
• Temperature. A knob from 0 to 2. Controls randomness. 0 = deterministic, 1 = default, 2 = chaos. Default 0.7 is fine for most chat.
• System prompt. Instructions you send before the conversation that shape how the model behaves. "You are a helpful coding assistant who answers in Python."
• Stop tokens. Tokens that, when generated, halt output. The chat template defines these. Misconfiguration is why a model sometimes runs past its turn into a fake user message.
• Embedding. A different kind of model , turns text into a vector of numbers for similarity search. We hit these in post 10 (RAG).

What's next

Now that the words are sorted, the next post zooms into the part most local-LLM users hand-wave: how a 70B model file gets small enough to fit on a 24GB GPU. Three techniques (quantization, distillation, pruning), three trade-offs.