Fine-tuning a model locally

Fine-tuning sounds like the answer when prompting and RAG aren't getting you there. It usually isn't. But when it is the right answer, you can do it on a 16GB Mac with techniques that didn't exist in 2023.

This is post 12 of 13 in the Local LLMs series. After this you'll know what LoRA does, when to actually fine-tune, and how to do it locally without a datacenter.

When to fine-tune (and when not to)

The honest decision tree:

1. Is the model failing because it doesn't know your domain words? Try RAG first. Fine-tuning won't add facts as well as RAG does.
2. Is the model failing because it can't follow your output format? Try a stronger prompt with examples first. If that fails, structured generation (grammar-constrained output). Fine-tune if those don't work.
3. Is the model producing the wrong style or tone? This is where fine-tuning genuinely shines. RAG and prompting can't reliably teach style.
4. Do you have 1,000+ high-quality examples of the behavior you want? If yes, fine-tuning is on the table. If you only have 50, prompt with examples instead.
5. Can you live with the trained model being a snapshot in time? Fine-tuned models don't get newer when the base improves. You'll redo it.

In 2026, fine-tuning is the right answer maybe 10% of the time someone is tempted by it. The other 90% is better prompting, RAG, or a different base model.

That said , for the 10% where it fits, the wins are real. Style transfer, narrow-domain code generation, and specialized chat personas (medical, legal, customer support) are all places where a fine-tuned 7B can outperform a generic 70B.

Full fine-tuning vs LoRA

Full fine-tuning updates every weight in the model. For a 7B model, that means computing gradients for all 7B parameters, holding optimizer state for all of them, and writing a new 14GB+ file at the end. Memory needed: 4–8x the model size in VRAM. A 7B full fine-tune wants 60+ GB VRAM. Out of reach for consumer hardware.

LoRA (Low-Rank Adaptation) is the trick. Instead of updating the original weights, you add small "adapter" matrices to certain layers and train only those. The adapters are tiny (typically 1–5% of the original parameter count). At inference, the adapter outputs are added to the original layer outputs.

Result: a 7B LoRA fine-tune trains 50–200M new parameters instead of 7B. Memory drops 10–20x. You can fit it on consumer hardware.

QLoRA goes further: load the base model in 4-bit quantization, train LoRA adapters on top. Memory drops another 4x. A 7B QLoRA fine-tune fits in 8GB VRAM. A 14B fits in 12GB.

This is the technique that made local fine-tuning practical for anyone who isn't running a server farm.

What LoRA actually does, briefly

In each transformer layer, you replace W with W + ΔW. ΔW is the change. Instead of storing ΔW directly (which is the same size as W, expensive), LoRA decomposes it as ΔW = A × B, where A and B are much smaller matrices. You train A and B; you never touch W.

The "rank" of A and B is the knob. Rank 8 or 16 is typical for style fine-tunes; rank 64+ for substantial behavior change. Higher rank = more parameters trained = more memory but more flexibility.

The mechanic is well-studied; the practical effect is that you can encode "this model now responds in formal Indian English" or "this model now writes Rust idiomatically" in a 100MB adapter file, on top of a 5GB frozen base model.

The local fine-tuning toolchain

Three tools dominate in 2026:

• MLX-LM (Apple). The Apple Silicon native fine-tuning library. Fast on M-series, simple CLI. Pick this for a Mac.
• Unsloth. NVIDIA-focused, optimized for speed. 2x faster than baseline Hugging Face on the same hardware. Pick this for a CUDA box.
• Hugging Face PEFT + transformers. The standard, works everywhere, slowest. Pick this if portability matters more than speed.

All three produce LoRA adapters compatible with each other and with Ollama (which can load LoRA adapters at inference time).

A worked example: LoRA on Llama 3.2 3B with MLX

The full pipeline on a 16GB MacBook:

# install mlx-lm via pip

pip install mlx-lm

Prepare a dataset. MLX expects JSON lines:

# data.jsonl - one example per line

echo '{"text": "Q: capital of india?\nA: New Delhi."}' > train.jsonl
echo '{"text": "Q: capital of france?\nA: Paris."}' >> train.jsonl
# ... 500-2000 more examples

Run the fine-tune:

# lora fine-tune llama 3.2 3b for 1000 iterations

mlx_lm.lora --model meta-llama/Llama-3.2-3B-Instruct --train --data ./data --iters 1000

This takes 20–60 minutes on an M3 Max for a few hundred examples. It produces a adapters.safetensors file (about 30–100 MB).

Test the fine-tuned model:

# generate with the lora adapter loaded

mlx_lm.generate --model meta-llama/Llama-3.2-3B-Instruct --adapter-path ./adapters --prompt "Q: capital of nepal?\nA:"

Fuse the adapter into the base for deployment:

# merge adapter into base model

mlx_lm.fuse --model meta-llama/Llama-3.2-3B-Instruct --adapter-path ./adapters --save-path ./fused-model

Convert to GGUF for Ollama (one extra step using llama.cpp's converter):

# convert to gguf for ollama

python llama.cpp/convert_hf_to_gguf.py ./fused-model --outfile fused-model.gguf

Now you have a custom GGUF that can be imported into Ollama via a Modelfile and used like any other model.

Dataset quality is the whole game

Fine-tuning on 500 great examples beats fine-tuning on 50,000 mediocre ones. The hard part of any fine-tune is not the training. It's curating data.

Rules I've learned the painful way:

• Diverse beats voluminous. 500 examples covering 50 scenarios outperform 5,000 examples that all look similar.
• Show, don't tell. "Be polite" in a system prompt is fine. "Be polite" examples in fine-tuning data are 10x more effective.
• Include the failure modes you want to fix. If the model is too verbose, your fine-tuning data should show the concise version. Not just good examples, but good examples that contrast with the bad behavior.
• Validate by holdout. Reserve 10% of data as test set. Measure quality on it before and after. Fine-tunes that look good on training data and fail on holdout are overfitting.
• Beware of style drift. Aggressive fine-tuning can make a model dumber at general tasks while making it better at the specific style. Keep some general-purpose examples in the training data to prevent this.

Hyperparameters that matter

Of the dozen knobs LoRA exposes, three matter for outcomes:

• Rank. 8 or 16 for light style tweaks. 32 or 64 for substantial behavior change. Going beyond 128 rarely helps.
• Learning rate. 1e-4 to 5e-4 for LoRA on a Llama base. Too high = catastrophic forgetting; too low = no learning.
• Number of iterations / epochs. Watch validation loss. Stop when it stops dropping. Continuing past the optimum overfits.

The other knobs (alpha, dropout, target modules, etc.) have sensible defaults. Don't tune them on your first fine-tune.

What runs on what hardware (fine-tuning edition)

Different rules from inference. Training is hungrier.

• 16GB Mac (Air or Pro). LoRA on 3B models. 1B models comfortably. Allow 30–60 min per 1000 iterations on a 1k-example dataset.
• 24–32GB Mac. LoRA on 7B–8B models. Allow 1–2 hours per 1000 iters.
• 64GB Mac. LoRA on 14B comfortably; QLoRA on 70B is feasible but slow (4+ hours per 1000 iters).
• 8GB VRAM PC. QLoRA on 7B with Unsloth.
• 12–16GB VRAM PC. QLoRA on 14B comfortably; LoRA on 7B without quantization.
• 24GB VRAM PC (4090). LoRA on 13B without quant; QLoRA on 32B+. The sweet spot for serious local fine-tuning.

You don't need a server. The MacBook Air on the lower tier is enough to run real LoRA fine-tunes; you'll just wait longer.

After fine-tuning: deploying

Once you have a fine-tuned model:

• For local use: convert to GGUF, import into Ollama via a Modelfile. The Modelfile lets you set the chat template and any custom system prompt.
• For sharing: push the LoRA adapter (just the small file) to Hugging Face. Other people can apply it on top of the base.
• For production: serve via vLLM or llama.cpp's server. Both support loading LoRA adapters dynamically.

Don't ship the fused model unless you have to , fused models are 5GB+ files. The bare adapter is 100MB. Distribute the adapter, apply at load time.

What's next

You now have the full local-LLM stack: vocabulary, compression, runtime mechanics, model selection, hardware, install, integration, RAG, agents, fine-tuning. The last post is the safety net: what to do when any of this breaks, and how to keep up as the field changes.