Fine-Tuning a Local LLM: LoRA for Beginners (2026)

What LoRA Is (In One Paragraph)

Full fine-tuning updates every weight — expensive and memory-hungry. LoRA (Low-Rank Adaptation) freezes the original weights and trains small "adapter" matrices that capture the change. Instead of updating a 7B×7B weight matrix, it learns a low-rank factorization (e.g. 7B×r × r×7B with r=16–64) — a tiny fraction of the parameters. At inference, the adapter output is added to the frozen weights. The insight: fine-tuning changes are low-rank — a small set of directions captures nearly all the adaptation, so the small adapters capture almost all the quality.

Step 1: The Dataset (The Quality Lever)

# Format: instruction-style pairs (JSONL)
{"instruction": "Summarize this contract clause.",
 "input": "The parties agree to extend the term by 12 months...",
 "output": "The agreement's term extends by twelve months."}
{"instruction": "What is our refund policy?",
 "input": "",
 "output": "Full refunds within 14 days of purchase..."}

• Quality over quantity: 500 curated examples beat 50,000 noisy ones. Review every example that goes in.
• Match the model's chat template — the trainer formats examples using the model's own template (unsloth handles this automatically).
• Style tasks need fewer examples (100–1,000); skill tasks need more (1,000–10,000).
• Keep a holdout set — the honest way to know the tuning worked. See evaluation for the philosophy.

Step 2: Train with Unsloth (QLoRA, 8–16GB)

pip install unsloth

from unsloth import FastLanguageModel
import torch

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Qwen3-8B",
    max_seq_length=2048, load_in_4bit=True,  # QLoRA
)

model = FastLanguageModel.get_peft_model(
    model, r=16, lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0, use_gradient_checkpointing="unsloth",
)

# Load your JSONL dataset (Hugging Face datasets library)
from datasets import load_dataset
ds = load_dataset("json", data_files="tuning.jsonl")["train"]

# Unsloth's chat-template formatting applies the model's template
trainer = ...  # SFTTrainer with your examples, 3 epochs, lr 2e-4
trainer.train()

# Save the adapter (~200MB) — not a full model
model.save_pretrained("lora-qwen3-legal")
tokenizer.save_pretrained("lora-qwen3-legal")

The r (rank) and lora_alpha are the two knobs that matter: r=16–32 for general adaptation, r=64 for harder tasks; alpha ≈ 2×r is the standard starting point. Everything else is boilerplate.

Step 3: Merge, Convert, Run

# 3a. Merge adapter into the base weights
from unsloth import FastLanguageModel
model, _ = FastLanguageModel.from_pretrained("unsloth/Qwen3-8B")
model.load_adapter("lora-qwen3-legal")
merged = model.merge_and_unload()
merged.save_pretrained("qwen3-8b-legal-merged")

# 3b. Convert to GGUF and quantize (llama.cpp)
python convert_hf_to_gguf.py qwen3-8b-legal-merged \
    --outfile qwen3-8b-legal-f16.gguf
./llama-quantize qwen3-8b-legal-f16.gguf \
    qwen3-8b-legal-q4_k_m.gguf Q4_K_M
# (Full conversion details: the HF-to-GGUF tutorial)

# 3c. Import into Ollama
printf 'FROM ./qwen3-8b-legal-q4_k_m.gguf\n' > Modelfile
ollama create qwen3-legal -f Modelfile
ollama run qwen3-legal

Merging bakes the adaptation into the weights — the output is a normal GGUF file every runtime understands. (llama.cpp can also load the LoRA adapter directly at runtime with --lora, which keeps the adapter separate.) The conversion tutorial and quant guide cover steps 3b in full.

LoRA vs the Alternatives: The Decision

Frequently Asked Questions (FAQ)

What is LoRA?

LoRA (Low-Rank Adaptation) is a fine-tuning technique that freezes the original model weights and trains small low-rank adapter matrices instead — typically 0.1–1% of the parameters. The adapters are added to the frozen weights at inference. Result: fine-tuning a 7B model on a single 8–16GB GPU, with a tiny adapter file instead of a full model copy.

How is LoRA different from full fine-tuning?

Full fine-tuning updates every weight — huge VRAM (a 7B model needs 60GB+ in FP16), long training, and a full model copy per variant. LoRA freezes the model and trains small adapters — ~5–10GB VRAM with QLoRA, hours instead of days, and adapters are a few hundred MB. Quality is close to full fine-tuning for most tasks.

What is QLoRA?

QLoRA is LoRA on a quantized base model: the frozen weights are loaded in 4-bit (NF4), the small adapters train in full precision. This drops memory to ~5–8GB for a 7B model, making fine-tuning possible on consumer GPUs. The 4-bit base is dequantized at inference so quality loss is minimal.

What hardware do I need to fine-tune with LoRA?

A 7–8B model with QLoRA: 8GB VRAM minimum, 12–16GB comfortable, Apple Silicon 16–32GB via MLX. A 3–4B model fits 6–8GB. Datasets of 500–5,000 examples train in 1–4 hours on a 12GB GPU. No cloud, no API — everything local.

How much data do I need for LoRA?

For behavior/style changes: 100–1,000 high-quality examples often suffice. For skills (formatting, tool use, domain writing): 1,000–10,000 examples is the sweet spot. Quality dominates quantity — 500 curated examples beat 50,000 noisy ones. Start small, evaluate, add data only where the model still fails.

Can I run a LoRA-adapted model in Ollama?

Yes — merge the adapter into the base weights (the workflow in this guide), convert to GGUF, and import with a Modelfile. Merging bakes the adaptation into the weights so every runtime sees a normal model file. Alternatively, llama.cpp supports loading LoRA adapters directly at runtime.

Sources & Further Reading

• RAG vs Fine-Tuning: When to Use Which
• How to Convert a Hugging Face Model to GGUF
• Q4_K_M vs Q8_0: Which Quantization Should You Use?
• Build a Mini GPT in Raw PyTorch — the fine-tuning concept in miniature
• Local AI for GDPR Compliance
• Unsloth — fast LoRA/QLoRA training
• LoRA: Low-Rank Adaptation of Large Language Models (arXiv, 2021)