LoRA Fine-Tuning Local Guide: Train Custom AI Models

LoRA (Low-Rank Adaptation) is a fine-tuning technique that trains only 0.1-1% of a model's parameters using small adapter matrices, so the optimizer state and gradients shrink with them. With QLoRA, you can fine-tune a 7B model on an 8GB card using Unsloth. The LoRA paper (Hu et al., 2021) reports quality on par with full fine-tuning on the models it tested, the base model stays frozen, and the resulting adapter deploys directly to Ollama as a custom model.

What is LoRA?

LoRA (Low-Rank Adaptation) fine-tunes large models efficiently by:

• Training small adapter matrices (~0.1-1% of parameters)
• Keeping base model frozen
• Matching full fine-tuning on the tasks in the LoRA paper (Hu et al., 2021)
• Using about 3x less GPU memory than full fine-tuning (the LoRA paper's GPT-3 175B figure)

Full Fine-tuning:  Train all 7B parameters → 28GB+ VRAM
LoRA Fine-tuning:  Train ~50M parameters → 8GB VRAM

VRAM Requirements

QLoRA makes consumer GPU training practical.

The same numbers hold for vision-language bases, which cost extra VRAM for the image encoder — PaliGemma 3B is the usual starting point for fine-tuning a vision-language model because it was released specifically to be adapted to narrow tasks.

Setting Up Unsloth (Fastest Method)

Installation

pip install unsloth
pip install --upgrade transformers datasets

Basic Training Script

from unsloth import FastLanguageModel
from datasets import load_dataset
from trl import SFTTrainer
from transformers import TrainingArguments

# Load model with 4-bit quantization
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/llama-3.1-8b-bnb-4bit",
    max_seq_length=2048,
    load_in_4bit=True,
)

# Add LoRA adapters
model = FastLanguageModel.get_peft_model(
    model,
    r=16,  # LoRA rank
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                   "gate_proj", "up_proj", "down_proj"],
    lora_alpha=16,
    lora_dropout=0,
    bias="none",
    use_gradient_checkpointing="unsloth",
)

# Load dataset
dataset = load_dataset("json", data_files="training_data.jsonl")

# Training arguments
training_args = TrainingArguments(
    output_dir="./lora_output",
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,
    warmup_steps=10,
    max_steps=100,
    learning_rate=2e-4,
    fp16=True,
    logging_steps=10,
    save_steps=50,
)

# Train
trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset["train"],
    args=training_args,
    max_seq_length=2048,
)

trainer.train()

# Save LoRA
model.save_pretrained("./my_lora")

Preparing Your Dataset

Dataset Format (JSONL)

{"instruction": "Write a poem about AI", "input": "", "output": "Silicon dreams..."}
{"instruction": "Explain quantum computing", "input": "", "output": "Quantum computing uses..."}
{"instruction": "Translate to French", "input": "Hello world", "output": "Bonjour le monde"}

Conversation Format

{"conversations": [
  {"role": "user", "content": "What is AI?"},
  {"role": "assistant", "content": "AI is..."}
]}

Dataset Tips

1. Quality > Quantity: 500 excellent examples > 5000 mediocre
2. Diversity: Cover all variations of your use case
3. Format consistency: Same structure throughout
4. Length variety: Mix short and long responses

LoRA Hyperparameters

Merging and Deploying

Merge LoRA with Base Model

# Merge LoRA adapters into base model
merged_model = model.merge_and_unload()

# Save merged model
merged_model.save_pretrained("./merged_model")
tokenizer.save_pretrained("./merged_model")

Convert to GGUF for Ollama

# Install llama.cpp
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp && make

# Convert to GGUF
python convert.py ../merged_model --outtype f16 --outfile model.gguf

# Quantize
./quantize model.gguf model-q4.gguf q4_k_m

Create Ollama Model

# Create Modelfile
cat > Modelfile << 'EOF'
FROM ./model-q4.gguf
SYSTEM "You are a helpful assistant fine-tuned for specific tasks."
EOF

# Register with Ollama
ollama create my-custom-model -f Modelfile

# Run your model
ollama run my-custom-model

Training Time Estimates

We have not timed these runs, so estimate yours instead: total steps = examples × epochs ÷ (batch size × gradient accumulation). Run 10 steps, read the seconds per step from the trainer log, and multiply. Time per step grows with model size and sequence length. Unsloth's documentation reports about 2x faster training than standard HuggingFace code.

Common Use Cases

1. Custom Writing Style

Train on your existing content to match your voice.

2. Domain Expert

Train on medical, legal, or technical documents.

3. Company Knowledge

Train on internal docs for support chatbot.

4. Language Adaptation

Improve performance in specific languages.

5. Task Specialist

Optimize for specific tasks (summarization, extraction).

Troubleshooting

Out of Memory

• Reduce batch size to 1
• Lower LoRA rank (r=8)
• Use gradient checkpointing
• Reduce max_seq_length

Poor Results

• More/better training data
• Lower learning rate
• Train longer (more steps)
• Check data formatting

Overfitting

• Fewer epochs
• Add dropout (0.05-0.1)
• More diverse data

Key Takeaways

1. LoRA trains about 1% of parameters and, in the LoRA paper, matches full fine-tuning
2. QLoRA enables training on consumer GPUs
3. Unsloth is the fastest training framework
4. Quality data matters more than quantity
5. Deploy to Ollama for easy local use
6. 8GB VRAM can train 7B models with QLoRA

Next Steps

1. Set up Ollama to run your model
2. Compare free local models to choose a base
3. Build RAG with your fine-tuned model

---

LoRA democratizes AI customization—you can now create specialized AI models on consumer hardware that rival expensive cloud fine-tuning.