Loading models on GPU

Open in colab

Different methods of loading models on GPU is discussed using different libraries and frameworks.

1.Loading Hugging Face Models for Causal Language Modeling using Transformers

Hugging Face provides a seamless interface to load and use state-of-the-art NLP models. If you’re looking to load a model for tasks like text generation, dialogue modeling, or story completion, the AutoModelForCausalLM class is your go-to solution.

In this section, we walk you through how to load a pre-trained model from Hugging Face’s Model Hub and configure it to run on either GPU or CPU, depending on your hardware setup.

Step 1: Install Required Libraries

To begin, install or upgrade the transformers library from Hugging Face. You can install the latest release or directly from GitHub:

!pip install - upgrade transformers
!pip install git+https://github.com/huggingface/transformers.git

Loading a Pretrained Model

Here’s how to load a causal language model

from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
 "boltuix/bert-trip-plan", 
 device_map="cuda",        #Use "cuda" for GPU, "cpu" for CPU, or "auto" for automatic selection
 torch_dtype="auto",       #Automatically selects the optimal data type
 trust_remote_code=True,   #Allows loading custom model code from the repo
 use_auth_token=True,      #Needed if the model is private or gated
 low_cpu_mem_usage=False   #Set to True to save memory (optional)
)

AutoModelForCausalLM is a class from the Hugging Face Transformers library. It is a generic wrapper that automatically selects the correct model architecture (e.g., GPT, LLaMA, Falcon) for causal language modeling tasks, which predict the next word(s) in a sequence.

It works like this:

• You specify a model name from the Hugging Face Hub (like "gpt2" or "meta-llama/Llama-2-7b-chat-hf").
• AutoModelForCausalLM detects the architecture and loads the corresponding model.
• It is task-specific — only suitable for causal generation tasks.

2.Efficiently Loading Large Language Models on GPU Using bitsandbytes (8-bit Quantization)

As large language models continue to grow in size and capability, running them on consumer-grade hardware becomes increasingly challenging due to memory constraints. Fortunately, tools like bitsandbytes offer a solution to this problem through model quantization — reducing the memory footprint while preserving performance.

In this method, we show how to use 8-bit quantization to load a large causal language model like "llm-agents/tora-7b-v1.0" on your GPU using minimal VRAM.

Step-by-Step Implementation

First, install the bitsandbytes library:

!pip install -U bitsandbytes

Now, load the model and tokenizer with 8-bit quantization enabled:

from transformers import BitsAndBytesConfig, AutoTokenizer, AutoModelForCausalLM
# Define quantization config
quantization_config = BitsAndBytesConfig(load_in_8bit=True)
# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("llm-agents/tora-7b-v1.0")
# Load model with quantization on GPU
model = AutoModelForCausalLM.from_pretrained(
 "llm-agents/tora-7b-v1.0", 
 device_map="cuda", 
 quantization_config=quantization_config
)
# Run inference
prompt = "Hello, my llama is cute"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
generated_ids = model.generate(**inputs)
outputs = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)

bitsandbytes is a lightweight CUDA-based library developed by Tim Dettmers that enables 8-bit and 4-bit matrix multiplication. It allows you to load transformer models at significantly reduced memory usage, making it possible to run 7B, 13B, or even larger models on GPUs with limited VRAM (like 8–16 GB cards). This is achieved through quantization, which compresses the model weights from 16-bit/32-bit floats to smaller formats without significant loss in accuracy.(However this Library needs GPU to run)

Get Manjunath Patil’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Quantization is a model compression technique where the model weights are stored using lower-precision data types (like 8-bit integers instead of 32-bit floats). This leads to:

Faster inference speeds

Reduced memory usage

Lower power consumption

In this method, we use BitsAndBytesConfig(load_in_8bit=True) to specify that the model should be loaded in 8-bit precision. This configuration tells Hugging Face to use bitsandbytes under the hood to optimize the model for GPU inference with much lower memory usage.

3.Running GGUF Models Efficiently on GPU Using llama-cpp and cuBLAS

As open-source LLMs like Mistral, LLaMA, and OpenHermes continue to evolve, a lightweight and highly efficient solution to run them locally is through the llama-cpp library. This method introduces a new file format called GGUF, optimized for both CPU and GPU execution with minimal resource usage.

llama-cpp is a C++ -based inference engine for LLaMA and similar models. It's designed to be extremely fast and memory-efficient, and it can run directly on CPUs or GPUs.

Advantages of GGUF models

Smaller file size (heavily quantized models)

Faster loading

Easy compatibility across platforms (Windows, Linux, Mac)

Supports metadata and tokenizers directly within the file

cuBLAS (CUDA Basic Linear Algebra Subprogramming) is NVIDIA’s GPU-accelerated library for performing dense linear algebra operations — like matrix multiplication — which are core to running transformer models efficiently.

In this context:

• llama-cpp leverages cuBLAS to offload heavy matrix operations to the GPU
• This allows even quantized models like openhermes-2.5-mistral-7b.Q4_K_M.gguf to run smoothly on consumer GPUs (e.g., RTX 3060/4060)

Without cuBLAS, llama-cpp would default to CPU-based operations, which are significantly slower.

Step-by-Step Setup

# Install llama-index for LLM integration
!pip install -q llama-index
!pip install llama-index-llms-llama-cpp

# Uninstall existing llama-cpp and install GPU-enhanced version with cuBLAS
!pip uninstall -y llama-cpp-python torch
!pip install torch - pre - extra-index-url https://download.pytorch.org/whl/nightly/cu121
!pip install llama-cpp-python - force-reinstall - upgrade - no-cache-dir \
 - prefer-binary - extra-index-url=https://jllllll.github.io/llama-cpp-python-cuBLAS-wheels/AVX2/cu121

Now test the model

from llama_cpp import Llama

llm_verify = Llama(
 model_path="/content/drive/MyDrive/LLM models/openhermes-2.5-mistral-7b.Q4_K_M.gguf",
 n_gpu_layers=33, # 33 layers offloaded to GPU (recommended for 7B models)
 n_ctx=4096,
 seed=42,
 verbose=True
)

Using llama-cpp with quantized GGUF models offers several compelling advantages. First, GGUF models are highly optimized, enabling 7B+ parameter models to run efficiently even on lower-end GPUs with just 4GB–8GB of VRAM. The integration of cuBLAS allows for fast matrix computations directly on the GPU, significantly improving inference speed. This setup is also lightweight, eliminating the need for PyTorch after the initial configuration, which reduces system overhead. Additionally, it offers flexibility by supporting both CPU and GPU execution, making it suitable for a variety of hardware environments. Finally, it provides a simple Python API, allowing developers to prompt models and receive responses similarly to how they would with Hugging Face’s transformers.

Final Thoughts

This method is ideal for anyone who:

• Wants to maximize performance on a local GPU without massive VRAM.
• Is working with quantized GGUF models (smaller & faster).
• Prefers a lightweight alternative to traditional PyTorch-based LLMs.

llama-cpp with cuBLAS gives you an efficient, low-latency way to load and run LLMs like Mistral and OpenHermes with blazing speed — even on laptops or personal GPUs.