Virtualizing GPU for OpenPose Application using GVirtuS
Deep learning applications like OpenPose are heavily GPU-dependent due to the massive parallel computations they require. But what if you want to run such applications on lightweight edge devices or non-GPU machines? This is where GPU Virtualization comes into play.
In this article, we’ll explore what GPU virtualization is, why it is needed, how the GVirtuS framework makes it possible, and finally, how you can run OpenPose virtually using GVirtuS in real-time.
What is GPU Virtualization and Why is it Needed?
Modern AI applications demand immense computational power. CPUs, though powerful for general-purpose tasks, fall short when it comes to handling the highly parallel workloads of neural networks, image processing, and deep learning pipelines. GPUs, with their parallel architecture, are the go-to hardware for such workloads.
However, not every device has a dedicated GPU. Edge devices, laptops, and small servers often lack the GPU horsepower needed to run GPU-rich applications. Buying multiple GPU machines for distributed environments is also costly.
GPU Virtualization bridges this gap. It allows a non-GPU device to “borrow” GPU resources from a remote GPU-enabled server. By virtualizing the GPU, we make high-performance computing available to lightweight devices — unlocking flexibility, cost efficiency, and scalability in AI deployments.
2. Introduction to GVirtuS Framework
GVirtuS https://github.com/ecn-aau/GVirtuS, is an open-source GPU virtualization software that enables applications built with CUDA Toolkit v12.6+ to efficiently leverage GPU acceleration remotely.
It provides partial or full virtualization for key CUDA libraries, including:
- cuDNN
- cuFFT
- cuBLAS
- cuSPARSE
- cuSOLVER
GVirtuS follows a split-driver model, meaning the workload is separated between a non-GPU frontend and a GPU backend.
The framework has three main components:
- Frontend — Captures CUDA API calls made by the application running on a non-GPU device.
- Backend — Executes those CUDA calls on the physical GPU hardware.
- Communicator — Transfers data and instructions between frontend and backend (via TCP/IP, RDMA, etc.).
This architecture ensures seamless execution of GPU-accelerated workloads, even from devices that lack a GPU themselves.
3. OpenPose Application and Why It’s GPU-Rich
OpenPose is one of the most popular human pose estimation frameworks. It detects body, hand, and facial keypoints from images and videos in real-time.
Under the hood, OpenPose uses the Caffe deep learning library, which is highly optimized for GPU execution. Since OpenPose involves:
- Processing high-resolution images,
- Extracting features through deep convolutional neural networks (CNNs), and
- Running inference for multiple keypoints per frame,
…it becomes an extremely GPU-intensive application. Running OpenPose on a CPU-only machine leads to slow performance, making real-time pose estimation nearly impossible.
This is where GVirtuS becomes invaluable.
For more detailed setup instructions, check my previous guide:
https://medium.com/@darshankt/openpose-installation-guide-a-step-by-step-walkthrough-d1092b4c5fb9
4. How GVirtuS Helps Virtualize OpenPose
Get Darshan K T’s stories in your inbox
Join Medium for free to get updates from this writer.
As explained earlier, GVirtuS virtualizes CUDA calls by splitting them between frontend and backend. Here’s how it works for OpenPose:
- The GVirtuS Frontend hosts the OpenPose application on a non-GPU device.
- When OpenPose makes CUDA calls (e.g., for matrix multiplications or convolution operations), the frontend intercepts them.
- The Communicator (such as TCP/IP) transparently forwards these calls to the backend.
- The GVirtuS Backend, running on a GPU-enabled machine, executes the CUDA calls directly on the GPU hardware.
- The results are then sent back to the frontend through the communicator.
From the user’s perspective, OpenPose runs smoothly on the non-GPU device, producing real-time keypoint detection results, even though the heavy GPU computation actually happens on a remote server.
This makes GVirtuS a powerful solution for running GPU-hungry applications like OpenPose in edge and distributed environments.
5. Running OpenPose Virtualized with GVirtuS
Let’s walk through the real-time execution of OpenPose with GVirtuS.
Step 1: Download GVirtuS
Clone the official repository:
git clone https://github.com/ecn-aau/GVirtuS.git
cd GVirtuSStep 2: Start the Backend
On the GPU-enabled device, run the backend component:
cd GVirtuS
make run-gvirtus-backend-devThis launches the GVirtuS backend process, ready to serve CUDA calls from remote devices.
Step 3: Run OpenPose on the Frontend
On the non-GPU device, run the integrated OpenPose test:
cd GVirtuS
make run-openpose-testThis command triggers:
- A Docker container,
- The OpenPose-GVirtuS integrated script (
00_test.cpp), - Execution of OpenPose with GPU virtualization.
In the end, you’ll see OpenPose keypoint detection results running on your non-GPU device, while the GPU-intensive work is handled transparently by the remote GPU backend.
Conclusion
GPU Virtualization is transforming how we deploy deep learning applications, making GPU acceleration accessible from almost any device. By using GVirtuS, we can unlock GPU resources across distributed environments and enable GPU-intensive frameworks like OpenPose to run even on lightweight edge devices.
This not only reduces hardware costs but also brings flexibility and scalability to AI workloads.
Acknowledgment
This work has been funded by the Clever Project
Special thanks to Associate Professor Sokol Kosta for his guidance and support, and to my colleagues Wenrui Yu, Tinghui Cheng, Xiaoyu, and Theodoros Aslanidis for their valuable contributions.
