<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Rishipramodch on Medium]]></title>
        <description><![CDATA[Stories by Rishipramodch on Medium]]></description>
        <link>https://medium.com/@rishipramodc?source=rss-4c2dae8da1dc------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*OLcScbFalhbcWi9CkDdxAw.jpeg</url>
            <title>Stories by Rishipramodch on Medium</title>
            <link>https://medium.com/@rishipramodc?source=rss-4c2dae8da1dc------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 17:17:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rishipramodc/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Learning GPU Kernels #2: How Threads Actually Execute on the GPU]]></title>
            <link>https://medium.com/@rishipramodc/learning-gpu-kernels-2-how-threads-actually-execute-on-the-gpu-055e2f4b6699?source=rss-4c2dae8da1dc------2</link>
            <guid isPermaLink="false">https://medium.com/p/055e2f4b6699</guid>
            <category><![CDATA[cpp]]></category>
            <category><![CDATA[gpu-programming]]></category>
            <category><![CDATA[cuda]]></category>
            <category><![CDATA[nvidia]]></category>
            <category><![CDATA[parallel-computing]]></category>
            <dc:creator><![CDATA[Rishipramodch]]></dc:creator>
            <pubDate>Wed, 23 Sep 2026 15:06:18 GMT</pubDate>
            <atom:updated>2026-09-25T17:10:38.737Z</atom:updated>
            <content:encoded><![CDATA[<p>If you missed the first part of this series, you can read it here: <a href="https://medium.com/@rishipramodc/learning-gpu-kernels-1-threads-blocks-and-first-cuda-kernel-d0a64dcd16e5"><strong>Learning GPU Kernels #1: Threads, Blocks, and My First CUDA Kernel</strong></a></p><blockquote>In the previous blog, I learned the basic CUDA execution model, including kernels, threads, blocks and grids. I also implemented my first CUDA kernel using vector addition.</blockquote><blockquote>One question was still unanswered. What actually happens to all these threads after a kernel is launched?</blockquote><blockquote>In this part, I am going one level deeper and looking at how the GPU hardware executes those threads.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/1*58TXQb8v1cDpfC-GnLipFg.png" /></figure><h3>Hardware View</h3><pre>CUDA program<br>↓<br>Kernel launch<br>↓<br>Grid<br>↓<br>Thread blocks<br>↓<br>Blocks scheduled onto SMs<br>↓<br>Threads grouped into warps<br>↓<br>Warp scheduler<br>↓<br>Execution units</pre><p>In the first blog, I mostly focused on the upper part of this flow, where CUDA organizes work using grids, blocks, and threads. Now I want to look at the lower part of this flow to understand what happens when those thread blocks actually reach the GPU hardware.</p><h3>1. Streaming Multiprocessors (SMs)</h3><p>An SM is a parallel execution engine that manages and executes many CUDA threads. It contains resources needed to execute CUDA threads, including things such as:</p><pre>SM<br>│<br>├── Warp schedulers<br>├── Arithmetic execution units<br>├── Load/store units<br>├── Registers<br>├── Shared memory<br>├── L1/cache resources<br>└── Other specialized execution units</pre><p>Each SM contains several types of execution units for arithmetic, memory operations, and specialized computations. The exact number and organization of these units depend on the GPU architecture.</p><blockquote>SMs of a GPU are grouped into GPU Processing Clusters (GPC).</blockquote><p>Each SM contains fast on-chip storage such as registers, shared memory, and cache resources. The GPU also has much larger device memory outside the SMs, usually called global memory.</p><blockquote>I will go deeper into these memory types in the next blog, where I will study the GPU memory hierarchy.</blockquote><h3>2. Thread Block Scheduling</h3><p>Thread block scheduling is the process of assigning thread blocks to available SMs.</p><p>When a kernel is launched, the grid may contain more thread blocks than the GPU can execute at once. These blocks are gradually assigned to available SMs. Before an SM can accept a block, it must have enough hardware resources available for that block.</p><p>Since a GPU has a limited number of SMs, and each SM can keep only a limited number of blocks, a large grid is usually executed in multiple groups over time.</p><p>All threads in a block execute on the same SM. The block is divided into warps, and those warps are scheduled for execution over time. Threads inside the same block can cooperate using shared memory and block-level synchronization. Different blocks should not depend on a particular execution order.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pbZunkqcn0pvU4lk-pzXUg.png" /></figure><p>Before moving further, there is one additional level in newer CUDA architectures called a Thread Block Cluster.</p><h4>Thread Block Clusters</h4><p>On supported NVIDIA GPUs, CUDA provides an optional level of organization called <strong>Thread Block Clusters</strong>. A cluster groups multiple thread blocks together.</p><pre>Threads → Thread Blocks → Thread Block clusters → Grid</pre><p>Thread blocks in the same cluster are scheduled onto SMs within a single GPC. This allows threads in different blocks of the cluster to communicate and synchronize using CUDA’s cluster-level programming features. Threads in different blocks of the same cluster can also access distributed shared memory.</p><h3>3. Block-Level Synchronization</h3><p>CUDA allows threads in the same block to coordinate their activities using the block-wide barrier synchronization function <em>__syncthreads()</em>.</p><p>When a thread reaches __syncthreads(), it waits there until all required threads in the same block have also reached that synchronization point.</p><blockquote>“__” is used for indicating intrinsic functions in CUDA.</blockquote><blockquote><strong>Intrinsic Functions</strong></blockquote><blockquote>CUDA uses names beginning with double underscores (__) for special compiler-defined CUDA features such as __global__, __shared__, and __syncthreads().</blockquote><blockquote>An intrinsic function is a special function recognized directly by the compiler. The compiler can translate it into the corresponding CUDA or hardware operation instead of treating it like a normal library function.</blockquote><h4><strong>Barrier synchronization</strong></h4><p>It is a popular method for coordinating parallel activities.</p><p>eg., Imagine four students working on different parts of the same project. Before they combine their work into the final report, everyone must finish their assigned part. Students who finish early wait for the others. Once everyone is ready, the group moves to the next stage.</p><p>__syncthreads() works similarly. Threads that reach the barrier early wait until the rest of the block arrives.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/1*ps_Q_S2xXbUtlKikCu6GUw.png" /></figure><p>The diagram illustrates the execution of barrier synchronization. There are N threads, some of the threads arrive at the barrier synchronization program location early and some of them much later. The threads that arrive at the barrier early wait for those who arrive late.</p><h4><strong>Incorrect use of __syncthreads()</strong></h4><p>In CUDA, if a __syncthreads() statement is present, it must be executed by all threads in a block. When a __syncthreads() statement is placed in an if-statement, either all threads in a block execute the path that includes the __syncthreads() or none of them does.</p><pre>__global__ void incorrect_barrier_example() {<br>    if (threadIdx.x % 2 == 0) {<br>	__syncthreads();<br>    }<br>    else {<br>	__syncthreads();<br>    }<br>}</pre><p>The code shows an incorrect use of __syncthreads() where two __syncthreads() are used in the if-else statement.</p><p>Threads with an even threadIdx.x enter the first branch, while the remaining threads enter the second branch. A block-wide barrier must be reached consistently by the threads participating in that synchronization.</p><h3>4. WARPS and SIMD hardware</h3><p>Threads inside a block are organized into groups of 32 called <strong>warps</strong>. A warp executes a kernel code in a SIMT paradigm. In SIMT, all the threads in the warp are executing the same kernel code, but each thread follow different branches through the code.</p><h4>Why does a warp have 32 threads?</h4><p>There is no mathematical proof for 32 threads. The choice of 32 is part of NVIDIA’s hardware architecture. It comes from engineering trade-offs involving execution width, scheduling, resource usage, control flow, and memory behavior. For CUDA programmers, the important point is that NVIDIA GPUs use a warp size of 32 threads.</p><blockquote><strong>Note:</strong> AMD GPUs use a similar execution group called a wavefront. Its width depends on the architecture, which shows that execution-group size is a hardware design choice.</blockquote><p>If the block size is not a multiple of the warp size, the final warp will contain some inactive lanes. If a block has 48 threads, it is divided into two warps. The first warp contains 32 active threads, while the second contains 16 active threads and 16 inactive lanes.</p><h4>Von Neumann Model</h4><p>Executing threads and warps illustrated in modified von Neumann model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/382/1*mv4Hyef52euAEaRfWLIWvw.png" /></figure><p>The control signals are sent to multiple processing units, allowing the same instruction to operate on different data.</p><p>This is a simplified conceptual model used only to visualize how an instruction can be applied across multiple processing units.</p><h3>5. Warp Scheduling and Latency Tolerance</h3><p>An SM can have many warps resident at the same time. However, not every warp is always ready to execute its next instruction. When an instruction to be executed by a warp is waiting for the results of a previously initiated long-latency operation, the warp is not selected for the execution. Instead, another warp that is no longer waiting for the results of previous instructions will be selected for execution.</p><h4>Active, Eligible, and Stalled Warps</h4><p>An <strong>active warp</strong> is a warp that belongs to a thread block currently resident on an SM. Being active does not mean that the warp is executing at that exact moment.</p><p>A warp whose next instruction is ready to execute is called an <strong>eligible warp</strong>.</p><p>A <strong>stalled warp</strong> is an active warp that cannot currently execute its next instruction because something it needs is not ready yet.</p><pre>Active Warp<br> ↓<br>Can the next instruction execute?<br><br>Yes → Eligible<br>No → Stalled</pre><h3>6. Latency Hiding</h3><p>Suppose Warp 0 requests data from memory:</p><pre>Warp 0 → Request data from memory → waiting…</pre><p>Instead of leaving the execution units idle while Warp 0 waits, the warp scheduler selects another warp which is ready to execute.</p><p>This does not make the memory operation itself faster. The latency still exists. The GPU performs useful work from other warps while one warp is waiting.</p><h4>Occupancy</h4><p>Occupancy compares the number of active warps on an SM with the maximum number of active warps that the SM can support.</p><pre>Maximum supported warps = 64<br>Active warps = 32<br>Occupancy = 32 / 64 = 50%</pre><p>Higher occupancy can give the scheduler more warps to choose from, which may help hide latency. Higher occupancy does not always mean better performance.</p><h3>7. Warp Divergence</h3><p>Consider this kernel:</p><pre>if (threadIdx.x % 2 == 0) {<br>      data[i] = data[i] * 2;<br>}<br>else {<br>      data[i] = data[i] + 1;<br>}</pre><p>Inside one warp, the threads now make different decisions.</p><pre>Thread 0  → if<br>Thread 1  → else<br>Thread 2  → if<br>Thread 3  → else<br>Thread 4  → if<br>Thread 5  → else<br>...<br>Thread 31 → else</pre><p>Half of the threads want to execute one branch while the remaining threads want to execute another branch. This situation is called <strong>warp divergence</strong>.</p><p>During divergent execution, some lanes remain inactive while another path is being executed, which can reduce warp efficiency.</p><h3>Warp-Aligned Branching</h3><pre>__global__ void warp_aligned_kernel(int *data) {<br>      int i = blockIdx.x * blockDim.x + threadIdx.x;<br>      int warp_id_in_block = threadIdx.x / 32;<br>      if (warp_id_in_block % 2 == 0) {<br>          data[i] = data[i] * 2;<br>      }<br>      else {<br>          data[i] = data[i] + 1;<br>      }<br>}</pre><p>Now the execution pattern becomes:</p><pre>Warp 0 → Branch A<br>Warp 1 → Branch B<br>Warp 2 → Branch A<br>Warp 3 → Branch B</pre><p>Comparing the Two Branching Patterns</p><pre>Divergent Kernel<br><br>Warp 0: ABABABABABABABABABABABABABABABAB<br><br>Warp-Aligned Kernel<br><br>Warp 0: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA<br><br>Warp 1: BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB</pre><p>Both examples contain conditional branches. The difference is that the first branch changes between neighboring threads inside the same warp, while the second branch changes only between warps.</p><h3>What I Learned</h3><blockquote>In the first blog, I learned how CUDA organizes work using threads, blocks, and grids.</blockquote><blockquote>In this part, I went below that programming model and looked at how the GPU actually executes those threads.</blockquote><blockquote>Thread blocks are assigned to SMs, and the threads inside those blocks are organized into 32-thread warps. Warp schedulers choose ready warps for execution, and when one warp stalls, another warp can run in its place. This allows the GPU to hide latency and keep more of its hardware busy.</blockquote><h3>What’s Next?</h3><p>At this point, I understand how CUDA threads move from blocks to SMs and how they are executed as warps.</p><p>But one important question is still left.</p><p>When a warp stalls while waiting for data, where is that data actually stored, and why are some memory accesses much faster than others?</p><p>That leads to the next part of this series, where I will study the <strong>GPU memory hierarchy</strong>, including registers, shared memory, caches, and global memory.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=055e2f4b6699" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning GPU Kernels #1: Threads, Blocks and First CUDA Kernel]]></title>
            <link>https://medium.com/@rishipramodc/learning-gpu-kernels-1-threads-blocks-and-first-cuda-kernel-d0a64dcd16e5?source=rss-4c2dae8da1dc------2</link>
            <guid isPermaLink="false">https://medium.com/p/d0a64dcd16e5</guid>
            <category><![CDATA[parallel-computing]]></category>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[cuda]]></category>
            <category><![CDATA[nvidia-gpu]]></category>
            <dc:creator><![CDATA[Rishipramodch]]></dc:creator>
            <pubDate>Sat, 12 Sep 2026 06:41:41 GMT</pubDate>
            <atom:updated>2026-09-12T06:41:41.276Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>I am currently learning GPU kernel programming from the fundamentals, with the eventual goal of understanding and implementing optimized ML and LLM kernels. I’m documenting what I learn as I implement each concept. This is the first part of that journey, starting with CUDA’s execution model and my first working kernel.</blockquote><p>GPUs were originally developed to handle the highly parallel computations required for computer graphics, especially real-time 3D rendering. Early GPUs relied heavily on fixed-function hardware designed to perform specific graphics operations efficiently. Over time, they evolved into general-purpose parallel processors capable of handling workloads far beyond graphics.</p><p>CPUs and GPUs are designed with different priorities. A CPU focuses on executing a relatively small number of instruction streams as quickly as possible. Its cores are powerful and optimized for tasks that involve sequential execution, complex control flow, and low-latency processing. A GPU takes a different approach. Instead of relying on a small number of powerful cores, it is designed to execute a very large number of lightweight threads concurrently.</p><p>Individual GPU threads may not be as powerful as CPU threads, but the ability to run thousands of them in parallel allows the GPU to achieve much higher overall throughput for workloads that contain large amounts of independent computation.</p><h3>What is a GPU kernel?</h3><p>Forget about CUDA syntax for a moment and let’s say we have 2 arrays.</p><p>A = [1, 2, 3, 4]<br>B = [5, 6, 7, 8]</p><p>and we want C = A + B.</p><p>A CPU could do:</p><p>C[0] = A[0] + B[0]<br>C[1] = A[1] + B[1]<br>C[2] = A[2] + B[2]<br>C[3] = A[3] + B[3]</p><p>But notice something C[0], C[1], C[2] and C[3] don’t depend on each other. These operations are independent of each other and operations can happen in parallel. That’s exactly the kind of problem GPUs love.</p><ol><li><strong>CPU vs GPU</strong></li></ol><p>CPU → Small number of powerful cores</p><p>Core<br>│<br>├── powerful<br>│<br>Core<br>│<br>├── powerful<br>│<br>Core<br>│<br>…</p><p>GPU → Huge amount of parallel computation</p><p>● ● ● ● ● ● ● ●<br>● ● ● ● ● ● ● ●<br>● ● ● ● ● ● ● ●<br>● ● ● ● ● ● ● ●<br>● ● ● ● ● ● ● ●<br>…</p><p>GPUs are excellent for workloads such as Matrix Multiplication, Neural Networks, Transformers and LLMs. The important idea is that GPUs aren’t “faster”, but GPUs are designed to perform lots of similar operations concurrently.</p><p><strong>2. What is a kernel?</strong></p><p>Before stepping into kernel, first let us understand the concept of heterogeneous systems. The CUDA programming model assumes a heterogeneous computing system, which means a system that includes both CPUs and GPUs. The CPU and the memory directly connected to it are called host and host memory. A GPU and the memory directly connected to it are called device and device memory.</p><p>CUDA applications execute some part of their code on the GPU, but applications always start execution on the CPU. The host code that runs on the CPU can use CUDA APIs to copy data between the host memory and the device memory, start code executing on the GPU, and wait for data transfers or GPU code to complete. The CPU and GPU can execute code simultaneously and the best performance is usually found by maximizing utilization of both CPUs and GPUs.</p><p>The code an application executes on the GPU is referred to as device code and a function that is invoked for executing on the GPU is called a kernel.</p><blockquote>A <strong>kernel</strong> is a specific function that runs on the GPU hardware.</blockquote><p>Instead of writing a loop that processes items one after another, you write a kernel as if it is only processing one single item. When a kernel is launched, CUDA creates many logical threads that execute the same kernel function, running it simultaneously on different portions/elements of the data.</p><p>CUDA follows a <strong>SIMT, or Single Instruction, Multiple Threads, execution model</strong>. Threads are organized into groups of 32 called warps. Threads within a warp execute the same kernel code, although individual threads can follow different control-flow paths.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/398/1*GLClXj5CYa3UfUKMnj8U2g.png" /></figure><p>In CUDA:</p><pre>__global__ void add(...)<br>{<br>// GPU code<br>}</pre><p>__global__ tells CUDA that this function is a GPU kernel but is launched from the host (CPU).</p><p>So think like:</p><p>Normal C++ function → Runs on CPU.</p><p>CUDA kernel → Runs on GPU → 1000s /1,000,000s of threads execute it.</p><p>So instead of declaring multiple elements</p><pre>add_element_0()<br>add_element_1()<br>add_element_2()<br>...<br>add_element_999999()</pre><p>You write one kernel</p><pre>__global__ void vector_add(...)<br>{<br>...<br>}</pre><p>and tell the GPU to run this function using many threads. Each thread gets an ID. Thread 0 handles element 0, Thread 1 handles element 1 and Thread 2 handles element 2. This idea is at the heart of GPU kernel programming.</p><h3>3. But GPUs don’t simply have “threads”</h3><p>CUDA organizes threads into blocks</p><pre>Grid<br>│<br>├── Block 0<br>│    ├── Thread 0<br>│    ├── Thread 1<br>│    ├── Thread 2<br>│    └── ...<br>│<br>├── Block 1<br>│    ├── Thread 0<br>│    ├── Thread 1<br>│    ├── Thread 2<br>│    └── ...<br>│<br>└── Block 2<br>     ├── Thread 0<br>     ├── Thread 1<br>     └── ...</pre><p>When an application launches a kernel, thousands of threads are assigned into blocks. A group of threads is called a thread block. These thread blocks are organised into grids. All thread blocks in a grid have the same dimensions.</p><pre>Threads → Blocks → Grid</pre><p>When a kernel is launched, it is launched using a specific execution configuration which specifies the grid and thread block dimensions. The execution configuration may also include parameters such as cluster size, stream, and SM configuration settings etc.</p><p>Streaming Multiprocessor (SM) → It acts as a mini-processor. It contains execution hardware, warp schedulers, registers, shared memory and caches used while executing thread blocks.</p><p>All threads of a thread block are executed in a single SM. This allows threads in the thread blocks to communicate, exchange information and synchronize with each other efficiently.</p><p>A grid may consist of millions of thread blocks, while the GPU executing the grid may only have tens or hundreds of SMs. All threads in a thread block are executed by a single SM. There is no guaranteed execution order between thread blocks, so one block should not rely on another block having already completed.</p><p>CUDA is designed so that a grid can scale across GPUs with very different numbers of Streaming Multiprocessors. For this to work, thread blocks should be independently executable: a block should not assume that another block has already completed or depend on a particular block execution order.</p><p>CUDA is therefore free to distribute blocks across the available SMs and execute them in whatever order the hardware scheduler chooses. Depending on the GPU and the workload, different blocks may execute concurrently or at different times.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7cPOfrc13jpM440TEU2TIw.png" /></figure><p><strong>CUDA gives every thread an ID</strong></p><pre>threadIdx.x<br>blockIdx.x<br>blockDim.x<br>gridDim.x</pre><p>threadIdx.x → ID of the thread inside a block</p><p><strong>eg., Block 0</strong></p><pre>Thread 0     threadIdx.x = 0<br>Thread 1     threadIdx.x = 1<br>Thread 2     threadIdx.x = 2<br>Thread 3     threadIdx.x = 3</pre><p>blockIdx.x → Suppose there are multiple blocks</p><pre>Block 0<br>	Thread 0<br>	Thread 1<br>	Thread 2<br>	Thread 3<br>	<br>Block 1<br>	Thread 0<br>	Thread 1<br>	Thread 2<br>	Thread 3</pre><p>Block 0 → Thread 0</p><p>Block 1 → Thread 0</p><p>Both threads are having same threadIdx.x. How do we distinguish them?</p><p>That’s where blockIdx.x comes in.</p><pre>Block 0 → blockIdx.x = 0<br>Block 1 → blockIdx.x = 1<br>Block 2 → blockIdx.x = 2</pre><p>blockDim.x → The block dimension is the total number of threads in a block.</p><p>gridDim.x → The grid dimension is the total number of blocks in a grid.</p><h3>4. Global thread index</h3><pre>int i = blockIdx.x * blockDim.x + threadIdx.x;</pre><p>This is the common formula used to give each thread a unique global ID.</p><p>eg: There are two blocks, each block containing four threads, so block dimension is 4.</p><p>Block 0 → blockIdx.x = 0</p><pre>Thread 0: i = 0 × 4 + 0 = 0<br>Thread 1: i = 0 × 4 + 1 = 1<br>Thread 2: i = 0 × 4 + 2 = 2<br>Thread 3: i = 0 × 4 + 3 = 3</pre><p>Block 1 → blockIdx.x = 1</p><pre>Thread 0: i = 1 × 4 + 0 = 4<br>Thread 1: i = 1 × 4 + 1 = 5<br>Thread 2: i = 1 × 4 + 2 = 6<br>Thread 3: i = 1 × 4 + 3 = 7</pre><h3>5. GPU kernel</h3><pre>__global__ void vector_add(float* A, float* B, float* C, int N){<br>    int i = blockIdx.x * blockDim.x + threadIdx.x;<br>    if (i &lt; N){<br>        C[i] = A[i] + B[i];<br>    }<br>}</pre><p>N → The total number of elements to be computed. In this example, each thread processes a single element.</p><p>Every GPU thread executes the same kernel, but each thread has a different ID. So one kernel becomes</p><pre>Thread 0 ─&gt; C[0]<br>Thread 1 ─&gt; C[1]<br>Thread 2 ─&gt; C[2]<br>Thread 3 ─&gt; C[3]<br>...<br>Thread N-1 ─&gt; C[N-1]</pre><h3>Writing and Running My First CUDA Kernel: Vector Addition</h3><p>Now it is time to put all of the concepts together and run an actual CUDA program.</p><p>For my first CUDA kernel, I implemented <strong>vector addition</strong>.</p><p>Suppose we have two vectors:</p><pre>A = [1, 2, 3, 4, 5, 6, 7, 8]<br>B = [10, 20, 30, 40, 50, 60, 70, 80]</pre><p>We want the GPU to calculate:</p><pre>C = A + B</pre><p>Instead of letting one CPU loop calculate every element sequentially, we will launch multiple GPU threads and let each thread calculate one element of C.</p><p><strong>The complete CUDA program for vector addition</strong></p><pre>#include &lt;stdio.h&gt;<br>#include &lt;cuda_runtime.h&gt;<br><br>// GPU kernel for vector addition<br>__global__ void vector_add(float* A, float* B, float* C, int N){<br>    int i = blockDim.x*blockIdx.x+threadIdx.x;<br>    if(i&lt;N){<br>        C[i]=A[i]+B[i];<br>    }<br>}<br><br>int main(){<br>    const int N = 8;<br>    const int size = N * sizeof(float);<br><br>    float A[N] = {1,2,3,4,5,6,7,8};<br>    float B[N] = {10,20,30,40,50,60,70,80};<br>    float C[N];<br><br>    float *d_A, *d_B, *d_C;<br>    cudaMalloc(&amp;d_A, size);<br>    cudaMalloc(&amp;d_B, size);<br>    cudaMalloc(&amp;d_C, size);<br><br>    cudaMemcpy(d_A, A, size, cudaMemcpyHostToDevice);<br>    cudaMemcpy(d_B, B, size, cudaMemcpyHostToDevice);<br><br>    int threadsPerBlock = 4;<br><br>    // Total number of blocks to launch<br>    int blocks = (N + threadsPerBlock-1)/threadsPerBlock;<br>    <br>    vector_add&lt;&lt;&lt;blocks, threadsPerBlock&gt;&gt;&gt;(d_A, d_B, d_C, N);<br><br>    cudaMemcpy(C, d_C, size, cudaMemcpyDeviceToHost);<br>    for(int i=0;i&lt;N;i++)<br>        printf(&quot;%f\n&quot;, C[i]);<br>    <br>    cudaFree(d_A);<br>    cudaFree(d_B);<br>    cudaFree(d_C);<br>    return 0;<br>}</pre><p><strong>Output:</strong></p><pre>11.000000<br>22.000000<br>33.000000<br>44.000000<br>55.000000<br>66.000000<br>77.000000<br>88.000000</pre><h3>Setting Up CUDA to Run This Program</h3><p>Before running the vector-addition kernel, we need a working CUDA development environment.</p><p>If CUDA is already installed on your system and both nvidia-smi and nvcc --version work correctly, you can skip the installation part and move directly to <strong>Setting Up Visual Studio Code</strong> and the compilation steps for your operating system.</p><p>To run CUDA programs locally, you generally need:</p><ul><li>A CUDA-capable NVIDIA GPU</li><li>A compatible NVIDIA GPU driver</li><li>The NVIDIA CUDA Toolkit</li><li>A supported C++ compiler</li><li>A code editor or IDE such as Visual Studio Code</li></ul><p>The CUDA Toolkit provides nvcc, NVIDIA&#39;s compiler driver for compiling CUDA C/C++ programs.</p><h3>Windows Setup</h3><h4>Step 1: Check whether your system has an NVIDIA GPU</h4><p>Open <strong>Command Prompt</strong> or <strong>PowerShell</strong> and run:</p><pre>nvidia-smi</pre><p>If the NVIDIA driver is installed correctly, this should display information about your GPU, driver, memory usage, and supported CUDA driver version.</p><p>If the command is not recognized or no NVIDIA GPU is detected, make sure your system has a CUDA-capable NVIDIA GPU and install/update the NVIDIA GPU driver.</p><h4>Step 2: Install the C++ compiler</h4><p>CUDA uses a host C++ compiler along with nvcc.</p><p>On Windows, install either:</p><ul><li>Microsoft Visual Studio, or</li><li>Microsoft Visual Studio Build Tools</li></ul><p>During installation, select:</p><p><strong>Desktop development with C++</strong></p><p>This installs the Microsoft C++ compiler and the required Windows development tools.</p><h4>Step 3: Install the NVIDIA CUDA Toolkit</h4><p>Download the CUDA Toolkit from <a href="https://developer.nvidia.com/cuda-downloads">NVIDIA’s official CUDA Downloads page.</a></p><p>Select your system configuration:</p><pre>Operating System → Windows<br>Architecture     → x86_64<br>Version          → Your Windows version<br>Installer Type   → exe</pre><p>Run the installer and follow NVIDIA’s installation instructions.</p><p>I recommend using the current CUDA version supported by your GPU and development environment instead of copying a specific CUDA version from this article, because CUDA releases and compiler support change over time.</p><h4>Step 4: Verify the CUDA installation</h4><p>Open a new terminal after installation and run:</p><pre>nvcc --version</pre><p>You should see output containing the installed CUDA compiler version.</p><p>You can also run:</p><pre>nvidia-smi</pre><p>If both commands work, the basic CUDA environment is ready.</p><p><a href="https://docs.nvidia.com/cuda/cuda-installation-guide-microsoft-windows/">NVIDIA’s official Windows installation guide.</a></p><h3>Linux Setup</h3><p>CUDA installation on Linux depends on the Linux distribution and version you are using.</p><p>For example, the installation process is different for Ubuntu, Debian, Fedora, RHEL, and other distributions.</p><p>Because of this, I recommend using <a href="https://docs.nvidia.com/cuda/cuda-installation-guide-linux/">NVIDIA’s official CUDA Linux Installation Guide</a> instead of copying commands intended for a different Linux distribution.</p><p>First, check whether the system detects an NVIDIA GPU:</p><pre>lspci | grep -i nvidia</pre><p>If the NVIDIA driver is already installed, you can also check:</p><pre>nvidia-smi</pre><p>Then use NVIDIA’s CUDA Downloads page and select your exact configuration:</p><pre>Operating System → Linux<br>Architecture     → Your architecture<br>Distribution     → Ubuntu / Debian / Fedora / etc.<br>Version          → Your distribution version<br>Installer Type   → deb / rpm / network / local</pre><p>NVIDIA will generate the appropriate installation commands for that particular system.</p><p>After installing the driver and CUDA Toolkit, verify the installation:</p><pre>nvidia-smi</pre><p>and:</p><pre>nvcc --version</pre><p>If nvcc is not found after installation, follow the <strong>post-installation environment configuration</strong> given in NVIDIA&#39;s Linux Installation Guide for the CUDA version you installed.</p><p>I prefer this approach instead of hard-coding /usr/local/cuda-XX.X/ into the article because the installation path can change depending on the CUDA version and installation method.</p><h3>What About macOS?</h3><p>Modern macOS systems cannot natively compile and execute CUDA applications.</p><p>CUDA requires NVIDIA GPU hardware, while modern Apple Silicon Macs use Apple’s own GPU architecture and frameworks such as Metal.</p><p>Therefore, if you are using a Mac and want to follow the CUDA examples in this series, a practical option is to use a remote or cloud Linux machine with an NVIDIA GPU.</p><p>The workflow would then look like:</p><pre>Mac<br> ↓<br>SSH / Remote Development<br> ↓<br>Linux machine with NVIDIA GPU<br> ↓<br>CUDA Toolkit<br> ↓<br>Compile and run CUDA kernels</pre><p>There are experimental community projects that attempt to translate parts of CUDA to Apple GPU APIs, but they do not provide the same environment as native CUDA on an NVIDIA GPU. For learning CUDA itself, I would use an actual NVIDIA GPU environment.</p><h3>Setting Up Visual Studio Code</h3><p>VS Code is not required to run CUDA, but it is the editor I am using for this learning series.</p><p>Open VS Code and install the <strong>C/C++</strong> extension from Microsoft.</p><p>You can also install <a href="https://docs.nvidia.com/nsight-visual-studio-code-edition/latest/install-setup/"><strong>NVIDIA Nsight Visual Studio Code Edition</strong></a> if you want CUDA-aware editing and debugging features.</p><p>Then create a working folder, for example:</p><pre>GPU_kernel/</pre><p>Inside it, create:</p><pre>vector_addition.cu</pre><p>The .cu extension indicates that the source file contains CUDA C/C++ code.</p><p>Paste the vector-addition program into this file.</p><h3>Compiling the Program on Windows</h3><p>Open your project folder in VS Code:</p><pre>GPU_kernel/<br>└── vector_addition.cu</pre><p>Open the integrated terminal and check:</p><pre>nvcc --version</pre><p>If it works, compile the program:</p><pre>nvcc vector_addition.cu -o program.exe</pre><p>Then run it.</p><p>In PowerShell:</p><pre>.\program.exe</pre><p>In Command Prompt:</p><pre>program.exe</pre><p>So the complete process is:</p><pre>vector_addition.cu<br>        ↓<br>       nvcc<br>        ↓<br>    program.exe<br>        ↓<br>      execute</pre><h3>What if Windows Says cl.exe Cannot Be Found?</h3><p>On Windows, nvcc uses Microsoft&#39;s C++ compiler as its host compiler.</p><p>If you receive an error saying that cl.exe cannot be found, open:</p><p><strong>x64 Native Tools Command Prompt for Visual Studio</strong></p><p>from the Windows Start menu.</p><p>Navigate to your project directory:</p><pre>cd /d R:\GPU_kernel</pre><p>In my case, R:\GPU_kernel is my project directory. Replace it with the location where you created your CUDA project.</p><p>Then compile:</p><pre>nvcc vector_addition.cu -o program.exe</pre><p>and run:</p><pre>program.exe</pre><p>You can also launch VS Code from this configured terminal:</p><pre>code .</pre><p>VS Code will then inherit the Visual Studio compiler environment from that terminal session.</p><h3>Compiling the Program on Linux</h3><p>Navigate to the folder containing the CUDA file:</p><pre>cd /path/to/GPU_kernel</pre><p>Compile:</p><pre>nvcc vector_addition.cu -o program</pre><p>Run:</p><pre>./program</pre><h3>What’s next?</h3><p>Vector addition gave me the basic execution model, but simply making a program run on a GPU does not mean that it is efficient.</p><p>The next step in my GPU-kernel learning journey is to understand more deeply how <strong>warps and Streaming Multiprocessors execute these threads</strong>, and then move toward the GPU memory hierarchy and kernel optimization.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d0a64dcd16e5" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>