Understanding SIMD: How Parallel Processing Accelerates Performance
-
SIMD is considered hardware-level parallelism that is implemented on CPU design phase. SIMD refers to a parallel processing technique that allows a single instruction to process multiple piece of data simultaneously.
Modern CPUs are equipped with wide registers (e.g, 256bit) that can store multiple numbers at once, enabling parallel data processing with in a single operation. This approach reduce a latency associated with fetching data from memory by retrieving the data once.
SIMD is a method that performs parallel processing by utilizing registers. But what are registers?
Registers are temporary memory storage units within the CPU, designed to hold data that the CPU is actively working with. Instead of fetching data directly from memory (RAM), which can be slow, the required data is first loaded into registers for faster access and computations.
Get Youngjin(Jay) Na’s stories in your inbox
Join Medium for free to get updates from this writer.
For instance, when two integers are loaded into registers, CPU can perform computations on them within a single clock cycle. With SIMD, this process is further optimized to enable hardware-level parallelism, allowing multiple computations to run simultaneously.
Element-wise addition in vector operations is a representative example of SIMD.
A typical CPU processes such calculations sequentially. However, since element-wise operations in a vector are independent and can be done at the same time, parallel processing is possible. By utilizing SIMD instructions, the computation speed can be improved.
Load both numbers of each vector in a single operation, and addition of all numbers happens in one clock cycle. After that storage back in memory happens in one cycle. In other words, vector operations can be performed efficiently.
In Julia, SIMD can be executed through macros.
A macro is a meta programming techniques which takes exisitng code as an input, and manipulates it to produce more code. In Julia the macro is indicated by @
If the CPU does not support these extensions, using SIMD instructions will not result in performance improvement.
There are SIMD extensions such as SSE and AVX, which are quite old command sets used to implement SIMD. SIMD can also be utilized in languages like Python, C, and Fortran, in addition to Julia. The way SIMD is used differs slightly in each language.
Furthermore, utilizing SIMD should be considered before using it, especially for calculations where accuracy is important.
SIMD performs multiple calculations simultaneously, so the order of operations may be handled differently, but because floating-point operations do not follow the associative law, errors can occur when performing parallel operations. For instance, (a+b) + c is not equal a + (b+c) since floating point arithmetic is not associative.
Benchmarking SIMD
Why the results have no big difference?
The reason is that using 0 as a variable for storing the total value may require type conversion by the compiler, which can potentially affect SIMD optimization. zero(eltype(numbers))is explicitly set to Float32 from the beginning, making it easier to optimize the operations.
Reference
1. Jamie Mair, “High Performance Computing Lecture” , University of Nottingham 2023.
