<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Martin Ayvazyan on Medium]]></title>
        <description><![CDATA[Stories by Martin Ayvazyan on Medium]]></description>
        <link>https://medium.com/@martin00001313?source=rss-4a528c57ee66------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*3E6LpaH0Y4R3LEpKjAACCg.jpeg</url>
            <title>Stories by Martin Ayvazyan on Medium</title>
            <link>https://medium.com/@martin00001313?source=rss-4a528c57ee66------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 08:48:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@martin00001313/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[ML Compilers: The Reality of Memory, Bandwidth, and Compiler Optimizations]]></title>
            <link>https://medium.com/@martin00001313/ml-compiler-optimizations-a-practical-c-guide-to-quantization-palletization-tiling-fusion-27153b7440e1?source=rss-4a528c57ee66------2</link>
            <guid isPermaLink="false">https://medium.com/p/27153b7440e1</guid>
            <category><![CDATA[cpp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[compilers]]></category>
            <category><![CDATA[cpu]]></category>
            <dc:creator><![CDATA[Martin Ayvazyan]]></dc:creator>
            <pubDate>Wed, 22 Apr 2026 06:36:10 GMT</pubDate>
            <atom:updated>2026-04-22T06:40:22.946Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*LLNNhNHIY-MznlthIes2Eg.png" /></figure><h3>Why ML Needs Its Own Optimization Playbook</h3><p>In my earlier articles on high-performance C++ (<a href="https://medium.com/@martin00001313/advanced-c-optimization-techniques-for-high-performance-applications-part-1-55aba61b4dad">Part 1</a>, <a href="https://medium.com/@martin00001313/advanced-c-optimization-techniques-for-high-performance-applications-part-2-6dc2ebf64813">Part 2</a>, <a href="https://medium.com/@martin00001313/advanced-c-optimization-techniques-for-high-performance-applications-part-3-4602df9284d8">Part 3</a>) we treated the CPU as the target and talked about branches, caches, SIMD, false sharing, and data-oriented design. ML workloads change the shape of the problem in three ways:</p><ul><li><strong>The compute unit is wider.</strong> A single NVIDIA H100 SM can issue 128 FP32 FMAs per clock per sub-core, and tensor cores push that into the thousands of effective ops when operating on tiles. On Apple Silicon the ANE is an even more specialized accelerator with its own IR and scheduling model. Extracting that throughput is a compiler problem, not a hand-written-assembly problem.</li><li><strong>The memory hierarchy grew a new tier.</strong> Between DRAM and registers sits <em>scratchpad / shared memory</em> — software-managed, banked, and dozens of times faster than L2. Good ML code lives in it.</li><li><strong>Numerical precision is negotiable.</strong> Unlike general-purpose C++, we can trade bits of precision for throughput and memory footprint. Quantization and palletization are only possible because ML models tolerate controlled error.</li></ul><p>This article walks through the optimization passes that modern ML compilers (XLA, TVM, IREE, Core ML, TensorRT, the MLIR ecosystem) apply, with compact C++ examples for each. Topics covered:</p><ul><li><strong>Quantization</strong> — why it’s a formula, not a cast</li><li><strong>Palletization</strong> — weight clustering + LUT lookup</li><li><strong>Tiling</strong> — from CPU cache blocking to GPU shared-memory tiles</li><li><strong>Shared memory bank conflicts</strong> — the GPU cousin of false sharing</li><li><strong>Operator fusion</strong> — elementwise chains and epilogues</li><li><strong>KV cache optimization</strong> — attention’s memory-bound reality</li></ul><h4>Quantization: It’s Not static_cast, It&#39;s an Affine Map</h4><p>The most common beginner question about INT8 quantization: <em>“Why can’t I just </em><em>static_cast&lt;int8_t&gt;(x) on every weight?&quot;</em></p><p>Because FP32 values are distributed over a wide, signed, often non-symmetric range — and int8_t only has 256 levels. Casting loses the scale entirely. What you actually need is an <strong>affine map</strong> between the float domain and the integer domain:</p><pre>quantize:   x_q = round(x_f / s) + z<br>dequantize: x_f ~= s * (x_q - z)</pre><p>Where s (scale) is a positive float answering &quot;how many floats does one integer step represent?&quot; and z (zero-point) is an integer in the quantized range answering &quot;which integer value represents the float 0.0?&quot;</p><p>Zero-point exists so that the quantized representation of 0.0 is <em>exactly</em> representable, which matters for padding, ReLU, and any operator that special-cases zero.</p><p><strong>Symmetric vs. asymmetric</strong></p><ul><li><strong>Symmetric</strong> (z = 0): scale picked so that [-a, a] in float maps to [-127, 127] in INT8. Simple, but wastes bits if the tensor is skewed (e.g., post-ReLU activations are non-negative).</li><li><strong>Asymmetric</strong> (z != 0): scale and zero-point both chosen from the observed [min, max]. Uses the full range for skewed distributions.</li></ul><p>Typical choice: <strong>symmetric for weights</strong> (usually near-zero-centered), <strong>asymmetric for activations</strong> (often post-ReLU, so one-sided).</p><p>A minimal quantize/dequantize pair</p><pre>#include &lt;algorithm&gt;<br>#include &lt;cmath&gt;<br>#include &lt;cstdint&gt;<br>#include &lt;limits&gt;<br><br>struct QParams {<br>    float scale;<br>    int32_t zero_point;<br>};<br><br>// Calibration: derive (s, z) from observed min/max.<br>// For asymmetric INT8: qmin = -128, qmax = 127.<br>QParams compute_qparams(float f_min, float f_max,<br>                        int32_t qmin = -128, int32_t qmax = 127) {<br>    // Always include 0 in the range so zero_point is representable.<br>    f_min = std::min(f_min, 0.0f);<br>    f_max = std::max(f_max, 0.0f);<br><br>    float scale = (f_max - f_min) / float(qmax - qmin);<br>    // Solve: f_min = scale * (qmin - z)  =&gt;  z = qmin - f_min/scale<br>    float z_real = static_cast&lt;float&gt;(qmin) - f_min / scale;<br>    int32_t z = static_cast&lt;int32_t&gt;(std::round(z_real));<br>    z = std::clamp(z, qmin, qmax);<br>    return {scale, z};<br>}<br><br>int8_t quantize(float x, const QParams&amp; q) {<br>    int32_t v = static_cast&lt;int32_t&gt;(std::round(x / q.scale)) + q.zero_point;<br>    return static_cast&lt;int8_t&gt;(std::clamp(v, -128, 127)); // std::clamp to pick right apx if needed<br>}<br><br>float dequantize(int8_t x, const QParams&amp; q) {<br>    return q.scale * (static_cast&lt;int32_t&gt;(x) - q.zero_point);<br>}</pre><p>Notice what quantize actually does: <em>divide, round, add, clamp</em>. That&#39;s four operations per element — a long way from a bare cast. And that cost is paid <em>once</em> for weights (at compile time or during PTQ calibration), but <em>every forward pass</em> for activations.</p><p>Why the compiler cares: Q/DQ op pair insertion</p><p>In the source graph you write FP32 matmuls. The quantization pass inserts <strong>Quantize (Q)</strong> and <strong>Dequantize (DQ)</strong> ops around each op it wants to run in INT8:</p><pre>FP32 activation --&gt; [Q] --&gt; INT8 --&gt; [MatMul_INT8] --&gt; INT32 --&gt; [DQ] --&gt; FP32<br>                                           ^<br>                           INT8 weights ---&#39;</pre><p>A later quantization folding pass then deletes adjacent DQ -&gt; Q pairs when their params cancel, collapses runs of INT8 ops, and rewrites the matmul itself to consume quantized operands directly. The output [DQ] survives only if the <em>next</em> op runs in FP32.</p><p>If you’re implementing this pass, the key invariant to enforce: a mixed-precision subgraph is legal only if every producer-consumer edge has matching quantization semantics, or you’ve materialized a Q/DQ on that edge.</p><p><strong>Per-tensor vs. per-channel</strong></p><p>For a weight matrix W of shape [out, in], per-tensor quantization uses one (s, z) for the whole thing. Per-channel uses one (s, z) per output channel. Per-channel is almost always better for conv weights — each output filter has a different natural scale — and costs nothing in the kernel if s is broadcast along the right axis in the INT32-to-FP32 requantize step.</p><p><strong>Further depth</strong></p><p>Post-training techniques worth knowing: <strong>GPTQ</strong> (layer-wise second-order error correction), <strong>AWQ</strong> (activation-aware scaling of salient weights), <strong>SmoothQuant</strong> (migrates quantization difficulty from activations to weights via channel-wise scaling). Each is a different strategy for picking per-tensor (s, z) that minimizes the end-to-end output error, not just the per-weight reconstruction error.</p><p>***</p><h4>Palletization: Weight Clustering with LUTs</h4><p>Palletization is the ML alternative of palette-based image compression. Instead of storing every weight at full precision, we:</p><ul><li>Cluster the weights into K centroids (typically K-means in FP16 or FP32).</li><li>Store the centroids in a small <strong>lookup table</strong> (the “palette”).</li><li>Store each weight as a log2(K)-bit index into that table.</li></ul><p>For K = 16 centroids, each weight becomes 4 bits. A 175B-parameter model&#39;s weights drop from ~700 GB (FP32) to ~87 GB (4-bit palettized) — and because decode is just a table lookup, the dequantization path is cheaper per element than affine quantization.</p><p>The storage layout</p><pre>#include &lt;array&gt;<br>#include &lt;cstdint&gt;<br>#include &lt;vector&gt;<br><br>// K=16 palette, so each weight is 4 bits. Two weights per byte.<br>template &lt;size_t K&gt;<br>struct PalettizedTensor {<br>    std::array&lt;float, K&gt; palette;       // K centroids, stored in FP16/FP32<br>    std::vector&lt;uint8_t&gt; indices;       // packed 4-bit indices<br>    size_t num_weights;<br>};<br><br>// Decode a single weight. In practice this loop gets fused into the<br>// matmul kernel so the LUT lives in registers or shared memory.<br>inline float decode(const PalettizedTensor&lt;16&gt;&amp; t, size_t i) {<br>    uint8_t byte = t.indices[i / 2];<br>    uint8_t nibble = (i &amp; 1) ? (byte &gt;&gt; 4) : (byte &amp; 0x0F);<br>    return t.palette[nibble];<br>}</pre><p><strong>Why it matters at the kernel level</strong></p><p>For transformer inference, the matmul is almost always <strong>memory-bandwidth bound</strong>, not compute-bound. A 4-bit palettized weight delivers 8x the bytes-per-FLOP ratio of FP32. If your kernel can decode indices into the FMA pipeline fast enough, you get an 8x effective bandwidth multiplier — and on Apple Silicon’s unified memory or an NVIDIA GPU’s HBM, that’s the difference between 5 tokens/sec and 40 tokens/sec.</p><p>The decode is so cheap because a 16-entry palette fits in one cache line. On a GPU, you broadcast-load the palette into shared memory at the start of the kernel and every lane reads from it in parallel without bank conflicts (more on those shortly).</p><p>Also, palletization is <strong>weights-only </strong>as<strong> </strong>activations change every token, so you’d have to re-run K-means at inference time. Affine quantization handles activations because (s, z) can be computed on the fly from rolling min/max.</p><p><strong>Composition: palletized + affine</strong></p><p>A production quantization pipeline typically palletizes weights and affine-quantizes activations, then writes an INT4-by-INT8 matmul kernel producing INT32 accumulators. The weight decode and the matmul happen in the same kernel — there’s no materialized FP32 weight tensor in memory at any point during inference.</p><p>***</p><h4>Tiling: From Cache Blocking to Shared-Memory Residency</h4><p>Tiling is loop nest transformation for locality. In <a href="https://medium.com/@martin00001313/advanced-c-optimization-techniques-for-high-performance-applications-part-1-55aba61b4dad">Part 1</a> I covered it as <strong>cache blocking</strong> — chunk your iteration space so the working set fits in L2. GPU tiling is the same idea, but the relevant memory tier is <strong>shared memory / scratchpad </strong>(NVIDIA: 48-228 KB per SM; Apple GPU: threadgroup memory; TPU: VMEM/CMEM), and the transfer mechanism is explicit — either a cooperative load or a DMA descriptor.</p><p><strong>The canonical matmul tile</strong></p><p>For C = A * B where A is [M, K] and B is [K, N], the outermost tiled loop partitions M and N into blocks of BM x BN; the reduction axis K is tiled into chunks of BK. Inside each outer iteration, a BM x BK slab of A and a BK x BN slab of B are loaded into shared memory and reused by every thread in the block.</p><p><strong>Picking tile sizes</strong></p><p>The tile shape (BM, BN, BK) is constrained by:</p><ul><li><strong>Shared memory capacity.</strong> BM*BK + BK*BN floats must fit, with enough left for double-buffering (see below). At 48 KB per block and FP32, BM = BN = 128, BK = 16 uses 2 * (128*16) * 4 = 16 KB for a single stage.</li><li><strong>Arithmetic intensity.</strong> The tile does 2 * BM * BN * BK flops against (BM + BN) * BK loads. You want the ratio above the hardware&#39;s flop-per-byte roofline (on an H100, roughly 10 FP32 flops/byte, much higher for FP16 tensor cores) or the kernel is memory-bound.</li><li><strong>Register pressure per thread.</strong> C_tile is held in registers; too large a tile spills to local memory (which on NVIDIA is DRAM-backed and slow).</li><li><strong>Wave occupancy.</strong> Very large tiles reduce how many blocks can coexist on an SM, cutting latency hiding.</li></ul><p><strong>Double buffering (software pipelining on the tile loop)</strong></p><p>While the compute consumes stage k, asynchronously load stage k+1:</p><pre>// Two scratchpad buffers, ping-pong between them<br>float A_s[2][BM][BK], B_s[2][BK][BN];<br>int cur = 0;<br>load_async(A_s[0], B_s[0], /*k=*/0);   // prime<br>wait();<br>for (int k0 = 0; k0 &lt; K; k0 += BK) {<br>    int nxt = 1 - cur;<br>    if (k0 + BK &lt; K)<br>        load_async(A_s[nxt], B_s[nxt], k0 + BK);   // overlap<br>    compute(C_tile, A_s[cur], B_s[cur]);<br>    wait();<br>    cur = nxt;<br>}</pre><p>On NVIDIA this uses cp.async instructions (Ampere+) backed by the L1/shared memory async copy engine. On Apple Silicon, simdgroup_async_copy plays the same role. On TPUs and the ANE, it&#39;s an explicit DMA descriptor pushed to a scheduler. The common thread: <strong>the load is a first-class IR operation, not a </strong><strong>memcpy the compiler hopes to overlap.</strong></p><p>For those of us coming from CPU-land, software pipelining is familiar from <a href="https://medium.com/@martin00001313/advanced-c-optimization-techniques-for-high-performance-applications-part-2-6dc2ebf64813">Part 2</a> — this is the same idea, promoted to a different memory tier.</p><p>***</p><h4>Shared Memory Bank Conflicts: The GPU Cousin of False Sharing</h4><p>This is the connection I found most satisfying when I started on GPU kernels: <strong>shared memory bank conflicts are the GPU analog of false sharing on CPUs</strong> — and the fix, padding, is literally the same fix.</p><p><strong>The hardware</strong></p><p>NVIDIA shared memory is divided into <strong>32 banks</strong>, one per warp lane. Each bank is 4 bytes wide. Consecutive 32-bit words map to consecutive banks, wrapping every 128 bytes:</p><pre>offset (bytes):   0    4    8   12   ...  124  128  132 ...<br>bank:             0    1    2    3   ...  31    0    1  ...</pre><p>A warp can do a single shared-memory transaction per cycle <strong>if the 32 lanes access 32 distinct banks</strong> (or all read the same word — that’s broadcast). If two lanes hit the same bank on different addresses, the hardware serializes them into two cycles. 32 lanes all hitting bank 0 means a 32x slowdown on that instruction.</p><p><strong>The classic conflict: column access of a tile</strong></p><p>A transpose kernel makes this concrete:</p><pre>__shared__ float tile[32][32];   // 32 rows x 32 cols, row-major<br><br>// Cooperative load -- one element per lane, row-major.<br>// Lane t of warp w writes tile[w][t]. Addresses within the warp differ<br>// by one float, so lanes hit 32 distinct banks -- conflict-free.<br>tile[threadIdx.y][threadIdx.x] = gmem_in[...];<br>__syncthreads();<br><br>// Store transposed -- lane t reads tile[t][w] (swapped indices).<br>// Now addresses within the warp differ by one ROW = 32 floats = 128 bytes.<br>// Stride 128 bytes is a multiple of (32 banks * 4 bytes), so every lane<br>// hits bank 0. Result: 32-way bank conflict, serialized into 32 cycles.<br>gmem_out[...] = tile[threadIdx.x][threadIdx.y];</pre><p><strong>The fix: pad by one</strong></p><pre>__shared__ float tile[32][33];   // &lt;-- note the 33</pre><p>A 33-wide row is 132 bytes, which is not a multiple of 128. Now consecutive elements of a column land on consecutive banks (the stride is 33 floats mod 32 = 1), and the column access becomes conflict-free. Storage cost: 32 extra floats = 128 bytes of shared memory. This is exactly the alignas(64) trick from <a href="https://medium.com/@martin00001313/advanced-c-optimization-techniques-for-high-performance-applications-part-3-4602df9284d8">Part 3</a>. There I padded std::atomic&lt;int&gt; members so two counters couldn&#39;t share a cache line. Here we pad a tile&#39;s stride so two lanes can&#39;t share a bank. The physics is different (cache coherence vs. banked SRAM), but the solution lives in the same cognitive slot: <strong>when the hardware serializes access to a shared physical resource, change your layout so that doesn&#39;t happen.</strong></p><p>***</p><h4>Operator Fusion: Killing Materialized Intermediates</h4><p>The naive execution of a small MLP layer looks like this:</p><pre>x --&gt; [MatMul W] --&gt; t1 --&gt; [Add b] --&gt; t2 --&gt; [ReLU] --&gt; t3 --&gt; [MatMul W2] --&gt; y</pre><p>Each tN is a full tensor written to DRAM and read back. For reasonable layer widths, that&#39;s gigabytes per token of wasted bandwidth — because every tN is consumed by exactly the next op, then discarded. <strong>Operator fusion</strong> rewrites the graph so that producer and consumer share a kernel:</p><pre>x --&gt; [MatMul + Bias + ReLU] --&gt; t3 --&gt; [MatMul W2] --&gt; y</pre><p>Now t1, t2 never exist as tensors. The bias-add happens on values already in the matmul&#39;s accumulator registers; the ReLU clamps those registers before the final store. A single kernel reads x and W once, writes t3 once.</p><p><strong>Two flavors of fusion</strong></p><p><strong>Elementwise chain fusion</strong> — the easy case. Any chain of elementwise ops (add, mul, ReLU, GELU, sigmoid, scale, cast) can be collapsed into one pass over the tensor. The compiler walks the use-def chain, collects the ops, and emits a single loop:</p><pre>// Before: three passes over memory<br>for (int i = 0; i &lt; N; ++i) t1[i] = x[i] + b[i];<br>for (int i = 0; i &lt; N; ++i) t2[i] = t1[i] * s;<br>for (int i = 0; i &lt; N; ++i) y[i]  = std::max(0.0f, t2[i]);<br><br>// After fusion: one pass, intermediates in registers<br>for (int i = 0; i &lt; N; ++i)<br>    y[i] = std::max(0.0f, (x[i] + b[i]) * s);</pre><p>Any competent compiler does this for dense tensors at -O3 anyway. The win is when the ops come from different source functions or different graph builders — the ML compiler can see the full graph, the C++ compiler can&#39;t.</p><p><strong>Epilogue fusion</strong> — fusing elementwise ops onto the end of a reduction kernel (matmul, conv, softmax). This is where the big wins live. A matmul kernel already has the output values in its accumulator registers right before the store; applying bias/scale/activation at that point is free. This is what cuBLAS-Lt, CUTLASS epilogues, and XLA’s fusion passes exist for.</p><p><strong>FlashAttention</strong> is the most famous epilogue-fusion-plus-tiling pattern: it fuses Q * K_transposed, softmax, and multiplication by V into one kernel so the N x N attention matrix never materializes. For a 4096-token sequence, that&#39;s 16 MB of FP32 per head that we no longer round-trip to HBM — the reason FlashAttention is 2-4x faster than the three-kernel implementation.</p><p><strong>Where fusion stops</strong></p><p>Fusion is safe only when the producer’s output is consumed by exactly one operator with a compatible access pattern. If t1 is also an input to a skip connection further down, you can&#39;t eliminate it — someone else needs it. The compiler&#39;s fusion pass therefore runs a use-count analysis first; ops with use_count &gt; 1 are fusion barriers unless recomputation is cheaper than the reload (a trade-off called <strong>rematerialization</strong>).</p><p>***</p><h4>KV Cache: Attention’s Memory-Bound Reality</h4><p>Autoregressive transformer inference is dominated by attention over the KV cache. For each generated token, the model computes Q for the new token, then attends to the concatenation of <em>all previous tokens&#39;</em> K and V. If you re-ran the full attention every time, you&#39;d redo quadratic work per token. The <strong>KV cache</strong> stores K and V across steps so per-token work is linear.</p><p><strong>Layout matters</strong></p><p>The cache has shape roughly [num_layers, 2, batch, num_heads, seq_len, head_dim]. Each decode step appends one row per layer per head. How you lay this out determines whether your decode kernel streams or thrashes.</p><pre>// Layout A: [batch, num_heads, seq_len, head_dim] — &quot;BHSD&quot;<br>// New token appends along seq_len (innermost changing dim) =&gt; scattered writes.<br><br>// Layout B: [batch, num_heads, head_dim, seq_len] — &quot;BHDS&quot;<br>// Writes contiguous along head_dim per head — friendlier for append.<br><br>// Layout C (PagedAttention): split seq_len into fixed-size pages,<br>// indirect through a page table. Decouples logical position from physical.</pre><p>Per-token decode reads about 2 * num_layers * num_heads * seq_len * head_dim * dtype_bytes of cache. For Llama-70B at 4096 tokens with FP16: roughly 2 * 80 * 64 * 4096 * 128 * 2 bytes = ~10 GB per decode step. This is the entire reason long-context inference is slow — you&#39;re streaming 10 GB of HBM for every token. Every trick that shrinks this dominates.</p><h3>Compounding tricks</h3><ul><li><strong>Quantize the cache.</strong> INT8 KV cache halves the bandwidth bill with negligible quality loss. INT4 cache (with group quantization) halves it again.</li><li><strong>Grouped-Query Attention (GQA)</strong> and <strong>Multi-Query Attention (MQA)</strong>. Share K and V across groups of query heads. Llama 2 70B uses 8-way GQA, cutting cache size by 8x.</li><li><strong>PagedAttention (vLLM).</strong> Treat the KV cache like a paged virtual memory system. Pages are fixed-size blocks of (K_page, V_page), and a per-sequence page table maps logical token positions to physical pages. This kills fragmentation across variable-length sequences in a batched server and is why vLLM gets 2-4x throughput over naive implementations.</li><li><strong>Sliding-window / dilated attention.</strong> Cap the cache at a fixed window (Mistral’s 4K sliding window). Bounded memory, bounded decode cost.</li><li><strong>Speculative decoding.</strong> Orthogonal, but worth mentioning: use a small model to draft k tokens, verify them with the big model in one forward pass. Amortizes the cache read across k tokens.</li></ul><p>If you’re designing the compiler pass that materializes the KV cache, the important decision is the memory layout contract between the attention kernel and the cache allocator. Paged layouts are harder to codegen for (indirect loads) but pay for themselves in batched serving; contiguous layouts are easier on the kernel but waste memory for variable-length workloads.</p><p>***</p><h3>Putting It Together: The Pass Pipeline</h3><p>A real ML compiler applies these as a pipeline, not in isolation. A simplified view of the stack for an NVIDIA GPU target:</p><ol><li><strong>Frontend</strong> — PyTorch FX / StableHLO graph in, lowered to high-level tensor IR (e.g. stablehlo dialect in MLIR).</li><li><strong>Algebraic simplification</strong> — constant folding, dead code elimination, transpose pushdown.</li><li><strong>Quantization</strong> — insert Q/DQ, fold adjacent Q/DQ, rewrite ops to quantized variants.</li><li><strong>Palletization</strong> — cluster weights, rewrite MatMul to PalettizedMatMul with embedded LUT.</li><li><strong>Fusion</strong> — elementwise chains, epilogue fusion onto reductions, FlashAttention-style fusions.</li><li><strong>Tiling</strong> — pick tile shapes per op, lower to scf/affine nested loops.</li><li><strong>Memory placement</strong> — assign tensors to DRAM vs. shared memory, insert cp.async loads, double-buffer.</li><li><strong>Vectorization and unrolling</strong> — vectorize inner loops, unroll the reduction axis.</li><li><strong>Bank conflict elimination</strong> — insert padding or swizzling into shared-memory allocations.</li><li><strong>Codegen</strong> — lower through LLVM NVPTX to PTX, then ptxas to SASS.</li></ol><p>Each stage depends on decisions made upstream. You can’t pick good tile sizes until you know the element type (changed by quantization). You can’t fuse an epilogue until algebraic simplification has canonicalized the activation function’s operator form. You can’t avoid bank conflicts until tiling has chosen the shared-memory layout. This ordering is the main thing experienced ML-compiler engineers argue about, and it’s why the passes aren’t commutative.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=27153b7440e1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ML Compilers Aren’t All the Same — Here’s Why]]></title>
            <link>https://medium.com/@martin00001313/ml-compilers-arent-all-the-same-here-s-why-63d48ce438b0?source=rss-4a528c57ee66------2</link>
            <guid isPermaLink="false">https://medium.com/p/63d48ce438b0</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[compilers]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Martin Ayvazyan]]></dc:creator>
            <pubDate>Sun, 19 Apr 2026 00:05:32 GMT</pubDate>
            <atom:updated>2026-04-19T00:11:37.187Z</atom:updated>
            <content:encoded><![CDATA[<p>If you’ve trained or deployed a model recently, you’ve depended on an ML compiler — possibly several of them stacked on top of each other. PyTorch’s torch.compile, TensorRT, CoreML, XLA, TVM, Triton: all compilers, all solving variations of the same problem, all making surprisingly different architectural choices.</p><p>The interesting question isn’t “which is fastest?” — that depends on your hardware and workload. The interesting question is: <strong>why do they look so different from each other?</strong> Why does an NVIDIA TensorRT engine refuse to run on a different GPU generation while a CoreML model happily hops between the Neural Engine, GPU, and CPU on any Apple device shipped in the last five years? Why does JAX recompile your function the first time you call it while CoreML ships a fully baked binary?</p><p>These aren’t implementation accidents. They’re consequences of two fundamental design decisions every ML compiler has to make: <strong>when do you compile</strong>, and <strong>what do you target</strong>. Get those two axes straight and the rest of the ML compiler landscape clicks into place.</p><p>This article covers AOT vs JIT compilation, the hardware-agnostic vs hardware-specific split, where each class of optimization actually lives in the compilation stack, and finally a few “ideological” optimizations — <a href="https://arxiv.org/pdf/2205.14135">FlashAttention</a>, <a href="https://arxiv.org/pdf/2309.06180">PagedAttention</a>, palettization, <a href="https://arxiv.org/pdf/2504.19874">TurboQuant</a> — that didn’t just make things faster but changed how people think about the problem.</p><h3>When do you compile? AOT vs JIT</h3><p>The first decision is temporal. You can compile ahead of time (AOT), producing a deployable artifact, or just in time (JIT), compiling on first use. Both are used in production, and the choice cascades into almost everything else about the stack.</p><p>It’s tempting to frame this as a binary — AOT compilers produce deployable artifacts, JIT compilers defer everything to runtime — but in practice almost every production stack is a <strong>hybrid</strong>. The real question isn’t “AOT or JIT” but “which decisions get made when.” Different compilers draw that line in very different places, and the line itself is often not a single point but a sequence of compilation stages that run at different times.</p><p>Think of it as a spectrum with three checkpoints: <strong>author time</strong> (when you write or export the model), <strong>build time</strong> (when you produce a deployable artifact), and <strong>load time / run time</strong> (when the artifact meets actual hardware and actual inputs). Every compiler commits some set of decisions at each checkpoint.</p><p><strong>Pure AOT</strong> stacks commit almost everything at build time. TensorRT is the textbook example: you hand the builder a graph and a set of “optimization profiles” (shape ranges), and it benchmarks candidate kernels on the target GPU, picks winners, lays out memory, and bakes everything into an .engine file. Building takes minutes because the builder is literally timing kernels against each other. The payoff is that load time is trivial and run time is deterministic — there&#39;s nothing left to decide. The price is rigidity: an engine built for Hopper won&#39;t run on Blackwell, and inputs outside the declared shape envelope get refused.</p><p><strong>Pure JIT</strong> stacks defer everything to run time. The first call to a jax.jit&#39;d function traces it with the actual input shapes, lowers to StableHLO, hands it to XLA, and caches the result. torch.compile does the same through TorchDynamo and Inductor, emitting Triton kernels for GPU and C++/OpenMP for CPU. The first call is slow — the infamous compile warmup — but subsequent calls with compatible inputs hit the cache. The win is adaptivity: the compiler specializes on the batch size you&#39;re actually using rather than guessing what you might use. The cost is that you&#39;re shipping the compiler as part of your runtime, and cache misses can happen in surprising ways (a slightly different shape, a changed dtype).</p><p><strong>The hybrid middle</strong> is where most production stacks actually live, and it’s the pattern that matters most in practice. CoreML is the clearest example. The offline conversion step (via coremltools, usually on a developer&#39;s Mac) does all the heavy lifting that doesn&#39;t require knowing the exact hardware: graph optimization, operator fusion, weight compression, memory planning, lowering into MIL, producing an mlmodelc bundle. That bundle is fully portable across Apple devices — the <em>same file</em> runs on an iPhone, an iPad, a Mac, a Vision Pro. But when the app loads the model, a system service takes over: it inspects what compute units are actually available on this device, decides which subgraphs should go to the ANE, GPU, or CPU, and JIT-compiles the ANE-destined portion into the (undocumented) ANE program format. Neither checkpoint alone would suffice. The offline step can&#39;t do ANE codegen because it doesn&#39;t know the target device. The load-time step can&#39;t afford to redo the global graph optimizations because it&#39;d blow the app launch budget. So the work is split: AOT does what it can with device-agnostic information, and JIT finishes the job with device-specific knowledge. The payoff is both fast startup <em>and</em> device portability, which you can&#39;t get from either extreme.</p><p>This pattern shows up everywhere once you look for it. TensorRT-LLM builds engines offline but supports weight refitting and KV-cache replanning at load time, so you can swap in a fine-tuned checkpoint without rebuilding. PyTorch’s AOTInductor produces a .pt2 shared library you can load from C++ without Python, but torch.compile in its default mode still JITs. ONNX Runtime loads an ONNX file (an AOT artifact) but does graph-level optimizations and execution-provider-specific kernel selection at session creation — another load-time JIT step. Triton kernels are JIT-compiled on first call but cached to disk, so subsequent runs of your process are effectively AOT.</p><p>The useful mental model is: <strong>AOT buys determinism and deployability, JIT buys adaptivity, and the hybrid middle buys device portability without paying the full JIT cost at run time</strong>. The reason CoreML can ship one artifact that runs on every Apple device — while TensorRT has to ship a separate engine per GPU generation is that CoreML defers the hardware-specific decisions to load time, and TensorRT commits them at build time. Different bets, both defensible.</p><p>One trend worth noting: <strong>LLM serving is pushing the AOT boundary as late as possible without crossing it</strong>. vLLM adopted torch.compile as a core component in 2025 but spends real effort making the compiled path stable and cacheable — you want the first request after startup to be fast, not a 30-second warmup. TensorRT-LLM, SGLang, and MLC-LLM are AOT-flavored by default but with runtime hooks for weight updates and KV-cache management. At serving scale, steady-state throughput dominates, and the goal is to have a fully-specialized kernel launch sequence (often captured into CUDA Graphs) that doesn&#39;t stutter because something decided to recompile mid-request.</p><h3>What do you target? The CoreML vs TensorRT split</h3><p>The second decision is about scope, and this is where the CoreML vs TensorRT contrast becomes genuinely illuminating.</p><p>A <strong>TensorRT engine is hardware-specific to a fault</strong>. Not just “NVIDIA GPU” specific — <em>generation</em> specific. An engine built on an A100 won’t run on an H100. An engine built on H100 won’t run on B200. NVIDIA’s documentation says this plainly: engines are tied to the compute capability of the GPU they were built on. Why? Because at build time, TensorRT’s engine builder runs actual microbenchmarks on the target GPU for every layer in your graph. It tries candidate kernels from cuDNN, cuBLASLt, cuSPARSELt, its own library, and your custom plugins, times each one at the shapes you care about, and picks the fastest. That choice is baked into SASS — the actual machine code of a specific NVIDIA architecture. The result is essentially unbeatable performance on that exact SKU and a brick on any other.</p><p>A <strong>CoreML model, by contrast, is hardware-agnostic within the Apple family</strong>. We saw the mechanism in the previous section — the hybrid split between offline MIL-level work and load-time ANE codegen. The consequence here is what matters: the same mlmodelc that runs mostly on the Neural Engine on an iPhone 15 Pro runs mostly on the GPU on a base-model iPad, with no rebuild required. One artifact, many devices.</p><p>Why the difference between Apple and NVIDIA? It’s structural.</p><p><strong>NVIDIA’s customers span generations.</strong> A cloud provider has Ampere, Hopper, and Blackwell GPUs all serving production traffic simultaneously. NVIDIA can’t defer the hardware-specific decision to runtime because there is no runtime context that would make it cheap — picking the best kernel for SM90 vs SM100 requires actually running both, and you don’t want to do that on every model load. So the decision gets pushed to build time, and the compiler produces an artifact that refuses to run on the wrong hardware. This is the correct tradeoff for NVIDIA: maximum perf on each SKU, at the cost of an engine-per-SKU deployment matrix.</p><p><strong>Apple controls the entire vertical stack.</strong> The ANE, the GPU, the CPU, the unified memory architecture, the OS, the compiler, the model format. That means Apple can define an intermediate representation (MIL — Model Intermediate Language) that’s above the hardware-specific level, ship that in the mlmodelc, and let a system service do the final hardware mapping at load time. The unified memory architecture matters a lot here: there&#39;s no host-to-device copy at dispatch time because there&#39;s no separate device memory, which means the runtime cost of moving work between the ANE, GPU, and CPU is a scheduling decision rather than a data-movement one.</p><p>The same logic explains a lot of other splits in the compiler world. <strong>XLA</strong> started as a TPU compiler (a single hardware family Google controls) and now targets TPU, GPU, and CPU through PJRT plugins — it can be relatively agnostic because the heavy lifting is partitioned behind a plugin ABI. <strong>TVM</strong> is militantly hardware-agnostic because it’s designed to be retargetable: its whole pitch is “write a backend, get the optimizer for free.” <strong>ONNX Runtime</strong> is the most extreme example of the agnostic-by-delegation approach: it partitions your graph across “execution providers” (CUDA, TensorRT, CoreML, OpenVINO, WebGPU) and lets each provider handle its subgraph however it wants.</p><p>There’s a practical rule here. <strong>The further you are from the hardware vendor, the more agnostic your compiler has to be.</strong> NVIDIA ships TensorRT because they own the hardware and can afford to be specific. Apple ships CoreML because they own the hardware but also care about a shippable app model that doesn’t require recompilation per device. Meta, Google, Microsoft, and the open-source community ship agnostic stacks because they have to run on hardware they don’t control.</p><h3>Where optimizations live</h3><p>Now for the part that matters when you’re debugging a slow model: knowing which layer of the stack is responsible for which kind of speedup. ML compilers lower programs through a sequence of increasingly concrete representations, and different classes of optimization live at different levels. Confusing which level does what is the most common mistake I see practitioners make.</p><p><strong>Graph level: hardware-independent algebraic cleanup</strong></p><p>The top of the stack is the graph as you exported it — nodes are ops like matmul, add, softmax, and edges are tensors with symbolic shapes. Most of the classic compiler optimizations happen here, and <strong>almost none of them care about the hardware</strong>:</p><ul><li><strong>Constant folding</strong>: if two inputs to an op are constants at compile time, evaluate the op. Sounds trivial until you realize that real exported graphs from PyTorch or TensorFlow are full of reshape, transpose, cast, and broadcast of weight constants that appear as runtime ops in the graph but can be folded away. A typical export goes from ~3000 ops to ~300 ops after aggressive constant folding.</li><li><strong>Common subexpression elimination and dead code elimination</strong>: the usual.</li><li><strong>Algebraic simplification</strong>: x * 1 → x, x + 0 → x, transpose(transpose(x)) → x, cast(cast(x, f16), f32) → x when safe.</li><li><strong>Layout transforms</strong>: NCHW vs NHWC vs blocked formats. This sits at the graph level because it’s a global decision — you want consistent layouts across the graph to avoid paying for transposes between ops — but it’s informed by hardware preferences, so it’s the bridge to the next level.</li></ul><p>These optimizations are portable. The StableHLO folder in XLA and the MLIR --canonicalize pass do essentially the same work — they just express it in different IRs.</p><h3>IR level: where fusion happens</h3><p>Below the graph level is a representation that still talks about tensors but has started committing to execution structure. In MLIR terms, this is roughly where linalg lives. In XLA, it&#39;s HLO with fusion decisions attached. In TVM, it&#39;s Relax lowering toward TIR. This is where the single most impactful class of optimization — <strong>operator fusion</strong> — does its work.</p><p>Why fusion matters is pure memory-bandwidth arithmetic. A chain like bias_add → gelu → residual_add has an arithmetic intensity of roughly one FLOP per byte of memory traffic. Every modern accelerator — GPU, ANE, TPU, CPU — can do thousands of FLOPs in the time it takes to read a byte from DRAM. That means the unfused version is bandwidth-bound, and fusing the chain into one loop that reads each input once, computes everything in registers, and writes once is not just faster — it&#39;s often three or four times faster on the exact same hardware.</p><p>Here’s what that looks like concretely:</p><pre>// Unfused: three passes over memory.<br>void unfused(const float* x, const float* b, const float* r,<br>             float* y, int N) {<br>    std::vector&lt;float&gt; t(N);<br>    for (int i = 0; i &lt; N; ++i) t[i] = x[i] + b[i];<br>    for (int i = 0; i &lt; N; ++i) {<br>        float v = t[i];<br>        float u = 0.7978845608f * (v + 0.044715f * v * v * v);<br>        t[i]   = 0.5f * v * (1.0f + std::tanh(u));<br>    }<br>    for (int i = 0; i &lt; N; ++i) y[i] = t[i] + r[i];<br>}<br><br>// Fused: single pass, intermediates live in registers.<br>void fused(const float* x, const float* b, const float* r,<br>           float* y, int N) {<br>    constexpr float kA = 0.7978845608f; // sqrt(2/pi)<br>    constexpr float kB = 0.044715f;<br>    #pragma omp parallel for simd<br>    for (int i = 0; i &lt; N; ++i) {<br>        float v = x[i] + b[i];<br>        float u = kA * (v + kB * v * v * v);<br>        float g = 0.5f * v * (1.0f + std::tanh(u));<br>        y[i]    = g + r[i];<br>    }<br>}</pre><p>The unfused version does three full passes over N elements, so roughly 6N bytes of DRAM traffic for fp32. The fused version does three reads and one write — 16N bytes — but in a single pass, which means cache behavior is dramatically better and parallelization is trivial. In practice the fused version is limited only by DRAM bandwidth; the math is free.</p><p>Every serious ML compiler does this. XLA has its fusion pass. TorchInductor emits fused Triton kernels by default. TensorRT fuses convolution with bias, ReLU, and often the following residual-add into a single kernel. CUTLASS generalizes this into “epilogue fusion” where arbitrary elementwise ops get folded into the tail of a GEMM. The interesting differences between compilers are mostly about <strong>how far</strong> they’ll push fusion (can you fuse across a reduction? across a reshape? across a matmul?) rather than whether they do it at all.</p><p><strong>Memory planning</strong> also lives at this level. Once the IR is in a form where you can see which tensors are live simultaneously, you can solve a bin-packing problem over tensor lifetimes and produce a <strong>single static arena</strong> with precomputed offsets. This is what gives AOT compilers their deterministic memory footprint and is a huge reason TensorRT, CoreML, and AOTInductor binaries are so predictable.</p><h3>Kernel level: hardware-specific everything</h3><p>Below the IR level is where you finally commit to a specific target. Tiling, vectorization, shared memory staging, register allocation — all the dirty work of making a loop run fast on actual silicon. This is where hardware-specific knowledge explodes:</p><ul><li>On NVIDIA GPUs, you care about warp size (32), shared memory banks (32, with conflict padding), register file size (65536 32-bit registers per SM), Tensor Core shapes (16×16×16 for fp16 on Ampere, 64×64×16 WGMMA on Hopper).</li><li>On CPUs, you care about SIMD width (AVX-512 = 16 fp32 lanes, NEON = 4, SVE variable, AMX = 32×32 fp16 tiles on Apple Silicon), cache hierarchy (L1D ~32–48 KB, L2 ~256 KB — 1 MB, L3 shared), prefetch distance.</li><li>On the ANE, you don’t care about any of those things, because Apple doesn’t tell you. You care about whether your op is in MIL’s supported set and whether your weights are in a format the ANE likes (typically palettized with per-grouped-channel lookup tables).</li></ul><p>The canonical kernel-level optimization is <strong>tiled/blocked matrix multiplication</strong>:</p><pre>// Tiled GEMM: three-level loop nest for cache-friendly access.<br>// BM, BN, BK sized to fit panels of A and B in L2.<br>// MR, NR define a register tile that stays in registers through the K loop.<br>constexpr int BM = 128, BN = 128, BK = 256;<br>constexpr int MR = 8,   NR = 8;<br><br>void sgemm_tiled(int M, int N, int K,<br>                 const float* A, const float* B, float* C) {<br>    for (int jj = 0; jj &lt; N; jj += BN)<br>    for (int kk = 0; kk &lt; K; kk += BK)<br>    for (int ii = 0; ii &lt; M; ii += BM) {<br>        for (int i = ii; i &lt; ii + BM; i += MR)<br>        for (int j = jj; j &lt; jj + BN; j += NR) {<br>            float c[MR][NR] = {{0}};<br>            for (int k = kk; k &lt; kk + BK; ++k) {<br>                float a[MR];<br>                for (int ir = 0; ir &lt; MR; ++ir) a[ir] = A[(i+ir)*K + k];<br>                float b[NR];<br>                for (int jr = 0; jr &lt; NR; ++jr) b[jr] = B[k*N + (j+jr)];<br>                for (int ir = 0; ir &lt; MR; ++ir)<br>                    for (int jr = 0; jr &lt; NR; ++jr)<br>                        c[ir][jr] += a[ir] * b[jr];<br>            }<br>            for (int ir = 0; ir &lt; MR; ++ir)<br>                for (int jr = 0; jr &lt; NR; ++jr)<br>                    C[(i+ir)*N + (j+jr)] += c[ir][jr];<br>        }<br>    }<br>}</pre><p>The specific constants — BM, BN, BK, MR, NR — are the knobs (aka tunable parameters). Their right values depend on the cache sizes and register file of your specific CPU (or GPU’s SM). TVM’s whole autotuning story, Triton’s @autotune decorator, and TensorRT&#39;s builder-phase kernel search are all, at their core, searching over these knobs. The <em>algorithm</em> is portable; the <em>parameter choice</em> is not.</p><p>This is the level where the hardware-specific vs hardware-agnostic distinction earns its keep. CoreML punts: the MIL-level representation doesn’t commit to tile sizes, and the ANE backend (which you can’t see) picks them at load time based on the actual hardware. TensorRT commits: the engine has specific kernels picked for a specific GPU baked in. Both are reasonable — they’re just different points on the build-time-vs-load-time tradeoff.</p><h3>Quantization: a case study in where things live</h3><p>Quantization is interesting because it sprawls across all three levels, and understanding where each piece lives is a useful sanity check on the framework.</p><p>The <strong>decision to quantize</strong> and which algorithm to use (symmetric vs asymmetric, per-tensor vs per-channel, PTQ vs QAT, GPTQ, AWQ, SmoothQuant) is offline — it happens before you hand the model to a compiler.</p><p>The <strong>representation of quantized ops</strong> lives at the IR level. ONNX has QLinearConv, QLinearMatMul, and QDQ (Quantize-DeQuantize) pairs. MIL has constexpr_blockwise_shift_scale and friends. The graph still looks algebraically like the original model, just with explicit quant/dequant boundaries.</p><p>The <strong>fusion of quant-dequant pairs</strong> happens at the IR-to-kernel boundary. A naive lowering of QDQ ops would dequantize to fp32, do the op, requantize — that’s 3× the memory traffic of just doing the quantized op directly. Every real compiler fuses: it recognizes the DQ → op → Q pattern and emits an integer kernel with a fused scale/zero-point rescale at the end.</p><p>The <strong>actual integer kernels</strong> are hardware-specific. On x86 you use VPDPBUSD (AVX-512 VNNI) for INT8 dot products. On ARMv8.2+ you use SDOT/UDOT. On NVIDIA Tensor Cores you use INT8 DP4A or Hopper’s FP8 tensor core paths. The compiler doesn’t care which one; the kernel library does.</p><p>Here’s what that looks like, stripping out most of the real-world complexity:</p><pre>// Per-tensor symmetric INT8 quantize: block-wise with an fp16 scale per block.<br>// Pattern used in llama.cpp Q8_0 and similar formats.<br>constexpr int QK8_0 = 32;<br>struct block_q8_0 { uint16_t d; int8_t qs[QK8_0]; };<br><br>void quantize_row_q8_0(const float* x, block_q8_0* y, int n) {<br>    const int nb = n / QK8_0;<br>    for (int i = 0; i &lt; nb; ++i) {<br>        float amax = 0.0f;<br>        for (int j = 0; j &lt; QK8_0; ++j)<br>            amax = std::max(amax, std::fabs(x[i*QK8_0 + j]));<br>        const float d  = amax / 127.0f;<br>        const float id = d ? 1.0f/d : 0.0f;<br>        y[i].d = f32_to_f16(d);<br>        for (int j = 0; j &lt; QK8_0; ++j)<br>            y[i].qs[j] = (int8_t)std::nearbyintf(x[i*QK8_0 + j] * id);<br>    }<br>}</pre><p>Block-wise quantization with a small fp16 scale per 32 weights is a cheap way to reduce quantization error compared to a single tensor-wide scale — the extra half-bit per weight buys a lot of accuracy. This is the foundation of every GGUF low-bit format used by llama.cpp and friends.</p><h3>Four ideological optimizations</h3><p>Most ML compiler optimizations are variations on themes that <strong>have existed in general-purpose compilers for decades</strong>. The interesting ones are the ideas that couldn’t have come from anywhere else — they required someone to look at a neural network workload and realize the conventional wisdom was wrong. Four stand out.</p><h3>Palettization: quantization is the wrong question</h3><p>Linear quantization assumes weights are uniformly distributed, or at least that you want to represent them as if they were. You pick a scale, you pick a zero point, you round every weight to the nearest grid point.</p><p>Real neural network weights are not uniformly distributed. They’re clustered, and they’re often heavy-tailed — a few outliers with a dense mass in the middle. Linear quantization wastes bits on the parts of the number line where there are no weights and underrepresents the dense cluster where most of the weights actually are.</p><p><strong>Palettization takes the idea seriously.</strong> Instead of a uniform grid, you run K-means on the weights and get a codebook of K centroids that cluster where the weights actually live. Each weight becomes an index into the codebook: 4-bit indices if K = 16, 3-bit if K = 8, and so on. At inference, you look up the centroid and use it directly.</p><pre>// 4-bit palettized matvec with a per-group codebook.<br>// packed_idx : uint8[O][I/2]       (two 4-bit indices per byte)<br>// lut        : fp16[O/G][16]       (one 16-entry LUT per group of G rows)<br>// Pattern mirrors CoreML&#39;s constexpr_lut_to_dense op.<br>void palettized_matvec(<br>    const uint8_t*  packed_idx,<br>    const uint16_t* lut,<br>    const float*    x,<br>    float*          y,<br>    int O, int I, int G) {<br>    for (int o = 0; o &lt; O; ++o) {<br>        const uint16_t* my_lut = lut + (o / G) * 16;<br>        float cb[16];<br>        for (int k = 0; k &lt; 16; ++k) cb[k] = f16_to_f32(my_lut[k]);<br><br>        float acc = 0.0f;<br>        const uint8_t* row = packed_idx + o * (I / 2);<br>        for (int i = 0; i &lt; I; i += 2) {<br>            uint8_t byte = row[i &gt;&gt; 1];<br>            uint8_t lo   = byte &amp; 0x0F;<br>            uint8_t hi   = (byte &gt;&gt; 4) &amp; 0x0F;<br>            acc += cb[lo] * x[i] + cb[hi] * x[i + 1];<br>        }<br>        y[o] = acc;<br>    }<br>}</pre><p>The key trick is that there’s no separate dequantization pass. The codebook lives in registers (64 bytes of fp32 for K=16), each MAC is one byte load plus an indexed gather plus an FMA, and the whole thing lowers on NEON to vqtbl1q_u8 and on CUDA to __shfl-based lookups. Because the centroids cluster where the weights actually are, 4-bit palettization typically beats 4-bit linear quantization on perplexity by a measurable margin.</p><p>Apple bet hard on this. The ANE loves lookup tables — they’re what the hardware is built around — and Apple’s on-device foundation models use 2-bit quantization-aware training combined with palettized projections, averaging roughly 3.7 bits per weight and achieving 4–6× memory reduction versus fp16 with minimal quality loss. That’s the kind of compression you need to fit a 3B-parameter model on an iPhone and run it at 30 tokens per second.</p><p>The story doesn’t end with palettization. A newer line of work — Google’s <strong>TurboQuant</strong>, alongside QuaRot and SpinQuant from the broader research community — attacks quantization from a different angle: instead of picking a smarter grid, reshape the data to fit the grid you already have. The insight is that quantization error is dominated by outliers and non-Gaussian distributions. Apply a random orthogonal rotation (a structured Hadamard-like transform that’s cheap to compute) to weights and activations, and the distributions become near-Gaussian — at which point a plain uniform quantizer suddenly works. TurboQuant pushes this down to 3–4 bits on both weights and the KV cache with near-information-theoretic optimality. The ideological move is the same as the other three: the problem wasn’t the quantizer, it was the coordinate system you were quantizing in.</p><h3>FlashAttention: attention is memory-bound, actually</h3><p>Everyone “knew” that attention is quadratic in sequence length. What people didn’t focus on is that it’s quadratic in the <em>wrong thing</em>: the problem isn’t the FLOPs, it’s the memory. Standard attention materializes the N×N attention matrix P = softmax(QKᵀ/√d) to HBM and then reads it back for P·V. On modern GPUs, that round-trip through HBM is the entire runtime. The FLOPs are almost free.</p><p><a href="https://arxiv.org/pdf/2205.14135">FlashAttention’s</a> insight was that you can compute softmax online and tile along K/V, never materializing the full attention matrix. The trick is the online softmax recurrence:</p><pre>m_new = max(m_old, m_tile)<br>l_new = exp(m_old - m_new) * l_old + exp(m_tile - m_new) * l_tile<br>O_new = (l_old * exp(m_old - m_new) * O_old + exp(m_tile - m_new) * O_tile) / l_new</pre><p>This updates the running max m, the running denominator l, and the running output O as you scan through K/V tiles, without ever materializing the full P. Memory traffic drops from O(N²) to O(N·d), which at sequence length 4K is a 10-20× reduction. The answer is bit-exact identical to regular attention.</p><p>The impact isn’t just the speedup (2–4× on FlashAttention-1, up to 75% of H100 peak on FlashAttention-3). It’s that FlashAttention taught an entire generation of practitioners that memory hierarchy is the real problem, not FLOPs. Everything downstream — flash decoding, paged attention, speculative decoding — comes from that worldview shift.</p><h3>PagedAttention: the KV cache is just virtual memory</h3><p>Once people started running LLMs in production, a new bottleneck emerged: the KV cache. Every generated token requires the attention keys and values for every preceding token, so the cache grows linearly with context length, and at batch size 32 with 32K context, it’s easily tens of gigabytes.</p><p>The naive approach — allocate a contiguous buffer per sequence, sized to the maximum possible length — wastes 60–80% of the memory you allocate. Most sequences don’t hit the max. Sequences finish at different times, leaving fragmented holes. You can’t meaningfully share memory across sequences even when their prefixes are identical.</p><p>vLLM’s PagedAttention solved this by noticing that the KV cache looks exactly like process memory in an operating system. You don’t allocate contiguous physical memory for a process; you allocate pages and maintain a page table mapping virtual addresses to physical pages. Do the same for KV cache: chop it into fixed-size blocks (typically 16 tokens), maintain a block table per sequence, and let the attention kernel follow the indirection.</p><p>The payoff is enormous: KV cache waste drops from 60–80% to under 4%, batch sizes go up 2–4×, throughput follows. Better still, when two sequences share a prefix — say, a common system prompt — their block tables can <em>point to the same blocks</em>. Prefix caching becomes a reference-counting exercise rather than a copy. TensorRT-LLM, SGLang, and basically every modern LLM serving stack has adopted this pattern.</p><p>What’s interesting about all four optimizations is how much they come from taking a fresh look at what the workload actually is. FlashAttention noticed the problem was memory, not FLOPs. PagedAttention noticed the KV cache is a memory allocator problem. Palettization noticed that weight distributions aren’t uniform. TurboQuant noticed the coordinate system was the problem, not the quantizer. None of these are “compiler optimizations” in the traditional sense — they’re algorithmic reformulations that a compiler can then exploit.</p><h3>What to pick, when</h3><p>If you’ve read this far, you probably want actionable guidance. Here’s the short version:</p><ul><li><strong>Shipping on Apple hardware?</strong> CoreML, with coremltools for the conversion. The ecosystem integration is unmatched, and the compression tooling (palettization, joint pruning+quantization) is the best in the industry for on-device.</li><li><strong>Serving on NVIDIA GPUs at scale?</strong> TensorRT-LLM for LLMs, TensorRT for everything else. Plan for per-generation engine rebuilds.</li><li><strong>Training or research workload, especially distributed?</strong> JAX + XLA on TPU remains the gold standard; PyTorch with torch.compile is catching up fast and is the pragmatic default for most teams.</li><li><strong>Deploying to a wide matrix of hardware without vendor lock-in?</strong> ONNX Runtime with appropriate execution providers for the lowest-friction path; IREE or TVM/MLC if you’re willing to invest.</li><li><strong>Running LLMs on the edge, especially on hardware without a blessed vendor stack?</strong> MLC-LLM is remarkably portable; llama.cpp is the hobbyist default and increasingly production-viable.</li><li><strong>Writing custom GPU kernels?</strong> Triton first, CUDA only when Triton’s block-level abstraction genuinely isn’t enough.</li></ul><p>The larger takeaway is that there is no “best ML compiler,” only compilers that have made different bets about what matters. The hardware-agnostic stacks are betting that deployment flexibility outweighs peak performance. The hardware-specific stacks are betting that the last 20% of performance is worth the operational complexity. Both bets are reasonable. The trick is knowing which one you’re making.</p><p>And the ideological optimizations — palettization, FlashAttention, PagedAttention, TurboQuant — are a good reminder that the biggest wins still come from rethinking what the problem actually is, not from grinding harder on the problem as stated. If your model is slow, the answer is more likely to be “we’re computing the wrong thing” than “we need a better kernel.”</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=63d48ce438b0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Compile-Time Polymorphism: A Comparison of C++ and Rust Metaprogramming]]></title>
            <link>https://medium.com/@martin00001313/compile-time-polymorphism-a-comparison-of-c-and-rust-metaprogramming-7be921cd01f8?source=rss-4a528c57ee66------2</link>
            <guid isPermaLink="false">https://medium.com/p/7be921cd01f8</guid>
            <category><![CDATA[rust-programming-language]]></category>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[cpp]]></category>
            <category><![CDATA[rust]]></category>
            <dc:creator><![CDATA[Martin Ayvazyan]]></dc:creator>
            <pubDate>Sun, 04 May 2025 03:06:45 GMT</pubDate>
            <atom:updated>2025-05-04T03:06:45.318Z</atom:updated>
            <content:encoded><![CDATA[<h4>Introduction: The Power of Compile-Time Computation</h4><p>Modern systems programming languages - C++ and Rust place a premium on performance and abstraction. Metaprogramming - the ability to write code that operates on code, is a cornerstone technology which enables simultaneously achieve both goals. By performing computations, generating code, and enforcing constraints during compilation, metaprogramming allows for the creation of highly optimized, yet expressive and safe software.</p><p>C++ has a long history of metaprogramming through its powerful template system — the backbone of the STL — which has steadily evolved into features like constexpr evaluation and C++20 Concepts.<br>Rust, while younger, was built for advanced metaprogramming from day one, pairing a sophisticated macro system for code generation with a <em>trait</em> system that favors <strong>static (compile‑time) polymorphism</strong>. Generic functions are monomorphised into concrete versions, giving “zero‑cost” abstractions; when true runtime flexibility is required you can still opt‑in to dynamic dispatch via dyn Trait, but the language and ecosystem encourage the static route in most cases for predictability, optimization, and safety.</p><p>The article focuses on a comparison of advanced metaprogramming techniques in C++ and Rust. It examines core concepts of compile-time polymorphism (C++ Concepts vs. Rust Traits), handling of variadic arguments, the nuances of value semantics (move, forward, copy), folding operations, and the role of inlining. Additionally, it underscores metaprogramming’s crucial role in enabling zero-cost abstractions, citing examples from the standard libraries of both languages.</p><h4>Foundations: Metaprogramming Paradigms</h4><p>Metaprogramming shifts computation and code generation to compile time, eliminating runtime overhead — but at the cost of greater compile‑time complexity and longer build times. Once the generated code “unfolds,” readability and mental tracing can become harder. Both C++ and Rust offer distinct tools for navigating this trade‑off, reflecting their differing design philosophies.</p><h4>C++: Templates and constexpr</h4><p>C++ provides two primary forms for compile-time programming, aside from C-style macros:</p><ol><li><strong>Template Metaprogramming (TMP):</strong> This technique leverages the C++ template system to perform computations during compilation. Templates, originally designed for generic programming , can be instantiated by the compiler to generate temporary source code, constants, data structures, and even entire functions based on template parameters (which can be types, non-type values like integers, or other templates). TMP operates in a purely functional style; it lacks mutable state at compile time, relying on recursion (instantiating templates recursively) instead of loops, and template specialization instead of conditional statements. TMP is <strong>Turing-complete</strong>, meaning <strong>any computable function can theoretically be implemented using templates</strong>. A classic example is compile-time factorial calculation using recursive template instantiation and specialization:</li></ol><pre>#include &lt;iostream&gt;<br><br>// Primary template definition<br>template &lt;unsigned N&gt;<br>struct factorial_t {<br>    static constexpr unsigned value = N * factorial_t&lt;N - 1&gt;::value; // Recursive instantiation<br>};<br><br>// Specialization to define tile and terminate recursion<br>template &lt;&gt;<br>struct factorial_t&lt;0&gt; {<br>    static constexpr unsigned value = 1;<br>};<br><br>int main() {<br>    // Factorial::value is computed at compile time<br>    constexpr unsigned fct_5 = factorial_t&lt;5&gt;::value;<br>    std::cout &lt;&lt; &quot;Factorial::value = &quot; &lt;&lt; fct_5 &lt;&lt; std::endl; // Output: 5!=120<br>    return 0;<br>}</pre><blockquote>Note: All major C++ compilers cap the maximum level of recursive template instantiation to keep compile‑time resource usage reasonable. If that limit is exceeded, like for factorial_t&lt;1000&gt;, the build fails with an error such as fatal error: template instantiation depth exceeds maximum of 900 (GCC). Typical defaults are: GCC ≈ 900, Clang ≈ 1024, which is tunable, — e.g. -ftemplate-depth=&lt;N&gt; for GCC/Clang o.</blockquote><p>2. <strong>constexpr Functions and Variables:</strong> Introduced in C++11 and significantly enhanced in C++14 and later standards, constexpr allows functions and variables to be evaluated at compile time under certain conditions. Unlike TMP&#39;s often arcane syntax, constexpr functions use standard C++ syntax, including loops, conditionals, and mutable local variables (since C++14). However, a constexpr function <em>can</em> also execute at runtime; its compile-time evaluation is only guaranteed when its result is required in a compile-time context (e.g., initializing a constexpr variable, static assertion) and its arguments are compile-time constants.</p><pre>// Simple factorial that *can* be evaluated at compile time<br>constexpr unsigned factorial_cxpr(unsigned n) {<br>    if (n == 0) {<br>        return 1;<br>    }<br>    unsigned result = 1;<br>    for (unsigned i = 1; i &lt;= n; ++i) {<br>        result *= i;<br>    }<br>    return result;<br>}<br><br>int main() {<br>    // ---- Compile‑time call ----<br>    // The argument `4` is a constant expression, so the compiler *must*<br>    // fold this call; `fact4` is literally the value 24 baked into the binary.<br>    constexpr unsigned fact4 = factorial_cxpr(4);<br>    std::cout &lt;&lt; &quot;factorial_cxpr(4) = &quot; &lt;&lt; fact4 &lt;&lt; &#39;\n&#39;; // -&gt; 24<br><br>    // ---- Run‑time (or optimised) call ----<br>    int runtime_val = 5; // NOT a constant expression - possibly evaluatted at runtime<br>    // This call must be generated as normal code, but an optimiser may still<br>    // fold it if it can prove that `runtime_val` is 5 at compile time.<br>    unsigned fact_runtime = factorial_cxpr(runtime_val);<br>    std::cout &lt;&lt; &quot;factorial_cxpr(runtime_val) = &quot;<br>              &lt;&lt; fact_runtime &lt;&lt; &#39;\n&#39;;                     // 120<br><br>    return 0;<br>}</pre><p>While constexpr offers better readability and can handle types like floating-point numbers that are problematic for TMP , TMP remains essential for computations involving types themselves (type manipulation). Often, modern C++ metaprogramming involves a combination of both techniques.</p><h3>Rust: Generics, Traits, and Macros</h3><p>Rust’s metaprogramming facilities are centered around generics/traits for polymorphism and macros for code generation:</p><ol><li><strong>Generics and Traits:</strong> Rust’s generics allow writing code that operates over multiple types, similar to C++ templates in this regard. Traits define shared behavior/interfaces that generic types can be constrained by (trait bounds). When generic code is used with concrete types, the Rust compiler performs <em>monomorphization</em>, generating specialized versions of the code for each specific type. This process facilitates compile-time polymorphism, ensuring that calls to trait methods on generic types are resolved statically, often allowing for inlining and performance equivalent to non-generic code. Traits define required function signatures, associated types, and constants.</li><li><strong>Macros:</strong> Rust features powerful macro systems for writing code that writes other code. Unlike C/C++ preprocessor macros which perform simple text substitution, Rust macros operate on token streams or the Abstract Syntax Tree (AST), <strong>making them syntax-aware and hygienic</strong> (avoiding accidental variable capture). There are two main types</li></ol><ul><li><strong>Declarative Macros (</strong><strong>macro_rules!):</strong> These work like match expressions on the structure of the input code tokens, replacing matched patterns with generated code. They can handle variable numbers of arguments - a key capability distinguishing them from functions. The vec! macro is a common example.</li><li><strong>Procedural Macros:</strong> These are more function-like, taking a TokenStream as input and producing a TokenStream as output. They come in three kinds: custom #[derive] macros (generating trait implementations), attribute-like macros (applying custom attributes), and function-like macros (invoked like functions). Procedural macros offer greater flexibility for complex code generation but must reside in special &quot;proc-macro&quot; crates.</li></ul><pre>// Minimal “command” generator.<br>// One identifier list → real functions *and* a match‑based dispatcher.<br>macro_rules! cmds {<br>    ( $( $name:ident ),+ $(,)? ) =&gt; {<br>        // Generate a function for every name.<br>        $( #[inline] fn $name() { println!(&quot;{} executed&quot;, stringify!($name)); } )+<br><br>        // Runtime entry‑point: run(&quot;start&quot;)<br>        pub fn run(cmd: &amp;str) {<br>            match cmd {<br>                $( stringify!($name) =&gt; $name(), )+<br>                _ =&gt; eprintln!(&quot;unknown cmd: {}&quot;, cmd),<br>            }<br>        }<br>    };<br>}<br><br>// One line defines everything.<br>cmds!(start, stop, status);<br><br>fn main() {<br>    start();       // compile-time call - start executed<br>    run(&quot;stop&quot;);   // runtime dispatch - stop executed<br>    run(&quot;cancel&quot;);    // runtime unknown command - unknown cmd: cancel<br>}</pre><p>Rust macros are primarily for syntactic abstraction and code generation, while generics and traits handle polymorphism and type constraints. Compile-time function execution (const fn) exists but is currently more limited than C++ constexpr, though rapidly evolving. While the comparison w/ constexpr holds true now, it’s one area that might change significantly over the next few years.</p><h4>Compile-Time Polymorphism: C++ Concepts vs. Rust Traits</h4><p>Both C++20 Concepts and Rust Traits serve to constrain generic parameters, enabling compile-time polymorphism and improving error messages compared to unconstrained C++ templates. However, they differ fundamentally in their approach and capabilities.</p><p>Core Differences: <strong>Structural vs. Nominal Typing</strong></p><p>The most significant distinction lies in their typing systems :</p><ul><li><strong>C++ Concepts (Structural Typing):</strong> A type <strong><em>implicitly</em></strong> models a concept the moment it <em>fits the mold</em>: every required expression is well‑formed and any mandated aliases or nested types exist. No implements keyword is needed—the compiler simply instantiates the template and, if everything lines up, the type is accepted. It’s compile‑time “<em>penguin typing</em>”: if it swims like a penguin and squawks like a penguin, the compiler treats it as one—without asking the bird to raise its flipper and swear an oath..<br>- <em>Advantage:</em> Flexibility. Any type — even from third‑party or legacy code — can satisfy a concept with <strong><em>zero</em></strong> changes. That makes cross‑library interoperability painless.<br>- <em>Disadvantage:</em> Accidental matches: A type might accidentally satisfy a concept <em>without truly matching its intended semantics</em>. Furthermore, code within a concept-constrained template can legally call methods or use operations on the type that are <em>not</em> part of the concept definition, potentially leading to errors only discovered during instantiation if the specific type used lacks that operation. Concepts already give clearer diagnostics and cleaner overload sets than <a href="https://en.cppreference.com/w/cpp/language/sfinae">SFINAE</a>. Ongoing standard‑library work (e.g., proposals for stronger “equational” and partial‑instantiation checks) aims to let the compiler verify more of a template’s body up front, shrinking the gap between what the concept promises and what the template actually uses.</li><li><strong>Rust Traits (Nominal Typing): </strong>A type implements a trait <em>explicitly</em> using an impl Trait for Type block. This declaration signifies intent and is mandatory for the type to be used where that trait is required.<br>- <em>Advantage:</em> Clarity and Safety. The explicit impl makes intent clear. Within generic code bounded by a trait (&lt;T: MyTrait&gt;), only the methods, associated types, and constants defined by MyTrait (and its supertraits) can be used on values of type T. This allows the generic function itself to be type-checked once at definition time, guaranteeing that any type satisfying the bound will work.<br>- <em>Disadvantage:</em> Less flexibility for combining unrelated libraries. A type from library A cannot be used with a trait from library B unless one of the crates explicitly provides the impl block (subject to coherence rules, discussed later).</li></ul><h4>Defining Requirements</h4><p>The way requirements are specified also differs:</p><ul><li><strong>C++ Concepts:</strong> a constraint is any compile‑time Boolean expression — ranging from simple type traits or other concepts to a full <strong>requires‑expression</strong> that validates operations, nested aliases, or conversions. When a structural check is needed, a requires‑expression supplies it; otherwise, ordinary constant expressions and logical operators (&amp;&amp;, ||, !) suffice.</li></ul><pre>#include &lt;concepts&gt;<br>#include &lt;vector&gt;<br>#include &lt;iostream&gt;<br>#include &lt;algorithm&gt;<br><br>template&lt;typename T&gt;<br>concept sortable_t = requires(T a, T b) { // to force support of ++ and &#39;&lt;&#39;<br>    { ++a } -&gt; std::same_as&lt;T&amp;&gt;;<br>    { a &lt;  b } -&gt; std::convertible_to&lt;bool&gt;;<br>};<br><br>template&lt;sortable_t T&gt;<br>void sort_and_print(std::vector&lt;T&gt;&amp; vec) {<br>    std::sort(vec.begin(), vec.end());<br>    for (const auto&amp; item : vec) std::cout &lt;&lt; item &lt;&lt; &#39; &#39;;<br>    std::cout &lt;&lt; &#39;\n&#39;;<br>}<br><br>struct custom_type {<br>    int m_val;<br>    custom_type&amp; operator++() { ++m_val; return *this; }<br>    bool operator&lt;(const custom_type&amp; rhs) const { return m_val &lt; rhs.m_val; }<br>    friend std::ostream&amp; operator&lt;&lt;(std::ostream&amp; os, const custom_type&amp; mt) {<br>        return os &lt;&lt; mt.m_val;<br>    }<br>};<br><br>int main() {<br>    std::vector&lt;custom_type&gt; v{{3},{1},{2}};<br>    sort_and_print(v);        // OK<br><br>    // std::vector&lt;std::string&gt; s{&quot;c&quot;,&quot;a&quot;,&quot;b&quot;};<br>    // sort_and_print(s);      // still fails: ++string isn&#39;t valid<br>}</pre><ul><li><strong>Rust Traits:</strong> Define requirements by listing method signatures, associated types, and associated constants that implementing types must provide. Trait bounds are purely additive (conjunctions).</li></ul><pre>use std::fmt::{self, Display};<br><br>// ---------------------------------<br>// Trait: the *contract* the function relies on<br>trait HasArea {<br>    fn area(&amp;self) -&gt; f64;<br>}<br><br>// A generic function that needs *both* HasArea and Display.<br>// Only items guaranteed by those traits (or built‑ins) can be used inside.<br>fn print_area&lt;T: HasArea + Display&gt;(shape: &amp;T) {<br>    println!(&quot;This shape - {} and its area - {}&quot;, shape, shape.area());<br>    // shape.some_other_method();           // &lt;— would not compile:as not in HasArea/Display<br>}<br><br><br>struct Circle {<br>    radius: f64,<br>}<br><br>// A type/Circle that explicitly fulfils both contracts<br>impl HasArea for Circle {<br>    fn area(&amp;self) -&gt; f64 {<br>        std::f64::consts::PI * self.radius * self.radius<br>    }<br>}<br><br>impl Display for Circle {<br>    fn fmt(&amp;self, f: &amp;mut fmt::Formatter&lt;&#39;_&gt;) -&gt; fmt::Result {<br>        write!(f, &quot;Circle(r={})&quot;, self.radius)<br>    }<br>}<br><br>// ---------------------------------<br>// A type with a *similar* method, but no trait impls<br>struct Square {<br>    side: f64,<br>}<br><br>// Inherent implementation: method exists but **no** trait is implemented<br>impl Square {<br>    fn area(&amp;self) -&gt; f64 {<br>        self.side * self.side<br>    }<br>}<br><br>fn main() {<br>    let c = Circle { radius: 1.0 };<br>    print_area(&amp;c);                  // OK — Circle meets both trait bounds<br><br>    let s = Square { side: 2.0 };<br>    // print_area(&amp;s);               // ERROR — Square lacks HasArea *and* Display<br>                                     // Even though `area()` exists, traits must be<br>                                     // *explicitly* implemented to satisfy bounds.<br>}</pre><h3>Overload Resolution and Specificity</h3><h4>C++ Concepts — Structural, Implicit, and Subsumable</h4><ul><li><strong>Automatic modeling</strong> — A type satisfies a concept whenever all required expressions are well‑formed; no implementskeyword is necessary.</li><li><strong>Constrained overloads</strong> — When several template overloads exist, the compiler orders them by <em>subsumption</em>: a definition whose constraints are “strictly stronger” wins. Ambiguity is reported only when two candidates neither subsume nor are subsumed by the other.</li><li><strong>Beyond SFINAE</strong> — Concepts replace most enable_if boilerplate, produce clearer diagnostics, and let template bodies be checked earlier, though a few edge cases still fall back to traditional SFINAE.</li></ul><p><em>Take‑away:</em> concepts offer flexible, <em>structural</em> matching — ideal for integration of unfamiliar third‑party types into an existing template interface.</p><h4>Rust Traits — Nominal, Explicit, and Coherent</h4><ul><li><strong>No ad‑hoc overloading</strong> — Free functions cannot be overloaded by signature. Variant behavior is achieved with distinct traits whose methods share a name, or by defining different types.</li><li><strong>Specialization (still unstable)</strong> — Overlapping impl blocks with one “more specific” than another remains experimental (min_specialization); production code<strong> cannot yet</strong> rely on them.</li><li><strong>Disambiguation via UFCS</strong> — When multiple traits expose the same method name, <em>Universal Function Call Syntax</em>(&lt;Type as Trait&gt;::method(&amp;value)) selects the desired one.</li><li><strong>Coherence &amp; safety</strong> — The <em>orphan rule</em> and explicit impl declarations guarantee that trait implementations are unique and visible at compile time, allowing generic code to be type‑checked independently of its instantiations.</li></ul><p><em>Take‑away:</em> traits provide stronger compile‑time guarantees and clearer intent, at the cost of extra boilerplate and some restrictions on overlapping implementations.</p><h3>Handling Variadic Arguments</h3><p>Both C++ &amp; Rust provide mechanisms to handle functions or constructs that accept a variable number of arguments, crucial for tasks like formatting strings, creating containers, or applying operations across parameter packs.</p><p><strong>C++: Variadic Templates</strong></p><p>C++11 introduced variadic templates, allowing templates (functions, classes) to accept an arbitrary number of template arguments (both type and non-type).</p><ul><li><strong>Mechanism:</strong> Uses typename... or auto... (since C++17) to declare a template parameter pack and Args... to declare a function parameter pack.</li><li><strong>Expansion:</strong> Parameter packs are typically expanded using recursion (prior to C++17) or fold expressions (C++17 onwards). Recursion involves processing the first argument and recursively calling the template with the rest. Fold expressions provide a concise syntax for applying a binary operator over all elements in a pack.</li><li><strong>Type Safety:</strong> Variadic templates are type-safe; the compiler knows the type of each argument in the pack.</li></ul><pre>#include &lt;iostream&gt;<br>#include &lt;string&gt;<br>#include &lt;utility&gt;<br><br>// A small type that we want to distinguish between “borrowed” (lvalue)<br>// and “temporary / to‑be‑moved‑from” (rvalue).<br>struct Item {<br>    std::string m_label;<br>    explicit Item(std::string lbl) : m_label{std::move(lbl)} {}<br>};<br><br>// Two overloads to prove the difference.<br>void log(const Item&amp; it) { std::cout &lt;&lt; &quot;[borrowed]  &quot; &lt;&lt; it.m_label &lt;&lt; &#39;\n&#39;; }<br>void log(Item&amp;&amp; it) { std::cout &lt;&lt; &quot;[moved] &quot; &lt;&lt; it.m_label &lt;&lt; &#39;\n&#39;; }<br><br><br>// 1. Perfect‑forwarding variadic wrapper ➜ keeps rvalues as rvalues<br>template&lt;typename... Args&gt;<br>void process_items(Args&amp;&amp;... args) {<br>    ( log(std::forward&lt;Args&gt;(args)), ... ); // forward arguments<br>}<br><br>// 2. Naïve wrapper (no std::forward) ➜ **loses** value category<br>template&lt;typename... Args&gt;<br>void process_items_naive(Args&amp;&amp;... args) {<br>    ( log(args), ... ); // each `args` is now an lvalue<br>}<br><br><br>int main() {<br>    Item a(&quot;A&quot;);<br><br>    std::cout &lt;&lt; &quot;=== perfect‑forwarding version ===\n&quot;;<br>    process_items(a, Item(&quot;B&quot;), Item(&quot;C&quot;)); // <br><br>    std::cout &lt;&lt; &quot;\n=== naive version (no std::forward) ===\n&quot;;<br>    process_items_naive(a, Item(&quot;D&quot;), Item(&quot;E&quot;));<br>}</pre><p><strong>Rust: Macros</strong></p><p>Rust achieves variadic-like behavior primarily through declarative macros (macro_rules!).</p><ul><li><strong>Mechanism</strong> − Macros match tokens, not types. Repetition ($(…)*, $(…)+) lets them capture any number of arguments.</li><li><strong>Expansion</strong> − Matched tokens are substituted into a user‑defined template, producing regular Rust code.</li><li><strong>Flexibility</strong> − Because matching happens on raw syntax, a single macro can accept mixed types, key–value pairs, etc., before the compiler even looks at types.</li><li><strong>Limitation</strong> − Type checking occurs <em>after</em> expansion, so the macro itself has no compile‑time view of types — unlike C++’s variadic templates, which are integrated with the type system from the start.</li></ul><pre>// Tiny “variadic” helpers built with `macro_rules!`<br><br>macro_rules! print_variadic {<br>    // accept 0‑N comma‑separated expressions, optional trailing comma<br>    ( $( $arg:expr ),* $(,)? ) =&gt; {{<br>        $( print!(&quot;{} &quot;, $arg); )*<br>        println!();<br>    }};<br>}<br><br>macro_rules! sum {<br>    ( $( $arg:expr ),* $(,)? ) =&gt; {<br>        0 $( + $arg )*<br>    };<br>}<br><br>fn main() {<br>    print_variadic!(1, &quot;hello&quot;, 3.14, 2.718);   // → 1 hello 3.14<br>    let total = sum!(1, 2, 3, 4, 5);     // → 15<br>    println!(&quot;Sum: {total}&quot;);<br>}</pre><p>C++ variadic templates are tightly integrated with the type system, ideal for generic programming over types. Rust macros provide powerful syntactic abstraction, suitable for creating domain-specific languages or simplifying repetitive code patterns involving variable arguments, operating before type checking is complete.</p><h3>Folding Operations</h3><p>Folding is a common pattern in functional programming and generic algorithms, involving applying a binary operation cumulatively to a sequence of elements.</p><p><strong>C++17 Fold Expressions</strong></p><p>C++17 significantly simplified operations on parameter packs using fold expressions. They provide a concise syntax for expanding a parameter pack with a binary operator.</p><ul><li><strong>Syntax:</strong> Uses ... along with a binary operator (e.g., +, *, &amp;&amp;, ||, ,). Can be unary (applied between elements) or binary (with an initial value) — left/right fold respectively.</li></ul><pre>(pack op ...)          // unary right fold   →  p1 op (p2 op … op pn)<br>(... op pack)          // unary left  fold   → ((p1 op p2) op … op pn)<br><br>(pack op ... op init)  // binary right fold →  p1 op (p2 op … op (pn op init))<br>(init op ... op pack)  // binary left  fold → (((init op p1) op p2) op … op pn)</pre><pre>#include &lt;iostream&gt;<br>#include &lt;vector&gt;<br><br>template&lt;typename... Args&gt;<br>auto sum(Args... args) {<br>    return (0 +... + args); // Binary left fold<br>}<br><br>template&lt;typename... Args&gt;<br>void print_all(Args... args) {<br>    ( (std::cout &lt;&lt; args &lt;&lt; &quot; &quot;),... ); //Unary right fold over comma operator<br>    std::cout &lt;&lt; std::endl;<br>}<br><br>template&lt;typename T, typename... Args&gt;<br>void push_back_all(std::vector&lt;T&gt;&amp; vec, Args&amp;&amp;... args) {<br>    // Unary right fold over comma operator, using perfect forwarding<br>    ( vec.push_back(std::forward&lt;Args&gt;(args)),... );<br>}<br><br>int main() {<br>    std::cout &lt;&lt; sum(1, 2, 3, 4.5) &lt;&lt; std::endl; // 10.5<br><br>    print_all(10, &quot;twenty&quot;, 30.0); // 10 twenty 30<br><br>    std::vector&lt;int&gt; my_vec;<br>    push_back_all(my_vec, 100, 200, 300);<br>    for(int v : my_vec) std::cout &lt;&lt; v &lt;&lt; &quot; &quot;; // Output: Vector: 100 200 300<br>    std::cout &lt;&lt; std::endl;<br><br>    return 0;<br>}</pre><p><strong>Rust: Iterator Adapters and Macros</strong></p><p>Rust doesn’t have a direct language equivalent to C++ fold expressions applied to generic parameter packs. Similar functionality is typically achieved using:</p><ul><li><strong>Iterator Adapters:</strong> The standard library’s Iterator trait provides powerful methods like fold, sum, product, all, any, which perform folding operations on sequences. These are the idiomatic way to perform such reductions in Rust.</li><li><strong>Macros:</strong> For variadic operations not easily expressible via iterators (especially at compile time or within macros themselves), macro_rules! can be used to simulate folding by recursively processing arguments or using repetition patterns.</li></ul><pre>fn main() {<br>    let numbers = vec![1, 2, 3, 4, 5];<br><br>    // Using Iterator::fold<br>    let sum_fold = numbers.iter().fold(0, |acc, &amp;x| acc + x);<br>    println!(&quot;Fold sum: {}&quot;, sum_fold); // 15<br><br>    // Using Iterator::sum (often more convenient)<br>    let sum_iter: i32 = numbers.iter().sum();<br>    println!(&quot;Iterator sum: {}&quot;, sum_iter); // 15<br><br>    // Using Iterator::all (boolean fold with &amp;&amp;)<br>    let all_positive = numbers.iter().all(|&amp;x| x &gt; 0);<br>    println!(&quot;All positive? {}&quot;, all_positive); // true<br><br>    // Using Iterator::any (boolean fold with ||)<br>    let any_even = numbers.iter().any(|&amp;x| x % 2 == 0);<br>    println!(&quot;Any even? {}&quot;, any_even); // true<br><br>    // Macro example from earlier for comparison<br>    macro_rules! sum_macro {<br>        ( $( $x:expr ),* ) =&gt; { 0 $( + $x )* };<br>    }<br>    let macro_sum = sum_macro!(1, 2, 3, 4, 5);<br>    println!(&quot;Macro sum: {}&quot;, macro_sum); // 15<br>}</pre><p>While C++ fold expressions offer concise syntax for variadic templates, Rust’s iterator methods provide a powerful and idiomatic functional approach for runtime sequences. Macros remain an option for compile-time or syntactic folding needs.</p><h3>Inlining and Metaprogramming</h3><p>Inlining, the replacement of a function call with the function’s body, is a critical optimization for eliminating call overhead and enabling further optimizations. Metaprogramming plays a critical role in maximizing inlining opportunities.</p><h4>C++: inline, constexpr, and Templates</h4><ul><li><strong>inline Keyword:</strong> A hint to the compiler suggesting a function should be inlined. It also affects linkage, allowing a function definition to appear in multiple translation units. Modern compilers often make independent inlining decisions based on optimization levels and heuristics, regardless of the inline keyword.</li><li><strong>constexpr Functions:</strong> constexpr functions are implicitly inline. When evaluated at compile time, the concept of a &quot;call&quot; is moot; the result is computed directly. When used at runtime, their potential simplicity often makes them good candidates for inlining by the optimizer.</li><li><strong>Templates:</strong> Template instantiation generates specialized code for each type used. This monomorphization means the compiler has the concrete function body available at the call site, making templates inherently easy to inline. Calls through template code often resolve to direct function calls (static dispatch), unlike virtual function calls which typically require indirect dispatch via a vtable at runtime. This static resolution is key to enabling inlining.</li></ul><h4>Rust: #[inline] and Generics/Traits</h4><ul><li><strong>#[inline] Attribute:</strong> Similar to C++&#39;s inline, this attribute provides hints to the compiler. #[inline] suggests inlining is desirable, #[inline(always)] strongly requests it (though not absolutely guaranteed), and #[inline(never)] discourages it. Like C++, the compiler ultimately uses heuristics.</li><li><strong>Generics and Traits (Monomorphization):</strong> Rust’s generics are monomorphized at compile time. The compiler generates a specialized version of the generic function <strong><em>for each concrete type it’s instantiated with</em></strong>. This means that calls within or to generic functions usually resolve to direct, statically dispatched calls, making them prime candidates for inlining, just like C++ templates.</li></ul><p>In both languages, the metaprogramming mechanisms that generate type-specific code at compile time (Templates in C++, Generics/Traits in Rust) are fundamental enablers of effective inlining. By eliminating the need for runtime dispatch for polymorphic code, they allow the compiler to see the concrete function body at the call site and potentially replace the call with the body, leading to significant performance improvements.</p><h3>Zero-Cost Abstractions &amp; Standard Libraries</h3><p>A core promise of both C++ and Rust is the principle of “zero-cost abstractions”: high-level language features (like containers, iterators, smart pointers) should compile down to machine code that is as efficient as carefully hand-tuned low-level code. Metaprogramming is the key technology enabling this.</p><ul><li><strong>Mechanism:</strong> Compile-time polymorphism via C++ templates or Rust generics/traits allows standard library components to be written generically but instantiated specifically for the types they are used with.</li></ul><p><strong>Examples:</strong></p><ul><li><strong>C++ </strong><strong>std::vector&lt;T&gt;:</strong> A template class. When std::vector&lt;int&gt; and std::vector&lt;std::string&gt; are used, the compiler generates distinct code specialized for int and std::string respectively. Operations like element access or push_back compile down to efficient, type-specific instructions without runtime overhead for type checking or dynamic dispatch.</li><li><strong>Rust </strong><strong>Vec&lt;T&gt;:</strong> A generic struct. Similar to C++, Vec&lt;i32&gt; and Vec&lt;String&gt; result in monomorphized code. Calls to methods like push or indexing are resolved statically at compile time.</li><li><strong>Sorting (</strong><strong>std::sort, </strong><strong>slice::sort):</strong> Generic sorting algorithms in both languages operate on iterators (C++) or slices (Rust). When called with specific types (e.g., sorting a vector of integers), the comparison operations and data movements are specialized for that type during compilation, avoiding runtime indirection.</li><li><strong>Smart Pointers (</strong><strong>std::unique_ptr, </strong><strong>Box&lt;T&gt;):</strong> These abstractions manage heap memory. Through templates/generics and move semantics, they ensure resource deallocation (RAII) with minimal or no runtime overhead compared to manual memory management.</li></ul><p>Metaprogramming allows the compiler to eliminate abstraction penalties. By generating specialized code at compile time, static polymorphism can be used instead of dynamic polymorphism, and function calls (like comparison functions in sorting, or element accessors in vectors) can be readily inlined. This makes it feasible to write high-level, expressive code using standard library abstractions without sacrificing the performance expected of systems languages.</p><h4><em>In summary</em></h4><p>Selecting between C++ and Rust metaprogramming boils down to your priorities — whether you need maximum flexibility (C++’s structural templates and concepts), stronger compile-time guarantees (Rust’s nominal traits and coherence), or a particular blend of performance, safety, and team familiarity. Both ecosystems now ship zero-cost-abstraction toolkits that let you build highly efficient, reliable systems without compromise.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7be921cd01f8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Level Up Your Rust: Mastering Unsafe and Async Pitfalls]]></title>
            <link>https://medium.com/@martin00001313/level-up-your-rust-mastering-unsafe-and-async-pitfalls-f5d4f15f28c2?source=rss-4a528c57ee66------2</link>
            <guid isPermaLink="false">https://medium.com/p/f5d4f15f28c2</guid>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[cpp]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[operating-systems]]></category>
            <category><![CDATA[rust]]></category>
            <dc:creator><![CDATA[Martin Ayvazyan]]></dc:creator>
            <pubDate>Sun, 20 Apr 2025 09:51:20 GMT</pubDate>
            <atom:updated>2025-04-20T09:51:20.294Z</atom:updated>
            <content:encoded><![CDATA[<p>My appreciation for Rust stems from its clear, strong, and well-designed foundation. This core design allows the language to evolve with new features while consistently prioritizing top performance and code clarity. Seemingly small decisions, like requiring explicit opt-in for types to be Copy (they are move-by-default) and enforcing immutability by default, fundamentally contribute to code quality; explicit deviations from these defaults naturally invite scrutiny during review. Coupled with strong community support, these foundational strengths make Rust a compelling choice. I always consider it a key alternative for implementing new services and performing iterative migrations where state-of-the-art performance is paramount.</p><p>By advancing in Rust, you’ll encounter unsafe code and async complexities. This article tackles these challenges head-on, guided by practical application and engineering discipline:</p><ul><li><strong>unsafe Rust Explained:</strong> demystifying the unsafe keyword, showing it&#39;s a tool for specific needs, not a general escape hatch, and how safe abstractions are key to managing its risks.</li><li><strong>Common Async Hurdles:</strong> unpacking two notorious pitfalls in asynchronous Rust — understanding implicit future cancellation and why Pin is necessary.</li></ul><p>The goal is to build a foundational understanding, providing the mental models and best practices to confidently wield these advanced features safely and effectively.</p><h3><strong>Demystifying </strong><strong>unsafe Rust: Beyond the Keyword</strong></h3><p>The unsafe keyword often evokes apprehension, sometimes misunderstood as a switch that disables all of Rust&#39;s safety checks. This isn&#39;t quite accurate. Let&#39;s clarify what unsafe truly represents and how to use it judiciously.</p><p><strong>What </strong><strong>unsafe <em>Really</em> Means</strong></p><p>At its core, unsafe doesn&#39;t turn off the borrow checker or other fundamental safety mechanisms for code <em>within</em> the unsafeblock; references, for instance, are still checked. Instead, unsafe grants access to a small set of operations, often called &quot;superpowers,&quot; that the compiler <em>cannot</em> statically verify for memory safety.</p><p>When you write unsafe {... } or declare an unsafe fn, you are making a promise to the compiler: &quot;Trust me, I have manually verified that the operations within this block uphold Rust&#39;s memory safety invariants, even though you can&#39;t prove it yourself&quot;. The responsibility shifts from the compiler to the programmer. The goal is not to arbitrarily bypass Rust&#39;s rules but to manually enforce them using logic or knowledge external to the compiler, such as the guarantees provided by a C library&#39;s API or the specific layout of hardware memory.</p><p>Crucially, unsafe Rust is not meant as a shortcut around the borrow checker or other rules you find inconvenient. It exists because static analysis is inherently conservative; sometimes valid code is rejected to prevent accepting invalid code.unsafe provides an escape hatch for these necessary scenarios, but it demands extreme care. Misusing these superpowers can lead to Undefined Behavior (UB) – dangling pointers, data races, invalid values, etc. – although the scope of UB in Rust is more constrained than in languages like C.</p><h4>The Five Unsafe Superpowers</h4><p>The unsafe keyword unlocks exactly five additional capabilities :</p><ul><li><strong>Dereference Raw Pointers:</strong> Safe Rust uses references (&amp;T, &amp;mut T) with compile-time borrow checking. unsafe allows dereferencing raw pointers (*const T, *mut T) using the * operator. Raw pointers differ significantly from references:<br>- They can ignore borrowing rules (multiple mutable pointers to the same location are allowed)<br>- They are not guaranteed to point to valid memory; they can be null or dangle.<br>- They don’t implement automatic cleanup (like Drop). Creating raw pointers (e.g., via casting references with as) is safe, but dereferencing them requires an unsafe block because the compiler cannot guarantee their validity.</li></ul><pre>let mut num = 5;<br><br>// Creating raw pointers is safe<br>let r1 = &amp;num as *const i32;<br>let r2 = &amp;mut num as *mut i32;<br><br>// Dereferencing requires unsafe<br>unsafe {<br>    println!(&quot;r1 is: {}&quot;, *r1);<br>    println!(&quot;r2 is: {}&quot;, *r2);<br>    // *r2 = 10; // Also requires unsafe<br>}</pre><ul><li><strong>Call </strong><strong>unsafe Functions or Methods:</strong> Functions marked unsafe fn signal that they have preconditions (invariants) that the caller must uphold for the call to be safe, which Rust cannot verify. Calling such functions requires an unsafe block or being within an unsafe fn. This is common in Foreign Function Interface (FFI) scenarios.</li></ul><pre>// Declaring an external C function (inherently unsafe)<br>extern &quot;C&quot; {<br>    fn abs(input: i32) -&gt; i32;<br>}<br><br>fn main() {<br>    let number = -5;<br>    // Calling it requires an unsafe block<br>    unsafe {<br>        println!(&quot;Absolute value of {} according to C: {}&quot;, number, abs(number));<br>    }<br>}</pre><p>By calling abs within unsafe, we assert we&#39;ve met its contract (e.g., passing a valid i32).</p><ul><li><strong>Access or Modify </strong><strong>static mut Variables:</strong> Global mutable state is inherently risky due to potential data races. Accessing or modifying static mut variables requires unsafe because the compiler cannot guarantee thread safety.</li></ul><pre>static mut COUNTER: u32 = 0;<br><br>fn add_to_count(inc: u32) {<br>    // Modifying requires unsafe<br>    unsafe {<br>        COUNTER += inc;<br>    }<br>}<br><br>fn main() {<br>    add_to_count(3);<br>    // Accessing requires unsafe<br>    unsafe {<br>        println!(&quot;COUNTER: {}&quot;, COUNTER); // Prints COUNTER: 3<br>    }<br>}</pre><p>Using safe synchronization primitives like Mutex or AtomicUsize is strongly preferred over static mut.</p><ul><li><strong>Implement </strong><strong>unsafe Traits:</strong> An unsafe trait indicates that implementing the trait carries semantic requirements or invariants that the compiler cannot check. The implementer must guarantee these invariants are met. Standard examples include Send and Sync, which are automatically derived for types composed entirely of Send/Sync types but require unsafe impl if implemented manually, asserting thread safety properties.</li><li><strong>Access Fields of </strong><strong>unions:</strong> Unions allow different types to occupy the same memory location. Accessing a field requires unsafe because the compiler doesn&#39;t track which variant of the union is currently active and valid.</li></ul><h4>The Golden Rule: Safe Abstractions</h4><p>While unsafe provides necessary capabilities, its use should be minimized and carefully managed. The single most important principle for using unsafe responsibly is <strong>encapsulation within safe abstractions</strong>.</p><p>The idea is to write a small amount of unsafe code to perform the operation the compiler couldn&#39;t verify, and then wrap that code within a function or method with a <em>safe</em> interface. This safe wrapper guarantees that it will only call the internal unsafe code in a way that upholds the necessary invariants.</p><p>Consider the standard library’s slice::split_at_mut. It splits one mutable slice into two mutable slices. A safe implementation attempt might look like this:</p><pre>fn split_at_mut_safe&lt;&#39;a&gt;(slice: &amp;&#39;a mut [i32], mid: usize) -&gt; (&amp;&#39;a mut [i32], &amp;&#39;a mut [i32]) {<br>    let len = slice.len();<br>    assert!(mid &lt;= len);<br>    // ERROR: cannot borrow `*slice` as mutable more than once at a time<br>    // (&amp;mut slice[..mid], &amp;mut slice[mid..])<br>    todo!()<br>}</pre><p>This fails because the borrow checker sees two simultaneous mutable borrows of the <em>same</em> original slice, even though we know they access distinct parts. unsafe allows us to implement this correctly:</p><pre>use std::slice;<br><br>fn split_at_mut(values: &amp;mut [i32], mid: usize) -&gt; (&amp;mut [i32], &amp;mut [i32]) {<br>    let len = values.len();<br>    let ptr = values.as_mut_ptr(); // Get raw pointer to start<br><br>    assert!(mid &lt;= len);<br><br>    // SAFETY: `ptr` is valid for `len` elements. We&#39;ve checked `mid &lt;= len`.<br>    // `ptr.add(mid)` is therefore valid and points to memory owned by `values`.<br>    // The two slices created do not overlap.<br>    unsafe {<br>        (<br>            slice::from_raw_parts_mut(ptr, mid), // Create first slice<br>            slice::from_raw_parts_mut(ptr.add(mid), len - mid), // Create second slice<br>        )<br>    }<br>}<br><br>fn main() {<br>    let mut v = vec![1, 2, 3, 4, 5, 6];<br>    let (left, right) = split_at_mut(&amp;mut v, 3);<br>    assert_eq!(left, &amp;mut [1, 2, 3]);<br>    assert_eq!(right, &amp;mut [4, 5, 6]);<br>    println!(&quot;Split successful!&quot;);<br>}</pre><p>Here, split_at_mut provides a safe public API. Internally, it uses unsafe to work with raw pointers (as_mut_ptr, from_raw_parts_mut, ptr.add), bypassing the borrow checker&#39;s limitation. However, the function&#39;s logic <em>ensures</em> the safety invariants (non-overlapping slices, valid pointers) are met before calling the unsafe operations. The unsafe block is minimal and justified by comments explaining the safety reasoning.</p><p>This pattern offers significant advantages:</p><ul><li><strong>Isolation:</strong> The dangerous code is confined to a small, well-defined area.</li><li><strong>Auditability:</strong> Reviewers can focus their scrutiny on the unsafe blocks and the logic guaranteeing their safety.</li><li><strong>Usability:</strong> Consumers of the API use safe Rust, benefiting from the compiler’s guarantees without needing to worry about the underlying unsafe implementation.</li></ul><p>Much of the Rust standard library itself is built this way — safe abstractions over carefully audited unsafe code implementing fundamental types like Vec, String, and HashMap. The goal is <em>soundness</em>: ensuring that safe code interacting with the abstraction cannot trigger UB, provided the unsafe implementation itself is correct.</p><p><strong>When to <em>Actually</em> Use </strong><strong>unsafe</strong></p><p>Given the risks and the emphasis on safe abstractions, unsafe should be a tool of last resort, employed only when necessary and justifiable:</p><ol><li><strong>FFI:</strong> Interfacing with non-Rust code (like C libraries) inherently requires unsafe to call external functions or work with raw pointers passed across the boundary. Safe wrappers are essential here.</li><li><strong>Performance:</strong> In rare, performance-critical hotspots, unsafe might be used to bypass compiler checks that are known to be overly conservative <em>in that specific context</em>. Example: using get_unchecked for slice/vector access when bounds have already been manually verified. This requires careful benchmarking and justification.</li><li><strong>Low-Level Abstractions:</strong> Implementing fundamental data structures (like Vec, Rc, Mutex) often requires direct memory manipulation via raw pointers or interaction with OS primitives, necessitating unsafe.</li><li><strong>Hardware Interaction:</strong> Talking directly to hardware often involves specific memory addresses or operations not representable in safe Rust.</li></ol><p>Before reaching for unsafe, always explore safe alternatives. If unsafe is required, minimize its scope, document the safety invariants meticulously within the unsafe block, and ideally, wrap it in a safe API. Tools like Miri can help detect certain kinds of UB in unsafe code during testing.</p><p>Ultimately, unsafe is a powerful, necessary feature that enables Rust to achieve its performance goals and system-level capabilities. Its responsible use hinges on understanding that it represents a <em>contract</em> where the programmer guarantees safety, best fulfilled through careful encapsulation.</p><h3>Navigating Async Pitfalls: Cancellation and Pinning</h3><p>Rust’s async/.await provides a powerful foundation for concurrent programming, enabling highly efficient I/O-bound applications. However, its design, particularly its interaction with Rust&#39;s ownership and borrowing system, introduces complexities that can trip up even experienced developers. Many find async Rust feels like &quot;Rust in hard mode&quot; , citing friction points like Send/Sync/&#39;static bounds, runtime dependencies, and the concepts we&#39;ll focus on here: cancellation and Pin</p><p><strong>Pitfall 1: Implicit Future Cancellation</strong></p><p>A fundamental aspect of Rust’s async model that often surprises newcomers is how cancellation works. <strong>In Rust, futures can be cancelled simply by being dropped</strong>. Unlike models requiring explicit cancellation signals, dropping a future <em>before it completes</em> prevents the runtime from polling it further, effectively stopping its execution.</p><p>Crucially, this cancellation can happen <strong>at any </strong><strong>.await point</strong>. Whenever a future yields control back to the executor (.await), it creates an opportunity for it to be dropped if the surrounding context decides not to poll it again. Common scenarios include:</p><ul><li><strong>Timeouts:</strong> Using tokio::time::timeout or a tokio::select! with a timer branch.</li><li><strong>tokio::select!:</strong> When racing multiple futures, select! polls them; once one completes, the others are dropped.</li><li><strong>Task Aborting:</strong> Explicitly calling abort() on a task&#39;s JoinHandle.</li><li><strong>Dropping Containers:</strong> If a future is stored in a data structure (like a Vec or FuturesUnordered) that gets dropped, the future is dropped too.</li></ul><p>This implicit cancellation means <strong>there is no guarantee that an async task will run to completion</strong>. If a future is dropped between .await points, any work between the last yield and the drop point won&#39;t execute, and any resources held by the future (locks, file handles, temporary state) might be dropped in an inconsistent state if not handled carefully in the Dropimplementation (which itself cannot be async).</p><p><strong>Cancellation Example: </strong><strong>tokio::select! with Timeout</strong></p><p>The tokio::select! macro provides a clear illustration of cancellation. It allows concurrently polling multiple futures, executing the code block associated with the <em>first</em> future to complete.</p><pre>use tokio::time::{sleep, Duration};<br>use std::io::{Error, ErrorKind};<br><br>// Simulate a task that takes time and might fail<br>async fn long_running_task() -&gt; Result&lt;(), Error&gt; {<br>    println!(&quot;Task started...&quot;);<br>    sleep(Duration::from_secs(2)).await;<br>    println!(&quot;Task part 1 complete...&quot;);<br>    sleep(Duration::from_secs(2)).await;<br>    // Simulate potential failure<br>    // return Err(Error::new(ErrorKind::Other, &quot;Something went wrong!&quot;));<br>    println!(&quot;Task finished successfully.&quot;);<br>    Ok(())<br>}<br><br>#[tokio::main]<br>async fn main() {<br>    let task_timeout = Duration::from_secs(3);<br><br>    println!(&quot;Running task with a {}s timeout...&quot;, task_timeout.as_secs());<br><br>    tokio::select! {<br>        // Bias ensures the timeout is checked reliably, though select! has some randomness.<br>        // Tokio recommends biasing timeout/cancel branches if needed.<br>        biased;<br><br>        result = long_running_task() =&gt; {<br>            match result {<br>                Ok(()) =&gt; println!(&quot;Task completed within timeout.&quot;),<br>                Err(e) =&gt; println!(&quot;Task failed: {}&quot;, e),<br>            }<br>        }<br>        _ = sleep(task_timeout) =&gt; {<br>            // This branch completes if long_running_task() takes longer than task_timeout.<br>            // When this branch is chosen, the future running `long_running_task` is DROPPED. [20, 21]<br>            println!(&quot;Task timed out and was cancelled!&quot;);<br>            // The task might have only printed &quot;Task started...&quot; or &quot;Task part 1 complete...&quot;<br>        }<br>    }<br>}</pre><p>In this example:</p><ol><li>long_running_task simulates work with intermediate sleep calls (representing .await points).</li><li>tokio::select! concurrently polls long_running_task() and sleep(task_timeout).</li><li>If long_running_task finishes within 3 seconds, its result is processed.</li><li>If sleep(task_timeout) finishes first (after 3 seconds), its branch (_ =... =&gt;...) is executed. Critically, at this moment, the long_running_task future, which was being polled by select!, is <strong>dropped</strong>.</li><li>The output will be “Task timed out and was cancelled!”. The long_running_task likely only executed up to its first or second sleep before being dropped, potentially leaving operations incomplete or resources in an intermediate state.</li></ol><p><strong>Implication:</strong> Developers must design async code with cancellation safety in mind. Assume any .await could be the last operation executed. State management needs to be robust against sudden drops, often involving atomic operations, careful use of RAII guards (being mindful they might be held across await points ), or explicit cleanup logic (though async cleanup in Drop is not directly possible). This implicit, drop-based cancellation is a significant departure from synchronous code and requires a mental model shift.</p><h4>Pitfall 2: Why Pin Exists - The Self-Referential Future Problem</h4><p>Another major hurdle in understanding async Rust is the concept of Pin. Pin&lt;T&gt; is a wrapper type that seems arcane at first glance but is fundamental to making async/.await work safely and efficiently with Rust&#39;s ownership and borrowing rules.</p><p>The core problem Pin solves stems from <strong>self-referential structs</strong> generated by the compiler for async blocks and functions. When you write async {... }, Rust creates a state machine (effectively a struct) that implements the Futuretrait. This struct holds the state needed to resume execution across .await points, including local variables captured from the environment.</p><p>Consider a simplified conceptual example from the async book :</p><pre>// Conceptual async block<br>async {<br>    let mut buffer = [0u8; 1024];<br>    let mut stream = get_stream(); // Some async stream<br><br>    // `read_future` holds a mutable reference to `buffer`<br>    let read_future = stream.read(&amp;mut buffer);<br><br>    read_future.await; // Yield point<br><br>    println!(&quot;Read data: {:?}&quot;, &amp;buffer[..]);<br>}<br><br>// Compiler generates something like:<br>struct MyFutureStateMachine&lt;&#39;a&gt; {<br>    buffer: [u8; 1024],<br>    stream: StreamType,<br>    // State field might hold `read_future`, which contains `&amp;&#39;a mut buffer`<br>    state: StateEnum&lt;&#39;a&gt;,<br>}</pre><p>The generated MyFutureStateMachine holds both the buffer and, within its state, a future (read_future) that contains a mutable reference (&amp;mut buffer) pointing <em>back into the same struct</em>. This is a self-reference.</p><p>Now, Rust normally allows values (including structs) to be freely moved in memory (e.g., when passed to functions, returned, or stored in collections). However, <strong>if you move a self-referential struct, its internal pointers become invalid (dangling)</strong> because they still point to the <em>old</em> memory location. Dereferencing such a pointer leads to Undefined Behavior.</p><p>This is precisely the problem Pin addresses. Pin&lt;P&lt;T&gt;&gt; (where P is a pointer type like Box, &amp;mut, etc.) provides a guarantee: the value T pointed to by P <strong>will not have its memory invalidated</strong> (e.g., by moving it or deallocating without dropping) after the Pin is created, <em>unless</em> T implements the marker trait Unpin.</p><ul><li><strong>Unpin:</strong> Most standard types (i32, String, Vec&lt;T&gt; if T is Unpin, etc.) implement Unpin. This signifies they don&#39;t care about having a stable memory address; moving them is always safe. For Unpin types, Pin has no effect.</li><li><strong>!Unpin (Not Unpin):</strong> Types that <em>cannot</em> be safely moved after creation (like self-referential futures generated by async blocks that borrow across .await points) do <em>not</em> implement Unpin. Pin provides the crucial guarantee for these types.</li></ul><p>The Future::poll method requires the future to be pinned: fn poll(self: Pin&lt;&amp;mut Self&gt;,...) -&gt; Poll&lt;Self::Output&gt;. This ensures that when the runtime polls the future, the future itself is guaranteed to be at a stable memory location, making it safe for the poll implementation to work with any internal self-references.</p><h3>Practical Impact: Dealing with Pin</h3><p>For most day-to-day async Rust development, Pin operates behind the scenes. When you .await a future, the compiler and runtime handle the necessary pinning implicitly. However, you&#39;ll encounter Pin explicitly in several situations:</p><ul><li><strong>Implementing </strong><strong>Future or </strong><strong>Stream Manually:</strong> If you write your own future or stream type, you&#39;ll need to work with Pin in your poll or poll_next methods.</li><li><strong>Using APIs Requiring </strong><strong>Pin:</strong> Some low-level async utilities or functions might require arguments of type Pin&lt;&amp;mut T&gt;.</li><li><strong>Storing and Polling </strong><strong>!Unpin Futures:</strong> If you need to store a future generated by an async fn (which might be !Unpin) and poll it later, you often need to pin it first. Common ways include:<br>- <strong>Heap Pinning:</strong> Box::pin(my_async_fn()) creates a Pin&lt;Box&lt;dyn Future +...&gt;&gt;, allocating the future on the heap and pinning it there. This is safe.<br>- <strong>Stack Pinning:</strong> Macros like tokio::pin! or pin_utils::pin_mut! allow pinning a future to its current stack frame. This involves unsafe code under the hood but is provided through safe macro interfaces.</li></ul><p>A common source of confusion is encountering compiler errors like “the future is not Unpin&quot; or &quot;Future is not implemented for &amp;mut MyFuture&quot;. This often happens when trying to poll a future via a mutable reference (&amp;mut fut) or move it in a context where Unpin is required, but the future is self-referential (!Unpin). The solution is usually to pin the future first using one of the methods above, as Pin&lt;&amp;mut F&gt; <em>does</em> implement Future even if F is !Unpin.</p><p>In essence, Pin is a clever (though initially complex) mechanism that allows Rust&#39;s efficient, stack-based async state machines to safely handle the self-references required for borrowing across await points, all without fundamentally changing Rust&#39;s move semantics or requiring garbage collection. While often invisible, understanding its purpose demystifies many async-related compiler errors and design patterns.</p><p>These async complexities — cancellation and pinning — are intertwined with other challenges like lifetime management (&#39;static bounds often required by runtimes ), thread safety (Send/Sync bounds ), and runtime coupling. Effectively writing robust async Rust often involves structuring code carefully, perhaps separating I/O-bound async logic from synchronous, computational logic , and choosing appropriate synchronization tools (sometimes a std::sync::Mutex is fine, even in async code, if the lock isn&#39;t held across an .await ).</p><p><strong>Conclusion: Wielding Advanced Rust with Principle</strong></p><p>From my perspective, effectively using these advanced features goes beyond syntax. It requires a deep understanding of the underlying mechanisms, the trade-offs involved, and the potential failure modes. Sound design principles — like isolating unsafe code and structuring async logic carefully – are paramount. Continuous learning and a healthy respect for the complexities involved are essential for building robust, maintainable, and performant systems.</p><p>To deepen your understanding, consider exploring these resources:</p><ul><li><strong>The Rustonomicon:</strong> The definitive guide to unsafe Rust, Undefined Behavior, and data layout.</li><li><strong>The Async Rust Book:</strong> Covers async/.await fundamentals, including Pin, Future, and ecosystem patterns.</li><li><strong>Tokio Documentation &amp; Tutorials:</strong> Provides practical guidance on using the Tokio runtime, including patterns for cancellation, task management, and I/O.</li></ul><p>By embracing these advanced concepts with diligence and principle, developers can leverage Rust to build truly exceptional software — systems that are not only performant and concurrent but also maintain a high degree of safety and reliability.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f5d4f15f28c2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Performance Engineering — Part 1]]></title>
            <link>https://medium.com/@martin00001313/performance-engineering-part-1-dce227aeb0ad?source=rss-4a528c57ee66------2</link>
            <guid isPermaLink="false">https://medium.com/p/dce227aeb0ad</guid>
            <category><![CDATA[c-programming]]></category>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[cpu]]></category>
            <category><![CDATA[cpp]]></category>
            <dc:creator><![CDATA[Martin Ayvazyan]]></dc:creator>
            <pubDate>Sat, 05 Apr 2025 21:02:34 GMT</pubDate>
            <atom:updated>2025-04-05T22:09:03.445Z</atom:updated>
            <content:encoded><![CDATA[<h3>Performance Engineering — Part 1</h3><h3>Root-Causing and Resolving issues related to Excessive Context Switching, Memory Fragmentation</h3><p>Performance issues on Unix/Linux systems can be tricky to diagnose. Sometimes CPUs sit idle while tiny tasks churn, or memory allocations somehow slow down over time. In these article-series, we’ll walk through <strong>five /complex application performance problems</strong> — and how to detect and fix them — in a hands-on, practical way. I assume you’re comfortable with Linux internals, but will keep it accessible with clear explanations, command examples, and code snippets. It will include: <strong>(1)</strong> CPU idle time caused by excessive context switching, <strong>(2)</strong> memory fragmentation, <strong>(3)</strong> disk I/O bottlenecks, <strong>(4)</strong>networking delays, and <strong>(5)</strong> cases where CPU is free but threads are blocked waiting.</p><p>For each, we’ll see how to <strong>detect</strong> the issue using tools (like top, vmstat, pidstat, perf, etc.), and then how to <strong>mitigate</strong> or fix it (with tuning and C++ code examples). Let’s dive in!</p><h3>CPU Idle Time from Excessive Context Switching</h3><p>When your CPU is mostly idle yet the system feels slow, one culprit can be <strong>excessive context switching</strong>. Context switches occur when the OS scheduler swaps out one thread for another. Each switch has overhead (saving/restoring registers, flushing caches), so too many context switches can waste CPU time without doing real work​. This often happens if an application spawns <em>too many short-lived tasks</em> or threads. <em>This is something I had to deal with while investigating performance of a low-level application which used Rust-Tokio for cooperative multitasking.</em></p><p><strong>Detecting Excessive Context Switching</strong></p><p><strong>Symptoms:</strong> High context-switch rates, CPU showing idle time despite work to do, and possibly high system CPU usage. The system may appear to “thrash” between tasks without making progress.</p><p><strong>Tools to use:</strong></p><p><strong>vmstat</strong> – Check the “<strong>cs” (context switch)</strong> column. For example, vmstat 1 5 will print stats every 1s. A very high “cs” value (thousands per second) is a red flag. In one case, vmstat showed over <strong>350k context switches</strong> in a short interval – an unusually high number pointing to a performance issue​</p><p><strong>pidstat -w</strong> – Shows context switches per process. Run pidstat -w 1 to see which processes are doing many voluntary (nvcswch/s) or involuntary (nivcswch/s) context switches per second. A process with extremely high context switches might be spawning threads or yielding frequently.</p><p><strong>top </strong>or<strong> </strong><strong>htop</strong> – These won’t show context switch counts directly, but if you see low CPU usage (%us and %sy) with non-zero load average, something is off. You can also press <strong>H</strong> in htop to show threads; if you see many threads of a process rapidly appearing/disappearing or all waiting, investigate context switching.</p><p><strong>perf stat</strong> – For deeper analysis, use Linux perf. For example: perf stat -a -e cs -e migrations -e faults sleep 4will count context switches, migrations, page faults system-wide in 4 seconds. You can also run a specific process with perf stat. A very high context-switch count (relative to work done) indicates overhead. In one real-world case, a workload saw ~1 million context switches in ~2.5 seconds​ – a huge overhead from thread thrashing.</p><p>If you suspect <em>thrashing</em> (CPU constantly switching tasks), monitor the run queue and interrupts as well. Tools like <strong>Netdata</strong> can visualize context switches over time and help spot sudden spikes​.</p><p><strong>Mitigating Context Switch Overhead</strong></p><p><strong>Root cause:</strong> Often, the application is creating too many threads or processes for tiny tasks. For example, spawning a thread for each of thousands of short tasks will thrash the scheduler. Another cause can be overly frequent I/O causing the CPU to switch context to wait (more on I/O wait later).</p><p><strong>Solutions:</strong></p><p><strong>Use Thread Pools:</strong> Instead of spawning a new thread for each task, maintain a pool of worker threads. The pool threads pick up tasks from a queue. This amortizes thread creation/teardown cost and cuts context switches dramatically​. Most languages/frameworks have thread pool libraries (e.g., C++20 &lt;thread&gt; or Boost thread_pool).</p><p><strong>C++ Example — Naive vs Thread Pool:</strong><br><em>Naive approach (spawn per task):</em></p><pre>// BAD: spawning 10000 threads for 10000 short tasks<br>std::vector&lt;std::thread&gt; tasks;<br>tasks.reserve(10000)<br>for(int i=0; i&lt;10000; ++i) {<br>    tasks.emplace_pack([]{ /* short task */ });<br>}<br>std::for_each(tasks.begin(), tasks.end(), [](auto &amp;t) {t.join();});</pre><p>Using a simple thread pool:</p><pre>thread_pool pool{10}; // 10 threads - a custom pull<br>for(int i=0; i&lt;10000; ++i) {<br>    pool.enqueue([]{ /* short task */ });<br>}<br>pool.join_all(); // join threads</pre><p>Here, 10threads handle all 10000 tasks. We avoid creating 10000 threads, reducing context switches. The CPU spends time doing work, not constantly launching threads.</p><p><strong>Batch work on one thread:</strong> If tasks are extremely short and sequential, it might even be faster to run them on one thread to avoid context switches altogether. For example, in a loop processing a small CSV file, using goroutines (threads) actually slowed things down due to overhead​. Removing concurrency (just looping in one goroutine) solved the issue​. Moral: don’t over-parallelize trivial work.</p><p><strong>Adjust scheduling or CPU affinity:</strong> In some cases, binding certain threads to CPUs (using <em>CPU affinity</em>) can reduce unnecessary context switching between CPUs and improve cache usage​. This is an advanced tweak — use if specific threads benefit from staying on the same core.</p><p>Reducing context switches often boils down to <strong>doing less context switching</strong> — use fewer threads, avoid sleep/yield loops, and let threads do more work per context. As a result, the CPU will spend more time executing your code and less time in scheduler overhead.</p><h3>Memory Fragmentation</h3><p>Memory fragmentation is a more insidious issue. Over time, a process or the OS might have plenty of free memory <em>in total</em> but not enough contiguous free blocks to satisfy large allocations. This can lead to <strong>increased allocation latency</strong> and even failures for big allocations. In extreme cases, the kernel has to <strong>compact memory</strong> (move pages around) to service an allocation, which stalls your program.</p><p><strong>Detecting Memory Fragmentation</strong></p><p><strong>Symptoms:</strong> Slowing memory allocations, spikes in <strong>system CPU (%sy)</strong> time, or random pauses. You might see periodic jumps in CPU usage during allocations, or increased <em>page fault</em> activity. In severe cases, processes might get stuck in “D” (uninterruptible sleep) state during allocations, or the system might start swapping despite plenty of RAM free. High system time can indicate the kernel is busy with memory management (e.g., page reclamation or compaction).</p><p><strong>Tools/Methods:</strong></p><ul><li><strong>vmstat and </strong><strong>/proc/meminfo:</strong> Watch for high <em>si/so</em> (swap in/out) in vmstat even when RAM is available – that could be fragmentation causing the kernel to shuffle memory. Also, check /proc/meminfo for signs like very large “Slab” usage (lots of cached small allocations) or low “MemFree” but high “MemAvailable”, which might hint at fragmentation of free memory.</li><li><strong>smem:</strong> The smem tool can show memory usage by process including shared memory. It’s not a direct fragmentation detector, but if one process’s memory is highly fragmented (lots of small allocations), smem -t (totals) might show large <strong>shared memory</strong> or <strong>slab</strong> usage. A large discrepancy between RSS and unique memory could indicate fragmentation or leaked memory in allocators.</li><li><strong>/proc/buddyinfo:</strong> This is a low-level diagnostic file that shows the distribution of free memory blocks by size order (the “buddy allocator” lists how many free blocks of order 0 (4KB), order 1 (8KB), order 2 (16KB), etc., up to higher orders). Run cat /proc/buddyinfo and look at the numbers. If all high-order free counts are zero (e.g., no free chunks of order 9 or 10), it means memory is fragmented – large contiguous allocations will fail​. <strong>Example:</strong> If you see zeros for orders 8 and above, any allocation requiring &gt;2⁸ * 4KB = 1MB of contiguous RAM will likely stall or fail​. This is a strong sign of fragmentation at the system level.</li><li><strong>perf (advanced):</strong> In extreme cases, you can catch the kernel in the act of compaction. Using perf top or perf recordwhile the issue is happening might show functions like compact_zone or migrate_pages on the stack​. This indicates the kernel is spending time defragmenting memory (compaction) to satisfy allocations.</li></ul><p>Keep in mind, <strong>diagnosing fragmentation is tricky</strong> — it often masquerades as other issues (high CPU, slow I/O, etc.). A combination of the above observations — high system CPU, processes stuck in kernel memory allocation, and buddyinfo showing few high-order frees — points to fragmentation as the root cause.</p><p><strong>Why it matters:</strong> Fragmentation primarily impacts performance when an allocation request can’t be fulfilled easily. The kernel then does extra work (compacting memory or swapping) to allocate, which <strong>increases allocation latency </strong>dramatically. In other words, malloc() or new might occasionally pause for milliseconds (or more) to rearrange memory, which can wreak havoc on latency-sensitive applications. <strong>Memory fragmentation</strong> is essentially “available memory, just not in one piece”​</p><h4><strong>Mitigating Memory Fragmentation</strong></h4><p>Once you suspect fragmentation, there are a few strategies to mitigate it:</p><p><strong>Memory Pooling:</strong> Use custom allocators or memory pools for frequently allocated small objects. A memory pool grabs a large chunk of memory up front and then doles it out in fixed-size pieces to your program. This has two benefits: it <em>reduces fragmentation</em> (because it avoids constantly requesting and freeing random-sized blocks from the OS) and often improves allocation speed (since it can be as simple as pointer arithmetic). As is noted, <em>“Memory pools reduce memory fragmentation and improve allocation efficiency by pre-allocating a large block of memory and dividing it into fixed-size chunks.”</em>​.</p><p><strong><em>Note</em></strong>: you may consider to implement your own <strong>static or dynamic</strong> memory pooling component, where statics are specially preffered for development of autonomous systems (cars, robots, etc).</p><p><em>A C++ Example — by Using a Memory Pool/Allocator:</em><br>Modern C++ provides &lt;memory_resource&gt; for custom allocators. For instance, a monotonic buffer resource is a simple pool.</p><pre>#include &lt;memory_resource&gt;<br><br>static std::array&lt;std::byte, 1024*1024&gt; buf;           // 1 MB buffer<br>std::pmr::monotonic_buffer_resource pool(buf.data(), buf.size());<br>std::pmr::vector&lt;my_obj_t&gt; vec{&amp;pool};                // uses pool for allocations<br>for(int i=0; i&lt;10000; ++i) {<br>    vec.emplace_back(/*...*/);  // all allocated inside our 1MB buffer<br>}</pre><p>In this example, all 10k my_obj_t instances are allocated inside one big 1MB block we provided. No fragmentation from those allocations will occur beyond that block, and there’s no overhead of 10k separate new calls. You can also implement your own pool with free lists for fixed-size objects if needed. The downside is you must estimate pool size and manage lifetime carefully, but for long-running processes it’s worth it.</p><p><strong>Tune the Memory Allocator:</strong> The default Linux allocator (glibc malloc) uses techniques like arenas and mmap to balance performance and fragmentation. In some cases, you can tweak it: for example, setting environment variable MALLOC_ARENA_MAX to a lower number can reduce fragmentation in multi-threaded apps (at cost of some contention). Alternative allocators like <strong>jemalloc</strong> or <strong>tcmalloc</strong> are designed to reduce fragmentation and have more predictable allocation performance – consider using them if fragmentation is a persistent issue.</p><p><strong>Release Memory Back to OS:</strong> Frequent allocations/deallocations can lead to “heap fragmentation” inside your process. If your process has freed memory but not returned it to the OS, malloc_trim() (glibc) or similar can give back free chunks to the OS, making them available as contiguous free RAM. Also, using huge pages for large allocations can bypass fragmentation (since 2MB pages reduce the need for contiguous 4KB chunks) – but huge pages come with their own complexity.</p><p><strong>Redesign data structures:</strong> In some cases, how you manage memory can be changed. For example, if you have a large vector that grows and shrinks, it may fragment memory. Using a deque or a linked structure might distribute allocation, or allocating large buffers and reusing them (instead of free/new cycles) can help. This is case-specific, but worth examining.</p><p>Sometimes, when memory fragmentation is unavoidable/unpredictable, it’s beneficial to switch from continuous (std::vector in C++) to chunking data structures (like std::dequeue). For me. this in a real scenario helped to improve application <strong>performance by 70%</strong>.</p><p>The key to fragmentation is <strong>preventative</strong>: use pooling or fixed-allocation strategies for objects that are allocated frequently in unpredictable patterns. This keeps the memory layout stable. Monitoring fragmentation levels (e.g., periodically checking /proc/buddyinfo and allocation latency) can inform whether your changes are effective. When done right, you’ll find memory allocations become more predictable and faster, and those random pauses due to memory compaction will disappear.</p><p><strong>In the next part</strong> of the PE series we will cover: <strong>(3)</strong> disk I/O bottlenecks, <strong>(4)</strong>networking delays, and <strong>(5)</strong> cases where CPU is free but threads are blocked waiting.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dce227aeb0ad" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Advanced C++ Optimization Techniques for High-Performance Applications — Part 3]]></title>
            <link>https://medium.com/@martin00001313/advanced-c-optimization-techniques-for-high-performance-applications-part-3-4602df9284d8?source=rss-4a528c57ee66------2</link>
            <guid isPermaLink="false">https://medium.com/p/4602df9284d8</guid>
            <category><![CDATA[gaming]]></category>
            <category><![CDATA[cpp]]></category>
            <category><![CDATA[c]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Martin Ayvazyan]]></dc:creator>
            <pubDate>Sun, 23 Mar 2025 07:20:45 GMT</pubDate>
            <atom:updated>2025-03-23T07:20:45.159Z</atom:updated>
            <content:encoded><![CDATA[<h3>Advanced C++ Optimization Techniques for High-Performance Applications — Part 3</h3><p>This article is continuation of my previous 2 articles(<a href="https://medium.com/@martin00001313/advanced-c-optimization-techniques-for-high-performance-applications-part-1-55aba61b4dad">part-1</a> &amp; <a href="https://medium.com/@martin00001313/advanced-c-optimization-techniques-for-high-performance-applications-part-2-6dc2ebf64813">part-2</a>) focused on optimization techniques, where we will cover topics/areas of optimization related to multithreading and parallel execution, such as:</p><ul><li>Threading and Parallelism Optimizations</li><li>Lock-Free and Atomic Operations</li><li>Avoiding False Sharing</li><li>Thread Affinity and NUMA Awareness</li><li>General conclusion</li></ul><p>Utilizing multiple cores via multithreading can vastly improve throughput, but it also introduces new performance considerations. Efficient multithreading is not just about using std::thread or std::async – it’s about minimizing overhead from synchronization and avoiding bottlenecks that can negate the benefits of parallelism.</p><h3>Lock-Free and Atomic Operations</h3><p><strong>Locks (mutexes)</strong> provide mutual exclusion but can become a scalability issue if many threads contend on the same lock. The context switch or spinning costs can add up, and if a lock is heavily contended, threads spend time waiting (which is wasted CPU time). <strong>Lock-free</strong> programming aims to avoid traditional locks by using atomic operations and careful algorithms so that threads can progress without blocking each other.</p><p>C++ offers atomic types (std::atomic&lt;T&gt;) which support operations like load, store, fetch_add, etc. atomically (often implemented via lock-free CPU instructions). For example, a simple counter increment across threads:</p><pre>std::atomic&lt;int&gt; counter{0};</pre><pre>// Each thread does:<br>for (int i = 0; i &lt; 1000000; ++i) {<br>    counter.fetch_add(1, std::memory_order_relaxed);<br>}</pre><p>This will run in parallel without a mutex. Under the hood, on x86 this fetch_add might use a LOCK XADD instruction which locks the memory bus for an instant to do the atomic add. It’s still not free (atomic ops have a cost, especially if many CPUs try to update the same variable, they serialize), but it avoids the overhead of a mutex and thread scheduling. Lock-free data structures (like queues, stacks, etc.) use atomic primitives (compare-and-swap, etc.) to coordinate multiple threads without explicit locks. They can achieve much higher concurrency, but are tricky to implement correctly.</p><p><strong>When to use lock-free:</strong> In high-performance scenarios where a mutex is a bottleneck. For example, a logging system where many threads produce log messages could use a lock-free queue to enqueue messages instead of a locked queue, to reduce contention. Or in real-time systems where blocking is unacceptable. Always weigh complexity vs benefit; lock-free algorithms can have subtle bugs (like ABA problem, memory reclamation issues). Sometimes a well-designed sharded locking (reducing contention by having multiple locks) or using higher-level frameworks (like concurrent queues from TBB or folly) is easier.</p><p>Also consider using higher-level parallel algorithms (C++20 &lt;execution&gt;) or libraries like Intel TBB, which manage worker threads and tasks efficiently, so you focus less on low-level synchronization.</p><h3>Avoiding False Sharing</h3><p><strong>False sharing</strong> is a sneaky performance bug in multithreading. It occurs when threads are <em>technically</em> working on different variables, but those variables happen to reside on the same cache line. Each core has its own cache, and when one core updates a cache line, it invalidates that line in other cores’ caches (per the cache-coherency protocol). If two threads keep ping-ponging updates to the same cache line (even if to different data on that line), they cause a lot of coherency traffic and stalls​. For example, suppose you have:</p><pre>struct Counters {<br>    std::atomic&lt;int&gt; c1;<br>    std::atomic&lt;int&gt; c2;<br>} counters;</pre><pre>void thread1() {<br>    for (...) counters.c1.fetch_add(1);<br>}<br>void thread2() {<br>    for (...) counters.c2.fetch_add(1);<br>}</pre><p>If c1 and c2 end up on the same cache line (likely, since they are adjacent in memory in the struct), thread1 and thread2 will <strong>interfere</strong> with each other’s caches even though they work on different atomic variables. Each increment invalidates the line for the other core, causing them to continually reload it. This false sharing can dramatically slow down throughput – sometimes by an order of magnitude, as the CPUs spend time passing the cache line back and forth​.</p><p><strong>Solution:</strong> <strong>Padding or aligning</strong> to separate such variables. We can prevent c1 and c2 from sharing a cache line:</p><pre>struct PaddedCounters {<br>    alignas(64) std::atomic&lt;int&gt; c1;<br>    alignas(64) std::atomic&lt;int&gt; c2;<br>};</pre><p>Given a typical cache line is 64 bytes, alignas(64) ensures each counter starts on its own cache line. The compiler may also insert padding between them because of the alignment, so they won’t reside on the same line. Now thread1 and thread2 operate on independent cache lines, and there’s no false sharing. The memory cost is a few dozen extra bytes, which is negligible compared to the performance gain in a hot multi-threaded loop.</p><p><strong>Detecting false sharing:</strong></p><p>It can be tricky to spot. Symptoms include: you have threads supposedly working on independent data but see no speedup (going from 1 thread to 2 threads gives little improvement or even slow-down). Tools like cachegrind or Intel VTune can help identify a lot of cache invalidation traffic. In some cases, just being aware of your data layout and aligning sensitive data is enough.</p><p>Beyond padding, sometimes reorganizing data structures can help. If you have an array of structures updated by multiple threads, and different threads mostly use different elements of the array, false sharing is less of an issue. But if threads use elements that are contiguous, they might share cache lines. In such cases, spacing them out (e.g., allocate an array of structures with padding in each such that each structure starts on its own cache line) could be beneficial.</p><h3>Thread Affinity and NUMA Awareness</h3><p><strong>Thread affinity</strong> refers to binding a thread to a specific CPU core or set of cores, ensuring it consistently runs on that core. Without explicit affinity, the operating system might schedule threads to run on different cores at various times, causing frequent cache invalidations and migrations, which negatively impact performance. When threads migrate between cores:</p><ul><li><strong>Cache locality</strong> is reduced because the thread loses cached data, increasing cache misses and latency.</li><li><strong>Cache coherency traffic</strong> increases, leading to performance degradation due to additional overhead.</li></ul><p>Setting thread affinity helps achieve:</p><ul><li><strong>Better cache locality</strong>: Data stays warm in the local cache.</li><li><strong>Reduced context-switch overhead</strong>: Threads experience fewer stalls, leading to smoother execution.</li><li><strong>Improved predictability and stability</strong>: Threads’ performance becomes more predictable, essential for real-time or latency-sensitive applications.</li></ul><p>In C++, you can set thread affinity using OS-specific APIs. For instance, on Linux:</p><pre>#include &lt;pthread.h&gt;<br>#include &lt;sched.h&gt;<br><br>void setAffinity(std::thread &amp;thread, int core_id) {<br>    cpu_set_t cpuset;<br>    CPU_ZERO(&amp;cpuset);<br>    CPU_SET(core_id, &amp;cpuset);<br>    pthread_setaffinity_np(thread.native_handle(), sizeof(cpu_set_t), &amp;cpuset);<br>}</pre><p><strong>NUMA (Non-Uniform Memory Access)</strong> considerations further enhance performance on multi-socket systems, where each CPU socket has local memory. Ensuring threads run on cores local to their frequently accessed data minimizes memory latency.</p><p>Best practices:</p><ul><li><strong>Profile before setting affinity</strong>: Ensure this optimization yields tangible performance gains for your specific application.</li><li><strong>Be mindful of NUMA nodes</strong>: Place threads on cores near their critical data to leverage local memory bandwidth optimally.</li><li><strong>Use OS-specific tools</strong> like numactl, taskset, or APIs provided by libraries such as Intel TBB for simpler NUMA-aware programming.</li></ul><p>In summary, careful management of thread affinity and NUMA-awareness can significantly boost parallel application performance, especially in scenarios with high cache dependency and frequent memory accesses.</p><p><strong>Other threading optimizations briefly:</strong></p><ul><li><strong>Reduce contention:</strong> If you must use locks, try to minimize how long they’re held. Use finer-grained locks if possible (lock splitting). Or use atomic techniques to reduce how often locks are needed (e.g., atomic flags for quick checks).</li><li><strong>Read-write locks</strong> if appropriate (many readers, few writers).</li><li><strong>Parallel algorithms and workload distribution:</strong> Ensure your threads are doing roughly equal work to avoid one straggler delaying the whole job (load balancing).</li></ul><p>Multithreading adds a new dimension to optimization. The key is to maximize parallel work while minimizing synchronization and interference. By using lock-free techniques when suitable and eliminating false sharing, you can achieve near-linear scaling on multi-cores for many workloads.</p><h3>Conclusion: Applying Advanced Optimizations in the Real World</h3><p>Optimizing C++ at this level is both an art and a science. We’ve covered a spectrum from CPU-level considerations (branch prediction, pipelining, SIMD instructions) to memory hierarchy (cache locality, alignment) to multi-core issues (parallelism and false sharing) and even design paradigms (data-oriented design). Here are some best-practice takeaways for leveraging these techniques in real-world applications:</p><ul><li><strong>Profile before and after:</strong> <em>Measure, don’t guess.</em> Use profilers and hardware counters to find where the bottlenecks truly are (e.g., cache misses, branch mispredicts, lock contention). This guides you to which optimizations will matter. After applying an optimization, measure again to ensure it helped (and didn’t hurt).</li><li><strong>Optimize the hot spots:</strong> Often, 90% of execution time is spent in 10% of the code. Focus your optimization effort there. It’s usually not worth complicating the entire codebase for micro-optimizations that only impact cold code. For example, if a game’s frame time is dominated by physics and rendering, optimize those modules heavily (perhaps using SIMD, multi-threading, cache-friendly structures) and don’t worry if your UI code uses virtual functions and OOP — it’s not in the hot path.</li><li><strong>Understand the hardware:</strong> Knowing roughly how many cycles a cache miss or branch mispredict costs, or how many floats fit in a vector register, will inform your decisions. This article provided some insights: e.g., memory is slow so prioritize locality​, branches can be expensive if unpredictable, etc. Continue learning about the specific hardware you target (desktop vs mobile vs console can differ).</li><li><strong>Use the tools and features available:</strong> Modern compilers are extremely powerful — flags like -O3, -march=native, LTO, and PGO can automatically apply many optimizations. Don’t hand-roll an unrolled loop without checking if the compiler would do it. Write clear code first, then check the assembly or optimization reports. Only go low-level when you need to surpass what the compiler can do or guide it (through pragmas, intrinsics, etc.).</li><li><strong>Keep algorithms in mind:</strong> All these low-level tweaks won’t save you if you use a poor algorithm. Always ensure you have a good big-O solution before micro-optimizing. That said, once you have the right algorithm, the techniques we discussed can sometimes yield <em>multiplicative</em> speedups (2x, 5x, even 10x) on top of it by utilizing hardware well.</li><li><strong>Balance maintainability and performance:</strong> The most optimal code might be very complex (lock-free structures, multi-buffered SIMD code, etc.). In a team setting or large codebase, ensure that critical optimized code is well-documented, isolated, and justified by need. Sometimes you can achieve performance and keep code clean by using libraries (for example, use Eigen for linear algebra — it will handle vectorization). But other times you must get your hands dirty. Just make sure the gain is worth the added complexity.</li></ul><p>Advanced optimizations are crucial in fields like <strong>game development</strong> (where you need to squeeze every drop of performance for real-time rendering and simulation), <strong>high-performance computing</strong> (where saving 10% on a computation can save thousands of dollars on a supercomputer run), and <strong>embedded systems</strong> (where hardware is limited and timing is critical). A game engine might use data-oriented design for its core loop, an HPC simulation will tile computations to fit caches and use SIMD math, and an embedded signal processor might carefully tune loops in assembly. The principles, however, remain consistent across domains.</p><p>In the end, <strong><em>performance optimization in C++ is about writing code that works with the CPU, not against it</em></strong><em>.</em> By applying branch prediction hints, organizing memory access patterns, leveraging parallel execution units (SIMD, multi-core), and structuring data optimally, you can often achieve substantial performance improvements. Keep learning and experimenting — the landscape of CPUs and compilers evolves, but a solid understanding of these advanced techniques will serve you for years to come in writing high-performance C++ software.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4602df9284d8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Advanced C++ Optimization Techniques for High-Performance Applications — Part 2]]></title>
            <link>https://medium.com/@martin00001313/advanced-c-optimization-techniques-for-high-performance-applications-part-2-6dc2ebf64813?source=rss-4a528c57ee66------2</link>
            <guid isPermaLink="false">https://medium.com/p/6dc2ebf64813</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[c]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[cpp]]></category>
            <dc:creator><![CDATA[Martin Ayvazyan]]></dc:creator>
            <pubDate>Sun, 23 Mar 2025 07:08:19 GMT</pubDate>
            <atom:updated>2025-03-23T07:10:32.646Z</atom:updated>
            <content:encoded><![CDATA[<h3>Advanced C++ Optimization Techniques for High-Performance Applications — Part 2</h3><p>In the <a href="https://medium.com/@martin00001313/advanced-c-optimization-techniques-for-high-performance-applications-part-1-55aba61b4dad">first</a> article of this series, we explored some of optimization techniques like branch prediction, cache optimization strategies, and SIMD vectorization. These foundational techniques help ensure your C++ applications leverage the full power of modern CPU architectures. Here, we delve deeper into additional powerful strategies, enabling even more fine-grained optimization. We’ll cover:</p><ul><li>Loop Unrolling and Vectorization</li><li>Function Inlining and Instruction Cache Effects</li><li>Return Value Optimization (RVO and NRVO)</li><li>Link Time Optimization (LTO) and Whole-Program Optimization</li><li>Memory Alignment and Padding</li><li>Data-Oriented Design vs. Object-Oriented Design for Performance</li></ul><p>These techniques, combined with those previously covered, will help craft high-performance C++ applications capable of fully utilizing the hardware’s potential, reducing latency, and maximizing throughput.</p><h3>Loop Unrolling and Vectorization</h3><p>Loop unrolling is a classic optimization where the loop body is repeated multiple times per iteration, reducing the loop overhead (index increment, branch/jump for loop) and sometimes enabling further optimizations. It often goes hand-in-hand with vectorization and instruction-level parallelism.</p><p>What is loop unrolling? It’s easier to show by example. Suppose we have a simple loop:</p><pre>// Simple loop<br>for (int i = 0; i &lt; N; ++i) {<br>    sum += arr[i];<br>}</pre><p>An unrolled version (unroll factor 4) would do something like:</p><pre>// Unrolled loop (by a factor of 4)<br>int i = 0;<br>for (; i + 3 &lt; N; i += 4) {<br>    sum += arr[i];<br>    sum += arr[i+1];<br>    sum += arr[i+2];<br>    sum += arr[i+3];<br>}<br>for (; i &lt; N; ++i) {  // handle leftover<br>    sum += arr[i];<br>}</pre><p>Now the loop overhead (the check and increment) happens every 4 elements instead of every element. This can reduce the number of branches and improve throughput. Additionally, unrolling may expose more opportunities for the CPU to pipeline instructions or for the compiler to schedule instructions better (hiding latencies).</p><p>Benefits:</p><ul><li>Fewer branch instructions (the loop counter check/jump) per amount of work, which means less branch prediction to worry about and less pipeline disruption.</li><li>More straight-line code in the inner loop, which can be optimized better by the CPU’s instruction scheduler. It can also help with auto-vectorization because the compiler may more easily see how to use SIMD across the unrolled group of operations.</li><li>Possibility of combining with other optimizations: e.g., software pipelining or unroll-and-jam (unrolling an outer loop and intermixing operations from inner loops)</li></ul><p>Drawbacks:</p><ul><li><strong>Code size increase:</strong> The unrolled loop is longer in terms of machine code. This can bloat the instruction cache or even prevent the CPU’s loop buffer/micro-op cache from holding the whole loop efficiently​. If the loop body is large or the unroll factor is big, you might end up hurting performance by causing more instruction cache misses. Agner Fog’s optimization guide notes that excessive unrolling can overload the code cache and even defeat the loopback buffer on some CPUs​.</li><li>If N is not a multiple of the unroll factor, you need extra handling for the leftover iterations (as shown above), which adds a bit of complexity.</li><li>Diminishing returns: Unrolling beyond a certain point gives no benefit and can even hurt. Often unrolling by 2x or 4x is enough. Compilers sometimes unroll small loops automatically at higher optimization levels, so manual unrolling might be redundant.</li></ul><p><strong>When to use it:</strong> Loop unrolling is useful in tight loops where the loop overhead is a measurable fraction of the work. It’s commonly used in low-level library code (like vector math, image processing routines) where every cycle counts. If you’re already using SIMD intrinsics, you’ll often unroll loops to handle more data per iteration (as seen in the AVX example earlier, where we processed 8 elements per loop instead of 1). However, always measure — modern CPUs are pretty good at predicting loop branches and overlapping instruction execution, so naive unrolling might not always help. And remember that the compiler might do it for you. If you do it manually, ensure the unroll factor is reasonable and test on target hardware. <strong>Combining with vectorization </strong>— Unrolling is complementary to vectorization. You might unroll by a factor equal to the SIMD width or a multiple of it. For instance, if using 256-bit vectors (8 floats) you could unroll by 2 to handle 16 floats per iteration (two vector operations). This can fully utilize execution units if the CPU can issue multiple vector operations in parallel.</p><p>In summary, <strong>loop unrolling</strong> reduces loop overhead and can expose instruction-level parallelism, but beware of the trade-off with code size and cache. Use it when you have a hot loop doing little work per iteration (so the loop control is a big part of the cost) and test to find the sweet spot of unroll factor. Often, trust the compiler’s judgment at -O3, but be ready to override it in critical cases where you know more about the context.</p><h3>Function Inlining and Instruction Cache Effects</h3><p>Frequent function calls can add overhead — not just the call/return instructions, but also potential instruction cache misses if the function’s code is far away in memory. <strong>Function inlining</strong> is an optimization where the compiler replaces a function call with the function’s body (as if you wrote it inline). This saves the call overhead and can also enable further optimizations across the function boundary.</p><p><strong>Inlining benefits:</strong></p><ul><li><strong>Eliminating call overhead:</strong> A normal function call involves jumping to another location (which may disrupt instruction flow), pushing return addresses, etc. Inline expansion avoids that. For very small functions (think getters, setters, tiny arithmetic functions), this can save a few cycles and improve performance</li><li><strong>Enabling further optimizations:</strong> Once inlined, the function’s code is part of the caller, so the compiler might optimize it more. It can const-propagate values, remove dead code, etc., across what used to be a call boundary. Inlining can also help with auto-vectorization or unrolling if the function was called inside a loop</li><li><strong>Smaller code (in some cases):</strong> If a function is only called once, inlining it will remove the need to keep a separate copy of its code and eliminate call setup, potentially making the program smaller​ (However, the opposite is usually true when a function is called many times; see below.).</li></ul><p><strong>Inlining drawbacks:</strong></p><ul><li><strong>Code bloat:</strong> Inlining replicates the function’s code at each call site. If you inline a medium-sized function that’s called from 10 places, you now have 10 copies of that code. This increases the binary size and can have negative effects on the <strong>instruction cache</strong>. The CPU’s instruction cache (and decode pipelines) have limited capacity, and if your hot code doesn’t fit, it can slow down. Thus, inlining is a trade-off: speed from avoiding calls <em>versus</em> potential slowdowns from a larger code footprint</li><li>The compiler might refuse to inline a function if it thinks the cost outweighs the benefit (or if the function is recursive/virtual/etc.). The inline keyword in C++ is more about linkage than a command to inline, though compilers do take it as a hint. Use [[gnu::always_inline]] (GCC/Clang) or __forceinline (MSVC) if you <em>really</em> need to force it, but that’s rarely needed except in critical low-level code.</li></ul><p>Tips:</p><ul><li>Inline small, frequently-called functions, especially those in tight loops. Good candidates are simple getters/setters or small math functions. Modern C++ standard library uses inline aggressively for things like std::vector::size() (which is typically just returning a member variable).</li><li>Be cautious about inlining large functions or those with complex logic, as it can blow up your code size. Measure the impact — sometimes keeping a function call (which might be predicted well by the CPU’s return predictor) is fine.</li><li>Remember that <strong>LTO (Link Time Optimization)</strong>, discussed later, can inline even across source files. So you don’t have to put everything in headers as inline functions to get cross-module inlining – LTO can allow inlining of non-header functions as well, if enabled.</li></ul><p><strong>Instruction cache note:</strong> The instruction cache (L1i) is usually around 32KB on many CPUs (and a micro-op cache on Intel decoders as well). This cache holds the instructions your program is executing. If you have a lot of code concentrated in a hot path (due to inlining and unrolling, for instance), you could overflow the cache and incur instruction fetch stalls. It’s a nuanced point: more inlining = fewer calls, but also = more code bytes. As an expert, you should be aware of this balance. Sometimes, <strong>less</strong> is more: it could be beneficial to <em>not</em> inline certain functions to keep the tight loop smaller, fitting in the uop cache or L1i. This is highly microarchitecture-dependent, so it’s an area to use profiling and perhaps processor performance counters (like ITLB misses or L1i misses) if you suspect instruction cache issues.</p><h3>Return Value Optimization (RVO and NRVO)</h3><p>C++ is often criticized for copying large objects when returning by value, but modern C++ compilers have optimizations to eliminate those copies. <strong>Return Value Optimization (RVO)</strong> and <strong>Named RVO (NRVO)</strong> refer to situations where the compiler can construct the return value <em>directly</em> in the caller’s space, avoiding any copy or move.</p><p>For example:</p><pre>struct Big { int data[1000]; };</pre><pre>Big makeBig() {<br>    Big b;<br>    // ... fill b ...<br>    return b; // NRVO can eliminate the copy here<br>}</pre><p>In the code above, one might think b is copied to the caller. However, compilers typically perform NRVO: they will construct b directly in the memory location where the function’s return value should go. Thus, no actual copy happens. Similarly, if you return Big{}; (an unnamed temporary), that’s the classic RVO case where the temporary is constructed in place at the call site.</p><p><strong>C++17 mandates</strong> copy elision in certain cases. Specifically, when returning a prvalue (temporary) of the same type as the function return type, the compiler <em>must</em> omit the copy and construct in place. For named return values (NRVO), the standard doesn’t guarantee it, but compilers are allowed to do it and generally will if they can​.</p><p><strong>Why it matters:</strong> If you’re writing code that returns large objects by value (which is a nice, safe style thanks to move semantics and RVO), you want to be sure the compiler optimizes it. Thankfully, in almost all modern compilers, RVO/NRVO will kick in for typical cases. For instance:</p><pre>Big f() {<br>    return Big(); // guaranteed RVO in C++17, no copy at all<br>}<br>Big g() {<br>    Big x;<br>    // ... <br>    if(condition) {<br>        return x;  // NRVO (optional, but most compilers do it)<br>    } else {<br>        return Big();<br>    }<br>}</pre><p>In function g, NRVO might not apply if there are multiple return paths (the compiler might not be able to prove one construction can be used for both). But the return of a temporary Big() will be RVO’d.</p><p><strong>Implications for writing code:</strong></p><ul><li>Feel free to return by value for large objects if it makes the interface cleaner. In the past (pre-C++11), people would avoid returning by value due to copy cost, but with modern compilers, those copies are usually eliminated or turned into moves. It’s often more elegant than output parameters.</li><li>However, ensure you’re not accidentally defeating RVO. For example, if you have:</li></ul><pre>Big makeBig(bool flag) {<br>    Big a, b;<br>    if(flag) return a;<br>    else return b;<br>}</pre><p>The compiler cannot apply NRVO here because two different local objects might be returned. It will likely do a move (C++11 onward) which is cheaper than copy, but if Big was not movable, it might copy. In scenarios like this, if it matters, you could refactor (perhaps use a single object and set it conditionally, then return).</p><ul><li><strong>Named RVO vs unnamed RVO:</strong> They are just variations, but as a developer, you generally just rely on the compiler. The standard​ assures that returning a temporary is safe. For named, most compilers (GCC, Clang, MSVC) do it when optimization is on.</li></ul><p>In summary, <strong>RVO/NRVO make returning large objects efficient.</strong> The best practice is to design functions in a natural way (return objects by value) and trust the compiler. If you’re curious or paranoid, you can test your specific compiler by printing log messages in copy/move constructors to see if they’re called. In optimized builds, you’ll often find they are not called at all, proving the optimization worked. This means cleaner C++ code without sacrificing performance.</p><h3>Link Time Optimization (LTO) and Whole-Program Optimization</h3><p>By default, each C++ source file is compiled in isolation, and then the linker just stitches the object files together. <strong>Link Time Optimization (LTO)</strong> is a mode where the compiler’s optimizer runs <em>after</em> all files are compiled, having a view of the entire program. This enables <strong>whole-program optimizations</strong> that aren’t possible when compiling file-by-file​.</p><p><strong>What LTO can do:</strong></p><ul><li><strong>Cross-module inlining:</strong> If you have a function defined in one .cpp and called in another, normally the compiler can’t inline it because it only sees a reference to it. With LTO, the compiler sees the body during link time and can inline it if beneficial (even if you didn’t mark it inline).</li><li><strong>Interprocedural optimizations across files:</strong> e.g., constant propagation across function calls that span files, removal of unused functions (dead code elimination on a global scale), optimizing virtual calls when the whole program is known (devirtualization if it can see a class is never subclassed, etc.).</li><li><strong>Better optimization decisions:</strong> Sometimes the compiler is conservative without whole-program knowledge. For example, it might not realize a function has a single caller, so it won’t inline or optimize it fully. With LTO it knows and can optimize accordingly (like static linking everything as one unit).</li></ul><p>Using LTO is usually as simple as a compiler/linker flag (e.g., -flto for GCC/Clang, or /GL and /LTCG for MSVC). It will make your build slower and memory-hungry because the optimizer is doing more heavy lifting at link time. But the runtime speed can improve.</p><p><strong>Real-world impact:</strong> LTO often yields a few percent performance improvement, but it could be more if your code has lots of cross-module interactions. In critical systems, those few percent might be worth it. Some modern applications and games enable LTO in release builds to squeeze out extra performance. It’s also helpful for size optimizations (it might remove redundant code).</p><p><strong>Best practices with LTO:</strong></p><ul><li>Ensure your build pipeline supports it (all objects must be built with LTO and the linker needs to handle it).</li><li>Use it for release builds, not necessarily for every debug build (to save compile time).</li><li>Combine LTO with Profile-Guided Optimization (PGO) if you really want to push the limit. PGO uses runtime profiling data to inform optimizations (like branch probabilities, hot/cold functions) which, combined with LTO, gives the optimizer full knowledge to rearrange code and optimize hot paths.</li></ul><p><strong>Whole-Program Optimization beyond LTO:</strong> In some cases, whole-program analysis can allow for clever things like <em>WHOLE program devirtualization</em>, aggressive inlining, or even things like identifying that certain checks are always false/true and removing them. These are niche but illustrate that more global knowledge = better optimization opportunities​.</p><p>One thing to note: LTO can sometimes uncover bugs or ODR violations in code that wasn’t apparent before (because the inlining/optimization might cause different behavior or catch inconsistencies). Make sure to test thoroughly.</p><p>In summary, <strong>LTO is a powerful switch</strong> to get the compiler to optimize your C++ program as a whole rather than per file. It often improves performance by enabling cross-module inlining and analysis​.If you’re aiming for maximum performance, it’s definitely worth enabling in production builds.</p><h3>Memory Alignment and Padding</h3><p>Memory alignment can affect performance at a low level. <strong>Alignment</strong> means that a data object’s address is a multiple of some power of two, typically the object’s size or the size of the bus transfers. CPUs often fetch memory in aligned chunks (cache lines), and certain instructions require or prefer aligned operands (e.g., SSE loads used to require 16-byte alignment, AVX 32-byte). Moreover, alignment can help avoid crossing cache line boundaries, and as we saw, it can prevent false sharing by aligning on cache line size.</p><p><strong>Key points:</strong></p><ul><li><strong>Natural alignment:</strong> By default, compilers align data structures so that their members sit at addresses optimal for the type (e.g., 4-byte alignment for int, 8-byte for double on 64-bit systems). They also insert padding in structs to satisfy these alignments for each member. For example:</li></ul><pre>struct Foo {<br>    char c;<br>    int x;<br>};</pre><p>Here x will typically be aligned at a 4-byte boundary, so there will be 3 bytes of padding after c. The struct Foo might be 8 bytes in size: 1 for c, 3 padding, 4 for x. This padding ensures that if you have an array of Foo, each x is properly aligned. The compiler does this for correctness and performance.</p><ul><li><strong>Cache line alignment:</strong> Sometimes you want to align a whole structure or global variable to a 64-byte boundary (assuming 64B cache line) to either optimize access or avoid false sharing. We saw using alignas(64) on a struct or member will do this. This is useful for frequently accessed data that you want on its own cache line.</li><li><strong>Alignment for SIMD:</strong> If you plan to use aligned SIMD instructions (like _mm256_load_ps), you need your data to be aligned to the vector width (32 bytes for AVX). You can allocate dynamic memory aligned using C++17 std::aligned_alloc or third-party allocators, or overallocate and adjust the pointer. Alternatively, use container types that allow specifying alignment. For stack or global data, use alignas(32) on the array. For example:</li></ul><pre>alignas(32) float vec[ EightOrMore ];</pre><p>Now vec will start at a 32-byte boundary, so _mm256_load_ps(vec) is safe. Misaligned accesses, if done inadvertently, either incur a slight penalty or (in older SSE) could even fault, so it&#39;s important when doing low-level intrinsics.</p><ul><li><strong>Padding for structure size:</strong> Sometimes you might add padding at the end of a struct to make its size a multiple of 64 bytes (cache line). This is done when you expect an array of such structs and want each to start on a new line to avoid cache line sharing between two elements. Example:</li></ul><pre>struct CacheLineSlot {<br>    int value;<br>    char _pad[60]; // pad the struct to 64 bytes<br>};<br>static_assert(sizeof(CacheLineSlot) == 64);</pre><p>Now an array CacheLineSlot arr[100] will have each element occupying exactly one cache line.</p><ul><li><strong>Data alignment and performance:</strong> Misaligned accesses (not on natural boundary) can sometimes be slower because the CPU might have to do two memory accesses instead of one (if data crosses a boundary). For instance, an 8-byte doublenot aligned to 8 might straddle a cache line boundary, forcing two cache lines to be accessed for that one value. Ensuring alignment avoids such penalties.</li></ul><p>In general, the compiler handles most alignment issues for you, but as an expert, you have tools (alignas, special allocators) to align critical data. Use them when you have evidence misalignment is causing slowdowns or when writing SIMD-heavy code that operates on large arrays.</p><p><strong>One caution:</strong> Over-aligning lots of small objects can waste space. If you align every int on 64 bytes, you’ll mostly be allocating padding. So use cache-line alignment judiciously for cases like arrays of counters for threads, or big structures where the cost is negligible. For normal structs, trust the compiler’s default alignment/padding which is usually optimal for general use.</p><p>To wrap up on alignment: it’s a subtle optimization that can bolster other techniques (SIMD, avoiding false sharing). It’s rarely the <em>first</em> thing to tackle, but it’s the kind of fine-tuning that can solidify a high-performance system.</p><h3>Data-Oriented Design vs Object-Oriented Design for Performance</h3><p><strong>Data-Oriented Design (DOD)</strong> is a design philosophy that emphasizes organizing code and data for optimal access patterns and performance, rather than around objects and encapsulation (which is the focus of classic OOP). The idea is to think of programs as transformations on data streams and to lay out data in memory to maximize throughput (especially with caches and SIMD), even if that means breaking some traditional OOP abstractions.</p><p>In an OOP approach, you might model a system as a collection of objects, each bundling data and behavior. For example, a game might have an Enemy class with position, velocity, health, etc., and methods like update() or takeDamage(). If you have a large array of Enemy objects, an OOP style update loop might do something like:</p><pre>for (Enemy* e : enemies) {<br>    e-&gt;update(deltaTime);<br>}</pre><p>Under the hood, this likely involves a virtual call (if Enemy is polymorphic) and each Enemy could be at a non-predictable memory location (especially if created via new all over the heap). The CPU will be jumping around memory to update each enemy, causing cache misses. Also, if update() uses only a subset of the data (say just position and velocity), the rest of the object’s data in the cache line is somewhat wasted.</p><p>In a Data-Oriented Design, you might separate the data from the behavior, and group similar data together. For example, you might have arrays for positions, velocities, healths, etc.:</p><pre>struct GameState {<br>    std::vector&lt;float&gt; posX, posY;<br>    std::vector&lt;float&gt; velX, velY;<br>    std::vector&lt;int&gt; health;<br>    // ... and so on for other properties<br>} state;</pre><p>Then update positions in a tight loop using the arrays (which are contiguous memory):</p><pre>for (size_t i = 0; i &lt; state.posX.size(); ++i) {<br>    state.posX[i] += state.velX[i] * deltaTime;<br>    state.posY[i] += state.velY[i] * deltaTime;<br>}</pre><p>This approach is essentially using a Structure of Arrays (SoA) instead of Array of Structures (AoS). The benefit is <em>excellent cache locality</em> and easy vectorization: all posX are in one array, likely in the same cache lines sequentially, so updating them is cache-friendly. You can load a bunch of posX values into SIMD registers and update multiple at once. Data-oriented design leads to <strong>predictable memory access</strong> and fewer cache misses. In our example, after processing one cache line of posX (maybe 16 floats, 64 bytes), the next iteration will hit a new cache line but sequentially, which is efficient. Contrast with chasing pointers to Enemy objects, where each object might bring in an unrelated cache line.</p><p>Another example is in algorithms: e.g., instead of doing an operation per object, collect all the data needed and do it in batch. This is how graphics APIs work (process arrays of vertices rather than one vertex at a time with a call).</p><p><strong>Data-oriented vs Object-oriented:</strong></p><ul><li>OOP is about abstraction, ease of maintenance, modeling the problem domain. But it can scatter data in memory (each object separately allocated) and focus on call patterns that may not be optimal for hardware.</li><li>DOD says: figure out how the data flows in your system and organize it linearly or in structures that the CPU can crunch easily. It often results in code that looks more procedural (operating on raw arrays of data) and might sacrifice some encapsulation (you might have global or external structures for data).</li><li>DOD doesn’t mean you abandon all OOP, but you separate the <em>hot</em> data path from the rest. For instance, in game development, a common pattern is Entity-Component-System (ECS): where entities are IDs, components are data (often stored in arrays by component type), and systems are functions that run on those arrays of data. This is a data-oriented design that still has some conceptual “objects” (entities) but the actual data layout is oriented by component type.</li></ul><p><strong>Performance Impact:</strong> Data-oriented design can have huge performance benefits. It’s not unusual to see an order of magnitude speedup when converting an algorithm from an OOP style to a DOD style for large data sets​(<a href="https://gamesfromwithin.com/data-oriented-design#:~:text=performance%20in%20modern%20hardware%2C%20with,some%20C%20code%20with%20assembly">gamesfromwithin.com</a>). By processing data in contiguous chunks and reducing cache misses, you unlock the full potential of the CPU (and SIMD). As an example, one could compare two approaches:</p><ul><li><em>AoS/OOP:</em> e.g., struct Particle { float x,y,z; /*...*/ }; and then an array of Particle. If you only need x, OOP would still load y,z into cache along with it, which is wasted bandwidth.</li><li><em>SoA/DOD:</em> separate float x[N], y[N], z[N]; – if you only need x for some calculation, you iterate over x array alone. Each cache line you load is fully utilized (64 bytes gives you maybe 16 floats of x, all used immediately). This efficiency translates to speed.</li></ul><p>One more aspect is <strong>branching and logic</strong>: OOP code might have lots of virtual calls or branches based on type (e.g., different subclasses implement update() differently). DOD often encourages grouping similar data and handling variations with data-driven approaches (like a separate array for type, and using a switch outside the inner loop, or even better, separate loops per type). This can reduce branch mispredictions as well, because each loop is homogeneous.</p><p><strong>Drawbacks of DOD:</strong> It can make code less intuitive if you’re used to thinking in objects. It often requires more upfront design to figure out the optimal layout. It might also increase memory usage slightly (padding, multiple arrays). But these are usually small trade-offs for the performance gained in scenarios that need it.</p><p>In practice, many high-performance C++ systems use a mix: critical inner loops and data structures are designed with DOD principles, while higher-level logic might still be OOP for organization. The key is to identify where the bottlenecks are and ensure the data in those parts is laid out optimally. As a famous saying in performance circles goes, “<strong>data layout trumps algorithm</strong>” when data size is large — meaning a linear scan through well-laid-out data can beat a theoretically faster algorithm that’s cache-unfriendly. To summarize, <strong>Data-Oriented Design</strong> is about focusing on how data is accessed and optimizing that, often by using arrays of data and sequential processing​</p><h3><strong>Conclusion</strong></h3><p>In this article, we’ve expanded our toolkit with several optimization strategies, from loop unrolling and function inlining to whole-program optimization and data-oriented design. These techniques significantly influence performance, particularly when carefully combined with the foundational strategies discussed in Part 1. By aligning your code’s structure closely with hardware realities — such as CPU caches, instruction pipelines, and memory access patterns — you can achieve dramatic performance improvements.</p><p>In our next installment - <strong>Part 3</strong>, we’ll explore threading and parallel execution strategies, lock-free and atomic operations, and methods to avoid common pitfalls such as false sharing.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6dc2ebf64813" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Advanced C++ Optimization Techniques for High-Performance Applications — Part 1]]></title>
            <link>https://medium.com/@martin00001313/advanced-c-optimization-techniques-for-high-performance-applications-part-1-55aba61b4dad?source=rss-4a528c57ee66------2</link>
            <guid isPermaLink="false">https://medium.com/p/55aba61b4dad</guid>
            <category><![CDATA[cpu]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[cpp]]></category>
            <category><![CDATA[c]]></category>
            <dc:creator><![CDATA[Martin Ayvazyan]]></dc:creator>
            <pubDate>Sun, 09 Mar 2025 08:10:47 GMT</pubDate>
            <atom:updated>2025-03-09T23:50:21.962Z</atom:updated>
            <content:encoded><![CDATA[<h3>Advanced C++ Optimization Techniques for High-Performance Applications — Part 1</h3><p>Performance is a critical feature. In domains like game development, high-performance computing (HPC), and real-time embedded systems, developers often push hardware to its limits. Modern CPUs offer extraordinary speed and parallelism but introduce complexities — pipelines, caches, branch predictors, and SIMD units — that skilled programmers can exploit. Naive code implementations can easily waste cycles: a single cache miss might stall the CPU, wasting cycles where hundreds of instructions could have executed, and branch mispredictions can flush pipelines costing 10–30 CPU cycles or more.</p><p>In this first part of our series, we’ll dive deep into advanced C++ optimization techniques:</p><ul><li>Branch prediction optimization</li><li>Cache optimization strategies (locality, prefetching, blocking)</li><li>SIMD (Single Instruction, Multiple Data) optimization</li></ul><p>In performance-critical software, small inefficiencies amplify at scale. If a game engine runs at 60 FPS, you have ~16ms per frame to do all computations; saving even 1ms through optimization can accommodate more game logic or better graphics. In HPC, a 10% speedup in a tight loop might save hours on a cluster job. And in embedded systems with limited CPU frequency, low-level optimizations can meet real-time deadlines without extra hardware. By understanding the CPU/GPU’s behavior — how it predicts branches, caches memory, executes multiple data in parallel, etc. — we can write C++ code that aligns with these hardware features and avoids performance pitfalls. Let’s explore these advanced techniques and see how to apply them in practice.</p><h4><strong>Branch Prediction Optimization</strong></h4><p>Modern CPU/GPUs guess the outcome of if statements and loops to keep their pipelines full. If the guess (branch prediction) is wrong, the CPU must discard work and correct course, incurring a <strong>branch misprediction penalty</strong>. This penalty can be hefty: on contemporary processors a mispredicted branch can cost on the order of 10–30 clock cycles <a href="https://johnfarrier.com/branch-prediction-the-definitive-guide-for-high-performance-c/#:~:text=trading%20systems%2C%20or%20real,millions%20of%20events%20per%20second">johnfarrier.com</a>(sometimes even more), which is <strong>significant</strong> if it happens in a hot loop. Therefore, writing branch-predictor-friendly code is crucial for low-latency C++.</p><p>Techniques to optimize branches:</p><p><strong>Favor predictable branches:</strong> Aim to structure conditionals so that one path is taken most of the time (and the CPU can learn that pattern). For example, handle rare error cases in separate branches and keep the common case straight-line. In C++20, you can use the [[likely]] and [[unlikely]] attributes to hint the compiler about which branch is expected. This allows the compiler to layout the code such that the likely path is the fall-through (no jump) path, improving instruction cache and prediction efficiency. For instance:</p><pre>if (value &gt;= 0) [[likely]] {<br>    processPositive(value);<br>} else [[unlikely]] {<br>    handleError(value);<br>}</pre><p>In the above snippet, we tell the compiler that the value &gt;= 0 path is the common case. The compiler may arrange the assembly so that the positive case doesn’t require a branch taken, and the error path is out-of-line. This <em>hints</em> the CPU’s branch predictor and can reduce misprediction frequency (though it’s not a guarantee—profile-guided optimization can further solidify such predictions).</p><p><strong>Branch elimination (branchless programming):</strong> Where possible, remove branches altogether by using arithmetic or bitwise operations. For example, instead of:</p><pre>// Count positives - with a branch<br>int count = 0;<br>for (int x : data) {<br>    if (x &gt; 0) { <br>        ++count;<br>    }<br>}</pre><p>You could write a branchless version:</p><pre>int count = 0;<br>for (int x : data) {<br>    // (x &gt; 0) is true/false, which in C++ converts to 1 or 0<br>    count += (x &gt; 0);<br>}</pre><p>Here the addition of 0 or 1 can be compiled to a conditional move or other branch-free sequence. The benefit is that we avoid unpredictable branching on each element. Branchless code is especially helpful when branch outcomes are data-dependent and random (e.g., processing an unsorted array of positive/negative numbers). However, keep in mind that branchless code can sometimes do extra work (e.g., always adding, even for negatives) – if the branch was highly predictable (say your data is <em>mostly</em> positive or mostly negative), a simple branch might actually be fine. Always consider the predictability of the condition.</p><p><strong>Loop and switch optimizations:</strong> If you have a chain of conditionals or a switch, order them so that the most likely cases come first. The CPU’s predictor uses recent history to guess, so stable patterns help. For example, in a game AI state check, if “idle” state occurs 70% of time, check for idle first. In some cases, replacing a chain of if with a lookup table or function pointer array can eliminate branches at the cost of an extra memory access (trading one type of cost for another). The right choice depends on measurability and predictability.</p><p>In summary, <strong>make branches easy to predict.</strong> Avoid complex, data-unpredictable branching in inner loops. Use compiler hints like [[likely]]/[[unlikely]] to improve layout​ (ref. <a href="https://johnfarrier.com/branch-prediction-the-definitive-guide-for-high-performance-c/#:~:text=The%20,and%20improves%20instruction%20cache%20performance">johnfarrier.com</a>). For critical branches that are hard to predict, consider transforming the code to use arithmetic or lookup logic instead of branching. These tweaks help keep the pipeline running smoothly and avoid costly stalls due to misprediction.</p><h4>Cache Optimization</h4><p>Memory access patterns often dominate performance, because fetching data from RAM is <strong>orders of magnitude</strong> slower than doing calculations. Modern CPUs have a hierarchy of caches (L1, L2, L3) to bridge the speed gap between the fast CPU and slow main memory. Accessing data that’s already in <strong>L1 cache</strong> might take only a few cycles, whereas a miss that goes out to main memory can take dozens or even hundreds of cycles​. Optimizing your code for caches — ensuring that data is loaded and accessed efficiently — can yield huge performance gains. Cache optimization breaks down into a few key ideas: <em>spatial and temporal locality</em>, <em>prefetching</em>, and <em>cache blocking (tiling)</em>. Let’s look at each.</p><p><strong>Cache Locality (Spatial &amp; Temporal Locality)</strong></p><p><strong>Spatial locality</strong> means accessing memory addresses that are close to each other, and <strong>temporal locality</strong> means reusing the same addresses within a short time. CPUs fetch memory in chunks called <strong>cache lines</strong> (typically 64 bytes on modern x86 systems​ <a href="https://en.wikipedia.org/wiki/False_sharing#:~:text=By%20far%20the%20most%20common,magnitude">en.wikipedia.org</a>). If your code accesses data contiguously in memory, one cache line fetch will bring in adjacent data that you’ll use next, resulting in fewer misses. Conversely, if you stride through memory with large gaps or follow pointer hops (linked structures), you’ll miss often and waste most of each cache line fetched.</p><p><strong>Example — Row-major vs Column-major traversal:</strong> Suppose you have a large 2D array int A[N][N]. Consider two ways to sum all elements:</p><pre>long sumRows = 0;<br>for (int i = 0; i &lt; N; ++i) {<br>    for (int j = 0; j &lt; N; ++j) {<br>        sumRows += A[i][j];<br>    }<br>}<br><br>long sumCols = 0;<br>for (int j = 0; j &lt; N; ++j) {<br>    for (int i = 0; i &lt; N; ++i) {<br>        sumCols += A[i][j];<br>    }<br>}</pre><p>Assuming a typical C++ layout (row-major), the sumRows loop accesses A[i][j] in memory order (one row at a time). This is cache-friendly: each 64-byte cache line might contain, say, 16 consecutive int values, so after the first element, the next 15 are already in cache. In contrast, sumCols jumps through memory by column: it accesses A[0][0], then A[1][0], etc. Each access is N elements apart in memory, likely on a different cache line. The sumCols loop will incur many more cache misses. The performance difference can be dramatic – iterating by rows can be several times faster than by columns for large arrays, purely due to cache effects.</p><p><strong>Guideline:</strong> Organize data access to maximize sequential access and reuse. If you have to traverse a multi-dimensional structure, do it in the memory layout order. If you have a data structure like an array of objects and only need one field, consider grouping that field into a separate array (more on data-oriented design later). Avoid “pointer chasing” (following linked list or tree pointers) in tight loops if possible; you might transform such structures into array-based structures for better locality.</p><h4>Prefetching</h4><p>Even with good locality, sometimes you access patterns that the hardware prefetcher (which tries to fetch upcoming cache lines automatically) might not anticipate. <strong>Prefetching</strong> is an optimization where you explicitly load data into cache <em>before</em>you need it, to overlap memory latency with computation. In C++, you can use compiler intrinsics like __builtin_prefetch() (GCC/Clang) or platform-specific functions to hint that a certain memory address will be needed soon. For example:</p><pre>for (size_t i = 0; i &lt; N; ++i) {<br>    if (i + 16 &lt; N) {<br>        __builtin_prefetch(&amp;array[i + 16], 0, 1);  // preload data 16 elements ahead<br>    }<br>    process(array[i]);<br>}</pre><p>Here, we tell the CPU to start fetching array[i+16] into cache while we are busy processing the current element. By the time we reach that future element, the data might already be in cache, avoiding a stall. The numbers (like prefetching 16 ahead) may need tuning based on how much work process() does and how your hardware behaves (too near and it might not arrive in time; too far and it might be evicted before use).</p><p>It’s important to note that manual prefetching is a micro-optimization — modern CPUs do a decent job at prefetching sequential accesses on their own. It’s most useful for non-linear access patterns (e.g., walking a linked list or a tree, where you can prefetch the .next pointer or child nodes). Use it sparingly and only after profiling shows a memory stall bottleneck. Also, prefetching useless data (that you never actually use) can <em>hurt</em> by evicting useful cache lines, so do it carefully.</p><p><strong>Cache Blocking (Loop Tiling)</strong></p><p><strong>Cache blocking</strong> (also known as loop tiling) is a technique to improve reuse of data in caches by working on subsets of data that <em>fit into the cache</em>. When an algorithm accesses a large data set with multiple loops, it might repeatedly bring data in and out of the cache. By blocking, we divide the problem into chunks that can stay in cache during computation, thus reducing memory bandwidth usage​ <a href="https://www.intel.com/content/www/us/en/developer/articles/technical/cache-blocking-techniques.html#:~:text=Blocking%20is%20a%20well,the%20user%20by%20making%20the">intel.com</a>.</p><p>Consider a simple example: multiplying two large matrices. A naïve triple-loop implementation will have poor cache reuse, because each element of the matrices might be loaded from memory many times. With blocking, you operate on submatrices (tiles) so that one chunk of A and B is loaded, you compute a tile of the result, then move on — ensuring that while computing that tile, the data stays in L1/L2 cache.</p><p><strong>Example — 1D blocking (conceptual):</strong> Suppose we have two arrays and we want to do something with every pair of elements. Instead of iterating one huge loop, we can process in blocks:</p><pre>const int BLOCK = 1024;<br>for (int start = 0; start &lt; N; start += BLOCK) {<br>    int end = std::min(start + BLOCK, N);<br>    for (int i = start; i &lt; end; ++i) {<br>        result[i] += compute(A[i], B[i]);<br>    }<br>}</pre><p>If BLOCK is chosen to be, say, 1024, then in the inner loop we’re working on 1024 elements of A and B at a time. This size might be tuned to fit in L2 cache. The effect is that those 1024 elements are brought into cache and reused within that block before moving on, instead of constantly streaming much larger arrays and evicting cache lines before reusing them. In a multidimensional scenario like matrix multiplication or image processing, blocking might involve two or three levels of nested blocks (for rows, columns, depth, etc.) to maximize reuse.</p><p>The key idea is <em>reuse</em>: by the time you move to the next block, you’ve fully utilized the data in the current block. Many libraries (BLAS, image libs) and compiler optimizations use cache blocking to great effect. It can significantly reduce cache misses in memory-bound applications​ . The downside is added code complexity and possibly more loops, but for large data processing it often pays off.</p><p><strong>Takeaway:</strong> Think about the cache as a working set of data you can operate on. Restructure loops and data traversal so that you work on chunks that fit in cache, use them thoroughly, then move on, rather than bouncing back and forth across a huge dataset. This can turn a memory-bound program into a cache-friendly one.</p><h4>SIMD Optimization (Single Instruction, Multiple Data)</h4><p>Most modern CPUs have vector instruction sets (SSE, AVX on x86, NEON on ARM, etc.) that can perform the same operation on multiple data elements simultaneously. This is called <strong>SIMD</strong>. For example, an Intel AVX2 register is 256 bits wide, so it can hold eight 32-bit floats at once — allowing you to add or multiply 8 floats in one CPU instruction​ <a href="https://www.intel.com/content/www/us/en/developer/articles/technical/accelerating-compute-intensive-workloads-with-intel-avx-512-using-microsoft-visual-studio.html#:~:text=number%20of%20elements,doesn%E2%80%99t%20cleanly%20fit%20into%20these">intel.com</a>. Using SIMD can potentially speed up math-heavy loops by a factor equal to the vector width (8x in this case, or even more with AVX-512 which operates on 16 floats at a time).</p><p>There are two ways to leverage SIMD in C++: <strong>automatic vectorization</strong> by the compiler, and <strong>explicit vectorization</strong> using intrinsics or libraries.</p><p><strong>Auto-vectorization:</strong> Compilers like GCC, Clang, MSVC will try to vectorize simple loops at optimization levels (-O2/-O3) when possible. For example, a loop adding two arrays element-wise might be auto-vectorized to use SSE/AVX instructions if the data alignment and loop trip count are favorable. You can help the compiler by writing <em>simple, straight-line code</em> in loops (avoid complex conditionals inside the loop, avoid data dependencies between iterations). Also, tell the compiler about pointer aliasing if applicable (use restrict or __restrict for raw pointers if you know they don&#39;t overlap, or rely on the compiler’s strict aliasing rules). Modern C++ high-level constructs like std::transform with execution policies (std::execution::par_unseq) can hint the compiler to vectorize and parallelize a loop. Example:</p><pre>#include &lt;algorithm&gt;<br>#include &lt;execution&gt;<br>std::vector&lt;float&gt; a, b, c;<br>// ... fill a and b ...<br>std::transform(std::execution::unseq, a.begin(), a.end(), b.begin(), c.begin(),<br>               [](float x, float y) { return x + y; });</pre><p>The unseq (unsequenced) execution policy tells the compiler it can vectorize this operation. Under the hood, it may use SIMD to add elements of a and b in chunks. <strong>Note:</strong> Always profile — compilers are good but not infallible at auto-vectorization. You can often use flags like -fopt-info-vec (GCC/Clang) to see if a loop was vectorized, and why not if it wasn’t.</p><p><strong>Explicit vectorization (intrinsics):</strong> For ultimate control, you can use SIMD intrinsics (found in headers like &lt;immintrin.h&gt;). These are functions that map directly to vector instructions. For example, using SSE/AVX intrinsics to add two arrays of floats:</p><pre>#include &lt;immintrin.h&gt;<br>void addArrays(const float* a, const float* b, float* result, int n) {<br>    int i = 0;<br>    // Process 8 floats at a time with AVX<br>    for (; i + 7 &lt; n; i += 8) {<br>        __m256 va = _mm256_loadu_ps(a + i);       // load 8 floats from a<br>        __m256 vb = _mm256_loadu_ps(b + i);       // load 8 floats from b<br>        __m256 vsum = _mm256_add_ps(va, vb);      // add the 8 floats pairwise<br>        _mm256_storeu_ps(result + i, vsum);       // store result<br>    }<br>    // Handle remaining elements (n might not be multiple of 8)<br>    for (; i &lt; n; ++i) {<br>        result[i] = a[i] + b[i];<br>    }<br>}</pre><p>This low-level code uses AVX registers to process eight floats per loop iteration. Intrinsics can be verbose, but they let you use instructions that the compiler might not choose on its own. You must be mindful of memory alignment (notice we used unaligned loads _mm256_loadu_ps for safety; if a and b were 32-byte aligned, we could use _mm256_load_ps which might be slightly faster). Intrinsics exist for a wide range of operations (add, multiply, bitwise ops, comparisons, etc.) and data types (integer vectors, double vectors, etc.).</p><p>Using explicit SIMD often yields <strong>big wins</strong> in performance for numeric code. For example, vectorizing a loop that multiplies matrices or applies a filter to an image can speed it up by 4x, 8x, or more depending on the available vector width and operation mix. The trade-off is code complexity and portability — intrinsics are tied to specific instruction sets. One should typically guard them with CPU feature checks or compiler flags (e.g., compile with -mavx2). Alternatively, higher-level libraries (Eigen, Blaze, XSIMD, etc.) or even the upcoming C++23 standard math utilities may provide abstraction over SIMD.</p><p>In summary, <em>make your code SIMD-friendly</em>. Leverage the compiler’s auto-vectorization by keeping loops simple and using hints like unseq. For critical routines, drop to intrinsics or assembly to fully utilize the vector units. The CPU can compute on 128-bit, 256-bit, or 512-bit chunks as easily as on 32-bit, so we want to give it that opportunity whenever data parallelism exists.</p><p><strong>Conclusion</strong></p><p>In this article, we examined how hardware-aware optimizations — branch prediction, cache management, and SIMD vectorization — can dramatically improve your C++ application’s performance. These techniques allow your code to fully exploit CPU capabilities, reduce latency, and maximize throughput. In Part 2, we’ll continue this exploration by discussing additional optimization strategies including loop unrolling, function inlining, threading, lock-free data structures, and techniques to avoid false sharing. Stay tuned!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=55aba61b4dad" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Decoding Memory: A Comprehensive Journey into C++ Smart Pointers]]></title>
            <link>https://medium.com/@martin00001313/decoding-memory-a-comprehensive-journey-into-c-smart-pointers-1b12f17a51eb?source=rss-4a528c57ee66------2</link>
            <guid isPermaLink="false">https://medium.com/p/1b12f17a51eb</guid>
            <category><![CDATA[operating-systems]]></category>
            <category><![CDATA[cpp]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[memory-management]]></category>
            <category><![CDATA[c-programming]]></category>
            <dc:creator><![CDATA[Martin Ayvazyan]]></dc:creator>
            <pubDate>Sun, 10 Dec 2023 22:25:43 GMT</pubDate>
            <atom:updated>2023-12-10T22:25:43.697Z</atom:updated>
            <content:encoded><![CDATA[<h3>O<strong>verview</strong></h3><p>In the world of C++ programming, the use of smart pointers is often associated with well-defined best practices in standard scenarios. Yet, beyond these commonly trodden paths lies a rich tapestry of memory-smartening techniques waiting to be explored. In this article, we set sail on a voyage into the profound seas of C++ memory management, peeling back the layers of abstraction to reveal the nuanced characteristics of smart pointers.</p><p>While smart pointers provide a convenient safety net, overlooking their intricate details can lead to a perilous journey fraught with undefined behavior and multi-threading issues. In this exploration, we&#39;ll uncover the subtleties that transcend the high-level best practices, shedding light on the considerations vital for writing robust and reliable C++ code focusing on concept and use of shared and weak pointers.</p><p><strong>Problem of raw pointers</strong></p><p>While raw pointers are applicable and can be used to refer to all — stack, heap and static memory, direct use of them for heap memory, which requires ownership of allocated resources and manual memory management, may lead to memory leaks and undefined behavior. The main reason is lack of RAII idiom for raw pointers:</p><ul><li><strong>Exceptions may lead to memory leak</strong>, such as in the below example the destructor of object A will never be called as the ‘delete ptr’ statement will never be executed because of the exception. LSAN(leak sanitizer) will also <a href="https://godbolt.org/z/Prrq4MrzP">report this as a memory leak</a>.</li></ul><figure><img alt="Exception leading to memory leak" src="https://cdn-images-1.medium.com/max/1024/1*G2Z2lB_q4BVS_VDOLM_iDA.png" /></figure><ul><li>Pointers/references referring to destroyed object(aka dangling pointer), leading to <a href="https://godbolt.org/z/r9PzjjW78">undefined behavior</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/694/1*ozGNmeVlqEzz79f7j-zoYg.png" /></figure><h3>Smart pointers</h3><p><strong>Shared and weak pointers</strong>: <a href="https://en.cppreference.com/w/cpp/memory/shared_ptr">std::shared_ptr</a> and <a href="https://en.cppreference.com/w/cpp/memory/shared_ptr">std::weak_ptr</a></p><p>Shared pointer provides shared ownership of dynamically allocated objects. The dynamically allocated object is only destroyed when the last std::shared_ptr that owns it is either explicitly reset, assigned a different pointer, or goes out of scope.</p><p>std::weak_ptr stands as a silent observer, providing non-intrusive glimpses into the existence of dynamically allocated objects. Unlike its shared counterpart, std::weak_ptr holds a non-owning, weak reference to an object. The essence of std::weak_ptr lies in its ability to coexist peacefully with the ephemeral nature of objects. While multiple weak pointers can point to an object, they carry no influence over its lifespan. Even if a legion of weak pointers ceaselessly observes an object, the object&#39;s destiny remains unaltered, succumbing to destruction when the last shared pointer releases its grasp. However <strong>it affects on lifecycle of the control block</strong> being created when the first shared pointer is initialized.</p><p>While C++ standard doesn’t provide any strong guarantees of underling implementation of the structures(i.e. how control block and other components behave), let’s check some of possible ones.</p><p><strong>Control block:</strong></p><p>The count of references and all additional information is kept in ‘control block’ — which is additional heap allocated memory space keeping state of the shared object, specifically:</p><ul><li>pointer to the heap memory block of the object</li><li>shared reference count — indicating how many shared_ptr instances are referring to the object</li><li>weak reference count — indicating how many weak_ptr instances are referring to the object</li><li>deleter object — the deleter object being responsible for releasing the heap memory resource of the object. This can be useful if there is a need to implement specific memory pool and manage heap allocation/deallocation</li><li>allocator object — the object being responsible to allocate the memory for both managed object and control block.</li></ul><p>So in summary, shared pointer, as an object keeps to references:</p><ul><li>pointer to the heap allocated object</li><li>pointer to the control block</li></ul><p>while weak pointer needs to kee[ a reference only to the control block. A high level visual representation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V_7iA3fqppsIHNsmxg5eVA.png" /><figcaption>The main components of memory model for shared &amp; weak pointers</figcaption></figure><p>The main observations:</p><ul><li>Shared pointers keeps an ‘extra’ reference to the heap location of the object, while could just keep a reference to the control block only and use control block’s reference to access the heap object. However, it would leave performance impact, as each object access would require 2 level of de-referencing: shared-ptr → control block → object.</li><li>As shared pointer keeps two properties as a member object, t<strong>he update of shared pointer’s state in MT mode can lead to data races and MT issues</strong>(e.g. while the first member is updated, another thread updates/reads the data). However, the control block itself is thread safe, i.e. there can’t be a case that the object is destroyed twice and increment/decrement of counters is atomic operation.</li><li>Control block keeps a counter of weak pointers too, as this information should be used to determine when the control block can be destroyed. Weak pointer uses control block to determine if the object is still available to construct shared pointer through weak_ptr::lock() function.</li></ul><p>The next question we may have is the advantage of using std::make_shared instead of standard std::shared_ptr&lt;T&gt;{new T{}};</p><p>To understand the advantage, let’s check the memory model representation of control block and shared instance in case of std::make_shared</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DUiISXZe1dj_-IhLQZ_Mwg.png" /><figcaption>memory model in case of std::make_shared</figcaption></figure><p>As shown, in case of std::make_shared , the memory block of the object is part of control block memory space, which significantly improves memory access and minimizes <a href="https://learn.microsoft.com/en-us/windows/win32/memory/low-fragmentation-heap">fragmentation of heap memory</a> space. Also, in scenarios where the constructor of the allocated object may throw an exception, using std::make_shared is preferable. This is because std::make_shared performs a single allocation for both the control block and the managed object, reducing the chances of resource leaks in the presence of exceptions. With the help of<a href="https://medium.com/@martin00001313/since-c-11-the-standard-provides-amazing-futures-to-implement-relatively-simple-and-high-optimal-ced5b13c57d1"> perfect-forwarding</a>, this makes the best choice to construct shared instance, if the constructor of the object<strong> is not private</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aYsvQ7bImUmstsHVBoJVUw.png" /></figure><p>In the realm of C++, the intricacies of managing shared ownership can be elegantly navigated with the assistance of std::enable_shared_from_this. This feature allows an object, currently under the guardianship of a std::shared_ptr, to gracefully spawn additional shared pointers, forging a collective ownership alliance. By publicly inheriting from std::enable_shared_from_this&lt;T&gt;, an object of type T gains access to the transformative shared_from_this member function, which acts as a portal, enabling the creation of new std::shared_ptr instances (pt1, pt2, and beyond) that seamlessly share ownership of the object with the original std::shared_ptr. One of the<a href="https://godbolt.org/z/7vG9qcq3Y"> use cases</a> is when object need to register itself as an observer of some specific events:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J1Hmh02ErpB-QmchFByZKA.png" /></figure><p>Few key observations for use of std::enable_shared_from_this</p><ol><li>The inheritance of std::enable_shared_from_this must be public. Otherwise, private/protected inheritance will lead to undefined behavior(<a href="https://godbolt.org/z/bbYTddfo9">example</a> <a href="https://godbolt.org/z/bbYTddfo9">here</a>). This is because the std::shared_ptr mechanisms rely on the ability to access the std::enable_shared_from_this members publicly. Specifically, the shared_from_this member function needs to be accessible from the derived class to create additional shared pointers that share ownership with the original std::shared_ptr managing the object.</li><li>Use of shared_from_this() function directly/indirectly in constructor or destructor of the class will lead to undefined behavior. This is because of order of statements execution, specifically the order is:<br>- allocation of heap block for the object(+control block in case of std::make_shared )<br>- placement new/constructor call of the class-object on the allocated heap memory space. <strong>I.e. no control block/shared-ptr is available yet!</strong><br>- initialization of control block, shared_ref_count=1 <br>- return the created shared_pointer<br>So, during constructor’s execution there is no info/details about shared pointer → use of shared_from_this() will lead to undefined behavior. The same about detractor, as the object’s destruction order is the opposite of construction, which means shared-ptr data will be destroyed first(example <a href="https://godbolt.org/z/3f38YaKGe">here</a>).</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VNnrPMXYwGSpCY0j7LcrxQ.png" /><figcaption>Issues calling shared_from_this in constructor/destructor</figcaption></figure><p>And finally, <strong>why to use </strong><strong>std::move</strong> for shared and weak pointers?</p><p>Using std::move with shared and weak pointers in C++ is a practice that involves transferring ownership. But other than that, from performance perspective, this helps to eliminate unnecessary update operations on control block(such as increment, decrement of shared/weak counters), which can be crucial in case of low level development.</p><h3>Summary</h3><p>In conclusion, navigating the intricate landscape of C++ smart pointers unveils a realm of possibilities for effective memory management, robust code, and enhanced performance.</p><p>It’s worth noting that while our exploration primarily centered around std::shared_ptr and std::weak_ptr, similar principles apply to std::unique_ptr. This versatile smart pointer, with its exclusive ownership semantics, is a powerful tool in the C++ arsenal, offering a unique perspective on memory management.</p><p>It’s crucial to note that while smart pointers provide a robust mechanism for memory management, there’s a fundamental consideration in low-level development domains such as EDA, finance, gaming, etc. Minimizing heap memory usage remains imperative in these contexts.</p><h3>What is next?</h3><p>Concluding our discussion on C++ smart pointers, the journey into technical frontiers continues. Upcoming article will be about C++ coroutines focusing on how they introduce a paradigm shift in handling concurrency and asynchronous tasks, offering a technical perspective on streamlining code execution. Stay tuned for in-depth discussions on the technical aspects of C++ coroutines.</p><p>Enjoy coding in C++!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1b12f17a51eb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering Static Objects in C++: Initialization, Destruction, and Best Practices]]></title>
            <link>https://medium.com/@martin00001313/mastering-static-objects-in-c-initialization-destruction-and-best-practices-760b17734195?source=rss-4a528c57ee66------2</link>
            <guid isPermaLink="false">https://medium.com/p/760b17734195</guid>
            <category><![CDATA[c]]></category>
            <category><![CDATA[embedded-systems]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[cpp]]></category>
            <category><![CDATA[development]]></category>
            <dc:creator><![CDATA[Martin Ayvazyan]]></dc:creator>
            <pubDate>Sat, 16 Sep 2023 06:57:13 GMT</pubDate>
            <atom:updated>2023-09-16T06:57:13.099Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Overview</strong></h3><p>Static objects hold a unique position in the software development world. They offer a kind of memory storage that persists across function calls while only being initialized once. This can make them incredibly useful, but also a bit difficult to manage if not understood correctly. In this post, we will delve into the lifecycle of static objects, their initialization, and destruction order.</p><p><strong>Lifecycle of C++ Program</strong></p><p>Before diving deeper into the nuances of static object lifecycles, it’s essential to have high level understanding of lifecycle of C++ programm from source code -&gt; compilation -&gt; running process . These stages significantly impact how static objects are initialized and destroyed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-Atb3pt903sPJ9YDkzulLw.png" /></figure><p>During the <strong>compilation</strong> stage, each <strong>translation unit</strong> (basically, a source file and its included headers) is compiled into an object file. Static objects declared in the translation unit are identified, but initialization code may not yet be executed. This is where the compiler ensures type safety and performs optimizations.</p><p>Then the <strong>Linker</strong> combines all the object files and resolves symbols. It sets the stage for static initialization by determining the layout of the objects in the executable. This stage is critical because it determines the order of initialization and destruction for static objects within each translation unit.</p><p>Next is<strong> Loader</strong>: When program is executed, the operating system loader places it into memory, making final preparations before the main() function starts. Some systems may perform initializations of static objects during this phase, especially for built-in types.</p><p>As loader ends preparation, next is <strong>Execution</strong> of the programm on the dedicated process: This is where static objects are initialized and later destroyed. Initialization occurs before entering main() for global and file scope static objects. For local static objects, initialization occurs the first time the control flow reaches their declaration. Destruction happens in reverse order when the program is terminated, either by calling exit() or leaving main(). Worth to mention that in case of abnormal termination of the process(e.g. abort()), the destruction of static object won’t be called at all.</p><p>There is final stae of app lifecycle — <strong>Unloading. </strong>This is when resources are returned to the operating system, but in the context of a C++ program, the static object destructors are invoked as part of the program’s execution, not afterward. I.e. this doesn’t affect of static objects lifecycle and behavior.</p><h3><strong>Types of static objects</strong></h3><p><strong>1. Global static variabler</strong><br>These are variables being defined outside of a class and function, in global or nested namespace. <br>Example: let’s define a class in header file with respective static variable associated as part fo definition</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sKbFvoh3SzljbI_WmCEkBg.png" /></figure><p>Here o is global static variable, which will be created <strong>for each translation unit</strong> in scope of include statement.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tziqGJYvxppiJDQKsTczoQ.png" /></figure><p>In above example, all the static variables defined are global to their respective translation unit, specifically to the t.cpp file. This ensures that only one instance of each static variable exists within that unit. These instances are initialized during the application&#39;s startup phase, before the main function begins execution.</p><p>However, it’s worth noting a subtle distinction when dealing with static variables defined in an unnamed namespace (as in the case of o3) or using the static keyword (as in the case of o4). These nuances come into play particularly when discussing extern static variables, a topic that deserves its own focused explanation.</p><p><strong>2. Static variables defined in the class</strong></p><p>These are static variables defined as part of the class with static keyword.</p><p>Such static variables consist of 2 key subcomponents:<br>a. <strong>declaration of the static object in the class </strong>— to specify ‘existence’ of the instance</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*83q--qHpeOtGWYCKtn-bTQ.png" /></figure><p>b. <strong>definition of the static object</strong> — to define and initialize/construct the static variable</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N2HfnNowqV8imVC23VHcUg.png" /></figure><p>In C++, defining a static class variable requires a separate definition outside the class declaration. Failing to do so will result in a linker error. The reason for placing this definition in a .cpp file is to avoid multiple instances of the variable being created across different translation units that include the class header file.</p><p>However, starting with C++17, the language introduced the inline keyword for static variables, allowing you to declare and define them simultaneously within the class declaration. This ensures that only one instance of the static variable is maintained, even if the class header is included in multiple translation units. The linker will select a single definition from one of the translation units and discard the rest, thus avoiding duplication.</p><p><strong>3. Local static variables</strong></p><p>These are static objects being defined in a function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MlRUrpfYnPqZGFCjotuDRw.png" /></figure><p>The instance of the static object is constructed the first time program execution reaches the object’s declaration point — in this case, the first time the function is called. It’s worth noting that, thanks to the C++11 standard, the construction of the static object is thread-safe. This means that even if multiple threads call the function simultaneously, the object is guaranteed to be created and initialized only once.</p><p><strong>4. Extern static variables</strong></p><p>These are global variables being declared with extern keyword.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/1*M14rYImyaxV8-dVLsyQw4A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J6TwfJJanbR8c9j0arwShg.png" /></figure><p>The extern keyword in C++ enables the referencing of global or static variables located in another translation unit. While this can be useful, it introduces the risk of unexpected and difficult-to-debug behavior. Therefore, usage of extern is advised only in essential situations.</p><p>To avoid or reduce the need for such external variable references, alternative approaches are available. Before the C++11 standard, the static keyword was often used to restrict a variable&#39;s linkage to its originating translation unit. An equally effective, and often preferable, method is to define the variable within an unnamed namespace, which accomplishes the same objective of limiting the variable&#39;s linkage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MAICdHn33-wko9yASGdWeg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iR934tSKqNB2UuqEg1gssg.png" /></figure><p><strong>5. Compile time known expressions</strong></p><p>Compile-time constant expressions are variables or values evaluated and fixed during the compilation stage, rather than at runtime. Their value lies in optimization potential: upon encountering these expressions, the compiler replaces any references to them with their actual, precomputed values. This preemptive substitution eliminates the need for repeated runtime evaluations, thereby boosting program performance. Moreover, compile-time evaluation allows for the early detection of potential errors, enhancing code robustness. It’s important to note that compile-time constant expressions always result in built-in types and have no specified lifetime, essentially becoming hardcoded values within the program.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Lug4F9m3JttIQ9OaAGUMyw.png" /></figure><p>While compile-time constant expressions can be defined through multiple avenues, including #define and occasionally const (depending on compiler support), constexpr stands out as the most preferable method. One of the key advantages of constexpr is its stringent validation process. Unlike #define, which performs simple textual substitution and lacks type checking, constexpr enforces strong type safety. Additionally, constexpr allows for more complex compile-time computations, making it possible to use functions, conditionals, and loops, all while ensuring that the variables meet the criteria for being constant expressions.</p><p>By using constexpr, not only is type integrity maintained, but the language features also facilitate more nuanced and powerful compile-time logic. These advanced validation capabilities make constexpr the superior choice for defining compile-time constant expressions, contributing to both code robustness and maintainability.</p><h3>Initialization and destruction order specifications for static objects</h3><p>Static orders initialization can be either <strong>ordered</strong> or <strong>unordered</strong>. In case of unordered specialization(aka Static Initialization Order Fiasco) there is no guarantee on sequence of initialization, hence use of such static objects to initialize other static instances will lead to undefined behavior. The main group of static objects having <strong>unordered initialization</strong> are:<br><strong>a.</strong> non-explicit specialized class template static data members (ISO/IEC 14882:2017–6.6.2).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MesCC9auMBmi-jwGA80-9Q.png" /></figure><p><strong>b. </strong>static objects in different translation units</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/718/1*suigtmRoTKkNMEOnpVohkQ.png" /></figure><p>So, use of such static objects in constructor/destructor of other static objects will lead to undefined behavior(<strong>UB</strong>).</p><p>In C++, the construction order of<strong> non-template</strong> or<strong> explicitly specialized class template</strong> static objects within the same translation unit is well-defined. They are constructed in the order in which they are defined. Similarly, the destruction order is also well-defined but in the reverse order of construction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/590/1*shG3qQ0gVNfNDIUsvpFg-w.png" /></figure><p>Destruction order is defined as <strong>reverse</strong> of <strong>ordered</strong> construction (ISO §6.6.3).</p><p>Additionally, if there are static objects<strong> in the same translation unit</strong> with cyclic dependencies (e.g. through their member objects), it can lead to undefined behavior or tricky bugs during initialization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FNS2_uFa4oWKu2efNBq0MQ.png" /></figure><p>In the example, the cyclic dependency between the static objects of ‘A’ and ‘B’ during their construction results in undefined behavior.</p><h3>Tips to avoid Undefined behavior for dealing with static objects</h3><p>Avoiding undefined behavior when dealing with static objects, especially those with interdependencies, requires careful design and some best practices. Here are some tips to help you avoid pitfalls:</p><ol><li><strong>Minimize the responsibility/logic of static objects constructors and destructors —</strong> Specifically, avoid using static objects in scope of constructor/destructor of other static objects.</li><li><strong>Minimize use of global static objects</strong> — If possible, avoid using global static objects, particularly those with complex constructors and destructors. For shared libs being used by different functionals, minimize the scope to have up to one global static instance which will ‘manage’ lifecycle of all remaning sub-components. The more global objects you have, the harder it is to manage their initialization order and dependencies.</li><li><strong>Use Two-Phase Initialization</strong> — For objects that depend on each other, consider using a two-phase initialization process. The constructor performs basic, order-independent initialization, and an explicit init method performs further initialization that might depend on other static objects.</li><li><strong>Function-Local Statics — </strong>When feasible, use function-local static objects to ensure that they are initialized when the function is first entered. This pattern is often used for Singleton implementations.</li><li><strong>Avoid Cyclic Dependencies</strong> — Design your classes and their initializations in such a way that you don’t create cyclic dependencies.</li><li><strong>Dependency Injection</strong> — Instead of having objects fetch their dependencies, consider passing them in as parameters. This can decouple the objects and make their dependency more explicit.</li><li><strong>Document Assumptions and Dependencies </strong>— Whenever you find it necessary to rely on the order of initialization, document it clearly in the code comments. Make sure any assumptions or dependencies are well-documented to aid in future debugging and maintenance.</li><li><strong>Use Dynamic Allocation </strong>— As a last resort, you can use dynamic allocation (new/delete) for problematic static objects to control their lifetime explicitly. However, this comes at the cost of manual memory management.</li></ol><p>By being cautious and employing these tips, you can minimize or eliminate undefined behavior associated with static object initialization in C++.</p><h3>Additional notes</h3><p>While static objects are generally constructed only once during an application’s lifetime, incorrect library configuration can lead to anomalies where the same static object is constructed multiple times. This unusual behavior is most often caused by a mix of static and shared dependencies of the same library within a single application. Specifically, if a library is linked both <strong>statically</strong>(.lib, .a) and <strong>dynamically</strong>(.so, .dll), separate instances of the same static object could be created, contradicting the single-instance expectation. This can also lead to violations of the One Definition Rule (<strong>ODR</strong>), as the static object might get defined in multiple places. Such inconsistencies can introduce subtle bugs and undefined behavior, making it crucial to ensure consistent library usage and linkage throughout the application. However its another big topic and I will delve into the complexities of the ODR and how it interacts with static and dynamic linking in a separate article.</p><p>Enjoy your C++ journey! Happy coding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=760b17734195" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>